Posted: 1/13/2009 10:45:45 AM EDT
|
How long (approximately) should I plan on it taking for my system to run a 25 gig upload to our offsite backup server farm over a T1 line?
The server farm will be doing 4096 bit encryption as it saves and compresses the data (consisting of 277,000 files). Just trying to get a sense of how many hours. |
|
I can speak from experience copying large amounts of data offsite on a t1 that it will take several hours(upwards of 24) to complete the copy assuming that there is no other traffic on the line. Is the computer that is managing the copy also doing the encryption in real time or are the files encrypted prior to copying
|
|
Quoted: Quoted: Let's see T1 = 1.544 Mbps or ~197KB/s. 25G = 25,600MB = 26,214,400 KB Shortest theoretical time = 133068s = ~37 hours. Add a fudge factor of 10% or more perhaps. See above. This. With TCP overhead and other misc factors, it tends to be closer to 175 KB/s, give or take. He didn't say what protocol he was using, though.    |
|
Quoted:
Quoted:
Quoted:
Let's see T1 = 1.544 Mbps or ~197KB/s. 25G = 25,600MB = 26,214,400 KB Shortest theoretical time = 133068s = ~37 hours. Add a fudge factor of 10% or more perhaps. See above. This. With TCP overhead and other misc factors, it tends to be closer to 175 KB/s, give or take. He didn't say what protocol he was using, though.
NetBIOS with Win3.11?
|
|
If my math is correct:
1.5megabits/sec = 193kilobytes/sec 25 Gigabytes / 192kbs = 129.5k secs 129.5k secs / 86400 sec/day = 1.4 days. That's assuming no additional delays from CPU overhead (encrypt/decrypt) or router bottlenecks, circuit delays etc. ETA: slow on the post
|
|
There's an old saying. What's the fastest way to get 500Mb to California? FedEx.
Wow, its very hard to put a real time estimate on something like this. Just running the numbers doesn't account for network load or latency. Lots of variables that you have to work out about your network and the hardware attached to it. There is a reason for doing local storage with a streaming compressed feed to a remote backup. This way things get backed up right now. In the mean time you stream your backup to the remote site as you can get it done. One of the best solutions is a local backup, write to tape and stream to a remote site. Might want to include some off-site storage too. The last big shop I worked in we used NetBack to do local backup and recovery with the backup going to a disk farm. From there it was streamed to a remote central repository on a dedicated circuit. No othere traffic on it, ever. We also wrote each backup to tape and Iron Mountain picked up and returned old stuff twice a week. Off site retention was on a 60 day cycle. |
|
Quoted:
Thanks guys... I'm going to start it Friday afternoon, and hopefully it won't take too much longer. I wish there were a way to "pause" the uploads during the first backup. We'll be going to incremental after this, but the first push is going to be a pain. I don't know how you do your backups but heres how I do mine. I use a program called duplicity. Its a free linux tool. Compresses and encrypts (via GPG). I backup locally. Then use rsync over ssh to send it to a remote site. You can control rsync's bandwidth usage via arguments. And if rsync loses connection, gets killed, or otherwise dies you can always resume where you left off. Sometimes I setup a cron job to start rsync at a particular time and then another cron job to kill it off if it is still going after a few hours. Just got to monitor that it doesn't get too far behind. For that You just don't kill it off over the weekend 
-Foxxz ETA - My backup jobs are 250GB for a full backup over a 2 Mbps line. ETA2 - duplicity rocks. It makes binary diffs for incremental backups. So if you have a 5GB file and maybe only a few MB change in the MIDDLE of the file it only backs up those changes and NOT the entire file again. |
|
Quoted:
Using super-expensive leased line to move data that could be moved on a few thumb drives???? A few? 64GB thumb drives are under $150!
http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=2013240522%201309439130&name=64GB -Foxxz |
|
Quoted: Small point of contention, but duplicity isn't doing that, rsync is. ETA2 - duplicity rocks. It makes binary diffs for incremental backups. So if you have a 5GB file and maybe only a few MB change in the MIDDLE of the file it only backs up those changes and NOT the entire file again. |