B2 Cloud Storage Question & a 50TB Sync Jobs

[notjoe]

I have the daunting task of backing up close to 50TB worth of data, offsite. I've decided to go with B2 since they seem to be the most cost effective solution. My question simply is this. During the course of uploading over 50TB worth of content into the "Cloud" there will surely be a time with the upload will be interrupted. When the sync process starts again will it resume where it left off or will it start from the beginning. I really could not find any sort of absolute answer from searching.

 

[Nick2253]

It depends on which client you are using. B2 itself does not care about the files you are storing on it; it's basically just providing you a giant volume that you can fill. It's up to your client to manage the connection, and verify the data that it's writing up to B2.

 

[notjoe]

It depends on which client you are using. B2 itself does not care about the files you are storing on it; it's basically just providing you a giant volume that you can fill. It's up to your client to manage the connection, and verify the data that it's writing up to B2.

I should have been a little more informative in my original post. My bad. I am using the Cloud Sync task and not some client I've installed myself inside of a jail or something.

 

[Nick2253]

Cloud Sync task

That I'm not sure about. A quick way to test is to start the sync task, pull the network cable from the FreeNAS, and see what happens.

 

[notjoe]

That I'm not sure about. A quick way to test is to start the sync task, pull the network cable from the FreeNAS, and see what happens.

LOL. videos editors might pull my head from my neck if I did that!

 

[Nick2253]

LOL. videos editors might pull my head from my neck if I did that!

I would hope you're not messing around on a live system

If you don't have a test box, spin up a test VM. You can use Virtualbox on your desktop if you don't have a hypervisor you can use. Then you can simply disconnect the network cable in software.

I hate to harp, but have you gone through test backups and restore with this service? I hope you're familiar with what you'd need to do, and have a client to do it, in the event that your FreeNAS server dies and you need to restore data.

 

[notjoe]

I'll most likely play around with it on my home installation of FreeNAS before I start fucking around with the production. The backup process I know will be a slow and painful one. At the end of the day, that really doesn't matter much since the intended usage is purely for archival sake. If we need to repopulate the contents I'll manually select the newer content first while the automated restore takes place.

I have a few more not important FreeNAS installations here that I can can play with but at the end of the day I was hoping to get some answers for some obvious issues I'll have along the way regardless of which offsite storage provider I've selected. The same question is true for the restore part of it.

 

[rclarkin]

Cloud sync does not deal happily with a network interruption. I had a network "hiccup" in the middle of syncing 130Gb of data (also to B2) and it died at that point. (Crashplan on another box also had it's backup interrupted but it just resumed a couple of minutes later -- I sure wish that cloud sync would do something similar!)

And all I can say about 50Tb is that I hope that your pipe is bigger than mine. My family just about killed me when I set the 130Gb to backup and saturated the line for 2 full days. I found out very quickly how addicted to the Internet we all are! :-o

 

[notjoe]

Cloud sync does not deal happily with a network interruption. I had a network "hiccup" in the middle of syncing 130Gb of data (also to B2) and it died at that point. (Crashplan on another box also had it's backup interrupted but it just resumed a couple of minutes later -- I sure wish that cloud sync would do something similar!)

And all I can say about 50Tb is that I hope that your pipe is bigger than mine. My family just about killed me when I set the 130Gb to backup and saturated the line for 2 full days. I found out very quickly how addicted to the Internet we all are! :-o

We have more than one internet connection in the office. I have the backup routed out the secondary line so it isn't creating any issues of bandwidth contention.
CloudSync will resume as far as I am aware. I've stopped and restarted it a few times already and I see it uploading new files and taking up more space within b2. I assume the sync option supplied by CloudSync to rclone (the underlying technology powering cloudSync) seems to work though though you could always add the --size (if i recall) and it will compare local and remote file sizes before deciding whether to send the file.

 

[notjoe]

Cloud sync does not deal happily with a network interruption. I had a network "hiccup" in the middle of syncing 130Gb of data (also to B2) and it died at that point. (Crashplan on another box also had it's backup interrupted but it just resumed a couple of minutes later -- I sure wish that cloud sync would do something similar!)

And all I can say about 50Tb is that I hope that your pipe is bigger than mine. My family just about killed me when I set the 130Gb to backup and saturated the line for 2 full days. I found out very quickly how addicted to the Internet we all are! :-o

I forgot to ass that cloudSync will run every X depending on what you set the intervals to. I'm running it manually for the initial sync.