Incremental, versioned cloud backup?

Status
Not open for further replies.

vbGvMWpXht4kEm

Dabbler
Joined
Oct 17, 2016
Messages
33
My goal is to get incremental offsite backups to something like S3 or Amazon Cloud Drive, and retain the ability to rollback to previous versions of the data. I've set up rclone, but that only mirrors the current state without the ability to rollback.

I have hourly snapshots, so it would be ideal to be able to somehow back them up and be able to restore them to/from cloud in a cost efficient manner. Replication needs a ZFS target, right?

Maybe I'm worried about low-probability events here, but I'd like to solve for issues that a non-versioned backup won't handle:

1. File silently goes corrupt from an app, don't notice
2. FreeNAS server burns down in house a week later
3. Restore from offsite backup
4. Notice corrupted file, rollback/clone snapshot
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
You might consider putting HashBackup in a jail. Works with b2, s3, Google, SFTP ... and can use rclone as an intermediary for services it doesn't support directly.
Replication needs a ZFS target, right?
Yes, so if you can afford it, rsync.net is out there.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Would it be enough to rclone the dataset and have the .zfs dir (+snapshots) come along for the ride?
I think you would end up with massive bloat, because rclone would see every subfolder of .zfs as a complete version of the dataset at the time the snapshot was taken.
 

vbGvMWpXht4kEm

Dabbler
Joined
Oct 17, 2016
Messages
33
HashBackup sounds promising, and I cannot afford rsync.net.

Right, that makes sense about snapshots and rclone seeing the entire dataset.

Thx!
 

NightNetworks

Explorer
Joined
Sep 6, 2015
Messages
61
How much data are you looking to backup? Is this for a business? And how fast do you need to be able to complete a full restore?
 

vbGvMWpXht4kEm

Dabbler
Joined
Oct 17, 2016
Messages
33
Minimum 200GB (high importance stuff), would prefer entire pool (2TB right now and growing). Personal use. Recovery time doesn't really matter too much.
 

vbGvMWpXht4kEm

Dabbler
Joined
Oct 17, 2016
Messages
33
Yeah, it's more the versioned/incremental part that I'm interested in solving. I use S3/Glacier with Arq on my desktop machine.

HashBackup + rclone seems like a decent replacement, though the potential for monetization is troubling depending on which model the developer goes with.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
HashBackup is versioned/incremental, and supports b2 natively, no need for rclone. That's only for destinations it doesn't have built-in.
 

snaptec

Guru
Joined
Nov 30, 2015
Messages
502
If you would like to use rsync or zfs replication drop me a pm, that's my business - cheaper than rsync.net ;)


Gesendet von iPhone mit Tapatalk
 

NightNetworks

Explorer
Joined
Sep 6, 2015
Messages
61
So with the information that you have provided I think that the method that I use for backing up my FreeNAS server would work for you as well.

OK lets get some basic information established first...

Server names and purpose,
- Crashplan01
Has direct access to all data on the FreeNAS server via CIFS shares. No actual data is stored on this server its job is to monitor for changes on the network shares, compress the data, de-duplicate the data, encrypt the data, send the data to one of two storage servers/crashplan cloud, and do regular health/integrity checks on the backup data.
- Storage01
Separate local server that is used ONLY for storing data that is received from the Crashplan01 server. This server has a single 2TB (no raid) hard drive that is used for storing the data.
- Storage02
Separate local server that is used ONLY for storing data that is received from the Crashplan01 server. This server has a single 4TB (no raid) hard drive that is used for storing the data.
- Crashplan Cloud.
This is NOT a local server... This is in reference to the "cloud" based backup storage that Crashplan (crashplan.com more information on this later) offers. This is used for the storage of offsite backups. Crashplan gives you unlimited storage space.

MISC Information,
- FreeNAS Server has a total of 6TB of usable storage space
- Total backup space between the two storage servers is 6TB... this can be expanded by using larger hard drives or by adding additional storage servers.

How it all works...

While this setup may seem complicated it really is pretty simple and very effective. The Crashplan01 server is the server that does all of the work. It is the server that has access to all of the data that you are storing on the FreeNAS Server and does all of the work when it comes to actually backing up that data. In my particular setup I have a Windows based virtual machine (named Crashplan01) that has the Crashplan Desktop Application installed. The Crashplan01 server through the use of a number of CIFS shares has direct/full access to the 4TB of data that is currently stored on my FreeNAS Server. The Crashplan Desktop Application monitors/scans these CIFS shares on a regular schedule and backs up the corresponding data to one of the two storage servers or the Crashplan Cloud. The Storage01 and Storage02 server are also separate Windows based virtual machines running the Crashplan Desktop Application. Neither of these servers have direct access to any of the data that is stored on the FreeNAS Server. Instead they receive all data via the Crashplan Desktop Application from the Crashplan01 server. There is an important reason as to why the data is not stored directly on the Crasplan01 server which I will explain later. You also have the ability to configure the Crashplan Software to send your data to there internet based cloud which gives you the ability to do off site backups. In my configuration I have the Crasplan01 server configured to backup all data locally to one of the two storage servers and to the Crashplan Cloud. This gives me both local backups and offsite cloud backups.

Local and Cloud backups...

So at this point you should realize that my data is backed up in two separate locations the Crashplan Cloud and my two local storage servers. There are really two primary reasons why I did this...

Reason One,
My data now exists in three locations... FreeNAS Server, local backups, and cloud backups. This helps to decrease the probability of losing data as all three locations would need to be destroyed.

Reason Two,
Speed... Did I mention speed? Backing up and restoring to/from the Crashplan Cloud can be slow (less than 20Mbps) depending on the type of data, time of day, etc. So while it is important to have an offsite backup you will want to do restores from a local server as often as possible due to the increase in performance. I have done local restores at speeds higher then 500Mbps.

Why three servers and not one doing it all...

As mentioned above I am not storing any data on the Crashplan01 server and instead I am sending/storing the data on two separate servers. Few reasons for this...

Reason One,
I can not stress how important this reason is so please read it and then read it again. Computer virus's and malware can be a real b**ch especially malware such as Cryptolocker. Hypothetically if I used just a single Windows server to backup/store all of my data via CIFS shares it would be very easy for the data on the FreeNAS Server to be encrypted along with your backups. Leaving you with only being able to recover from the Crashplan Cloud which would take a VERY LONG TIME depending on the amount of data that needed to be restored. Well wait how would the local backups get encrypted? Glad that you asked.. Lets say a user on the local network has a mapped network drive on there Windows PC and that user gets Cryptolocker on there PC. The malware is going to encrypt the data on the local PC and all of the data on that mapped network share. The problem is that your backup server (in my case Crashplan01) has that same network location mapped to it. Since that data is now infected with the malware it means that the malware can now encrypt all of the data that your backup server has access to including the locally stored backups. By configure a separate server which has no network drives and no access to the data on the FreeNAS Server you protect your backups from infection and can thus restore from a local backup.

Reason Two,
Save money!!! As I mentioned above my two storage servers are not using RAID there is just a single hard drive for each server. Generally speaking with data that you dont want to lose this is not a good idea... However in this case it works out since this is just backup data and the odds of the drive failing in the backup server and losing the data on the FreeNAS Server at the same time are low. To further minimize the risk I decided to use two storage servers that way I can store approximately 50% of my data on one drive and 50% on the other. That way if one of the drives does fail I only lose 50% of my backup data.

Reason Three,
The Crashplan Desktop Application will only let you choose one location to store what it refers to as inbound backup data (that is the data that is going to storage01 and storage02). Due to this and my desire to split the data between two drives I had to have two servers.

Crashplan Software and Crashplan Cloud...

Reliability,
I have been using the software for a number of years I have not really had any issues with the Crashplan software and its ability to backup my data.

Cost,
Its cheap especially since they give you unlimited data storage in there cloud... An individual PC would cost $5.00 per month and 10 PC's would cost $12.50 per month. In this example here you would only need to pay for the software on one PC the Crashplan01 server since this is the only device this is backing up to there cloud there would be no charge for the Storage01 or Storage02.

Support,
There technical support has been great!!! Even with this setup there tech support was very knowledgeable and able to understand what I was doing and assist me when needed.

Version Control and Datasets,
For both the local and cloud backups they give you the ability to virtually control all of the Crashplan Backups settings via the use of datasets. If you want to know more about what you can do with the software I would check out there website and support information. However I will tell you that you do get versioning like you were looking for.

Other,
This entire setup described here is 100% supported by Crashplan this is important... Know that the Crashplan software can be installed inside of a Jail on FreeNAS, but this is NOT supported by Crashplan and they DO NOT guarantee data integrity, I would not recommend using it inside of a Jail.

I think that this information should give you a pretty good starting point if you have any questions please feel free to let me know.
 
Status
Not open for further replies.
Top