Weekly rotating disk offsite backups for a small business?

[2nd-in-charge]

Hi All,

I'm new here, and new to FreeNAS, so please be gentle :)
By way of background, I work in a small company (~10 employees, including six s/w developers and an art director). We've been using a linux server for the past few years. It's a file server which also hosts mercurial source code database. The server had no redundancy, and its hdd failed recently. The system admin is a knowledgeable part timer (full time engineer, part time admin). He's been running manual monthly backups, so he restored most of the data from backups and brought the server back to life. It took a fair amount of time to restore the server though, most of it spent on restoring configuration. The team productivity suffered as a result of this server downtime. After that he decided that there is time for a new server, chose FreeNAS as the OS (no qualms from this forum, I suppose) and asked me to review the h/w specs for the new machine. I knew nothing about FreeNAS, so it's been a steep learning curve, and this forum has been great learning resource. The h/w is still under review, and I intend to post the specs in the appropriate section.

I want to start with asking this forum's opinion about our proposed server setup and backup procedure. I should probably mention that our internet connection is ADSL, and there is no second location to place a replication server is, so remote replication is probably not an option. The admin is a busy part timer, so it's important that the backup procedure is simple and not time consuming. If we need to script something it's not a problem, the admin and I are reasonably proficient.

Here is the procedure:

• zpool1 - formed by two 5Tb drives in mirror configuration (drives 1 and 2)
• zpool2 - formed by single 5Tb drive, exported and improted weekly, used for backups, with drives 3 and 4 on alternate weeks.

Regular automated tasks on the server:

• Every weekday night: snapshot (save the state after each work day).
• Friday night: local replication of Friday snapshot from zpool1 to zpool2
• Saturday: scrub on zpool1
• Sunday: scrub on zpool2
• Sunday night: delete snapshots that are more than 4 week old

System administrator actions:

1. Friday afternoon: insert disk 3 (or 4), import pool2
2. Monday morning: export zpool2, remove disk 3 (or 4), place a backup date label.
3. Monday evening: take disk 3 (or 4) offsite
4. Tuesday morning: bring the other backup to the office.

I found a 3 year old thread that mentions a similar method, but I'm not sure if information in it is still valid, for example if it is still difficult to export and import encrypted disks.

https://forums.freenas.org/index.ph...ks-from-single-freenas-primary-storage.17316/

We do use encryption (currently cryptsetup on linux) and intend to use this feature of FreeNAS.

It it a good method?
Can it be improved?
Anything else we should consider instead?

Thank you for your help!

 

[MatthewSteinhoff]

That procedure makes me SO TIRED.

Any manual procedure is too much. FreeNAS has amazing snapshotting and replicated built-in. You need two FreeNAS servers. Don't let that scare you.

You need neither a lot of storage nor a lot of processing power. Take a look at the Dell C2100/FS12-TY. You can be all-in with blazing performance for less than a grand a piece. (Probably less than $650 a piece if you have spare drives around the office.)

Set both up in your office. Get replication working. You've got redundant storage on the primary. You've got redundant storage on the secondary. Snapshot every hour not just every day. Keep two weeks or more of snapshots. Snapshots roll-back nearly instantly. Lose an entire server? It shouldn't take but half an hour to bring the replication target up as the primary server.

That's probably all you need. You had one failure in how many years? And it was hardware-related? This solves that problem and then some. Are you really worried about losing the primary server and the backup server in the same event? If that event happened, is your building still standing? Do you still have a staff?

But you still want off-site replication? Put the replication target offsite. Stick it in the owner's house. Or maybe the geekiest employee's house. Or spend some money and have the server co-located in a real data center.

You're a code shop? Code is small. Replication is extremely efficient. There is no reason you couldn't replicate all code changes and even the items from the art department overnight in a couple hours. Probably less.

Depending on who is hosting the replication target and their sensitivity to technology, you may want a smaller, less server-like machine for the replication target to better fit into their home office.

Cheers,
Matt

 

[2nd-in-charge]

Hi Matt, thank you for your reply!

You need two FreeNAS servers. Don't let that scare you.

We were considering two servers. For the replication/failover server I was thinking of building a G4400-based machine with single-disk detachable pool for offsite backups. Even though the second server would not have redundancy, overall we'd have four copies of the same data.

Set both up in your office. Get replication working. You've got redundant storage on the primary. You've got redundant storage on the secondary. Snapshot every hour not just every day. Keep two weeks or more of snapshots. Snapshots roll-back nearly instantly. Lose an entire server? It shouldn't take but half an hour to bring the replication target up as the primary server.

You've answered a lot of questions I was going to ask. Hourly snapshots sound great!

That's probably all you need. You had one failure in how many years? And it was hardware-related? This solves that problem and then some.

I know, that's why I'm here :)

Are you really worried about losing the primary server and the backup server in the same event? If that event happened, is your building still standing? Do you still have a staff?

I don't know, but I'd rather have offsite backups just in case..

You're a code shop? Code is small. Replication is extremely efficient. There is no reason you couldn't replicate all code changes and even the items from the art department overnight in a couple hours. Probably less.

The Art Director has a couple of terabytes worth of stuff he and his contractors produced over the years. We also use mercurial large files extension. A project usually contains hundreds of megabytes worth of images. ADSL connection here is only 1Mbit/s upstream, it will take over two hours to send one gigabyte. I think hourly replication is out of the question if the second server is offsite. Maybe we can get away with incremental repliction over the weekend.

You need neither a lot of storage nor a lot of processing power. Take a look at the Dell C2100/FS12-TY. You can be all-in with blazing performance for less than a grand a piece. (Probably less than $650 a piece if you have spare drives around the office.)

That sounds interesting, and it's attractive that it's all pre-flashed with the right firmware, but it might be a bit too heavy to be shipped to Australia.

 

[Robert Trevellyan]

ADSL connection here is only 1Mbit/s upstream

Unfortunately this makes offsite backup of a significant volume of data via internet extremely painful. You might have to settle for some kind of eSATA external storage disk rotation strategy.

 

[2nd-in-charge]

Hi Robert, does it have to be eSATA? Can I just use a hot swap bay to rotate disks?

 

[Robert Trevellyan]

Hi Robert, does it have to be eSATA? Can I just use a hot swap bay to rotate disks?

No, yes.
;)

 

[MatthewSteinhoff]

The Art Director has a couple of terabytes worth of stuff he and his contractors produced over the years. We also use mercurial large files extension. A project usually contains hundreds of megabytes worth of images. ADSL connection here is only 1Mbit/s upstream, it will take over two hours to send one gigabyte. I think hourly replication is out of the question if the second server is offsite. Maybe we can get away with incremental repliction over the weekend.

Let me blow your mind... You can do hourly replication all day while folks are working but schedule the offsite snapshots to replicate outside of normal business hours. That way you're not clogging the pipe while people are internetting.

Replication is very efficient with large files such as Virtual Machines. For example, we have a 60-gig VM. FreeNAS sees that as a single, very large file. As an active VM, parts of the file are updated all day long. Only changes made to that VM are replicated as part of the snapshots.

So, while the initial 60-gig transfer would be darn near impossible at 1mpbs (six days!), tossing just the snapshots is child's play. The largest snapshot we've sent for that FreeNAS pool in the last 30 days is just 58M (seven minutes, give or take).

Before you decide off-site replication won't work over your DSL line, @2nd-in-charge, look at the snapshots ZFS is creating. That's all you have to send after the initial replication. You can do the initial sync in the office over ethernet. That'll take care of the terabytes of existing data. I have no experience here with the Mercurial CMS but I think you'll be pleasantly surprised.

Cheers,
Matt

 

[Robert Trevellyan]

Before you decide off-site replication won't work over your DSL line, @2nd-in-charge, look at the snapshots ZFS is creating.

This is good advice. I suspect the workload hinted at will generate some huge deltas, but if not, so much the better.

 

[2nd-in-charge]

Depending on who is hosting the replication target and their sensitivity to technology, you may want a smaller, less server-like machine for the replication target to better fit into their home office.

Inspired by your C2100 recommendation, we started looking at the 11th generation Dell servers and bought a Dell PowerEdge T710 with 2xE5640 and 48Gb RAM. The dealer kindly replaced the Perc 6i with an H200 as well. 34kg shipping weight, an impressive unit.

Having plugged it in, I know what you mean by "sensitivity to technology". It might look like a long computer, but it sure sounds like a server!

 

[John Doe]

That procedure makes me SO TIRED.

Any manual procedure is too much. FreeNAS has amazing snapshotting and replicated built-in. You need two FreeNAS servers. Don't let that scare you.

...
Cheers,
Matt

2 NAS servers are not a backup.
A simple virus will kill or encrypt all your data you have on those 2 FreeNAS servers.

That's why offsite backup solutions are there for.
The plan of the threadstarter is a better plan.

 

[bigphil]

2 NAS servers are not a backup.
A simple virus will kill or encrypt all your data you have on those 2 FreeNAS servers.

That's why offsite backup solutions are there for.
The plan of the threadstarter is a better plan.

Unless the virus has the ability to make a snapshot writable...you're wrong.

 

[John Doe]

Unless the virus has the ability to make a snapshot writable...you're wrong.

With root access everything is possible. Even deleting and overwriting the snapshots or manipulating the ZFS drivers/modules. Reformating is also possible.

Also keep in mind, a lightning strike can also kill both of your NAS if they are connected to the power grid. A simple power switch doesn't protect your hardware.

EDIT:
And one more thing.
There are already malware in the free wild that specifically address specific NAS systems. FreeNAS could be on the target list too.

 

[bigphil]

If your root password is compromised, you should probably not be in control of it in the first place. Sure, anything is possible...I mean, we landed on the moon. But to suggest that "a simple" virus is going to wipe out your file system + snapshots to justify manual rotating disks is a bit of a stretch. NAS to NAS replication is the preferred backup method for the biggest names in the business...NetApp, EMC, Dell, etc. Also, the suggestion was to have an off-site secondary NAS to replicate to. And who has their NAS connected to a power switch or something cheap without a proper surge and UPS protection...also shouldn't be in charge.

 

[John Doe]

If your root password is compromised, you should probably not be in control of it in the first place.

Do you want to say, that you have never heared about zero day exploits?

Sure, anything is possible...

Exactly. That's why you make backups.
A NAS is never a backup.
A NAS is data availability, it is never data security.
Every IT professional knows that and learns that right at the beginning.

NAS to NAS replication is the preferred backup method for the biggest names in the business...NetApp, EMC, Dell, etc.

They also make backups because they know that NAS is not data security.

Also, the suggestion was to have an off-site secondary NAS to replicate to.

That might protect your data against a lightning strike, but not against malware or human failure.

And who has their NAS connected to a power switch or something cheap without a proper surge and UPS protection...also shouldn't be in charge.

It depends on if you run a company or use your NAS at home.

 

[MatthewSteinhoff]

A simple virus will kill or encrypt all your data you have on those 2 FreeNAS servers.

Help me out, Doe. How would a simple virus - or even a complicated virus - kill or encrypt all the data on the replication target given the most basic of security best practices?

Cheers,
Matt

 

[Ericloewe]

Help me out, Doe. How would a simple virus - or even a complicated virus - kill or encrypt all the data on the replication target given the most basic of security best practices?

Cheers,
Matt

We're not talking stuxnet levels of complexity, but it would take quite a complex attack to do anything, given basic security precautions.

The simplest way would be:

• Compromise a client - this part is not too difficult
• Identify the scenario and determine that you'll need FreeNAS' root password (harder than your average script kiddie can handle)
• Obtain said root password from the client (Very hard with decent security)
• Use the client to launch an SSH session (easy, given the above)
• Do evil stuff

The important part is that this would have to be a targeted attack with human intervention, and even then it would be highly non-trivial.

 

[MatthewSteinhoff]

Obtain said root password from the client (Very hard with decent security)

Yep. At that point it's game over. Toss it all in the river.

I'm still curious to know how Doe thinks an off-site backup will protect data when the intruder has root. Maybe he's confusing off-site with offline?

Cheers,
Matt

 

[Stux]

I agree with the triple disc rotation plan making me tired.

FWIW, we used to do this exactly 6 or so years ago. Mirror, with 3 discs rotated offsite. That didn't last.

In the end we built an automated offsite replication. Because the only way to go is automate.

The trick is to do the initial sync locally. Then as long as your delta is capable of being uploaded in a week, you're fine.

FWIW2, a FreeNAS Mini would probably make a good offsite target.

FWIW3, ours is at the directors house.