home office dual TN safe backup ideas/recommendations

[Matt Rupert]

Hello Forum Users,

I am very new to Truenas/Freenas and am looking for guidance and recommendations from all of you who have been working with these systems.

I have slowly but now successfully built up two Truenas servers. The servers are old servers from SaveMyServer and are Dell R720xd's. The server 1 is running 96GB of ECC ram, and has 8 4TB HGST Megascale drives for Tank1. I have 4 3TB drives for Tank2. Server #2 is running 32GB of ECC and has only 8 4TB HGST Megascale drives. Both running 12.0 release and ZFS with the option for 3 parity drives. I know they are not parity but not sure what to call it for zfs.

My home/office is in a rural environment where I have good internet some days and others where there will no internet. Server 1 will remain powered on 24/7. I will be storing personal multimedia data and engineering ecad, cad, and cam data. It will provide file server storage and redundancy for my engineering machines, Plex services for my family, Zoneminder for security cameras on Tank2. Server 2 was built for backing up the files on Server 1 tank 1. The plan is to have its network connection directly connected to server 1 where the rest of the network is not physically connected to server 2 reducing the chance of anything bad getting to it. (good idea or not?) I dont care about backing up the Zoneminder tank 2 files. Server 1 is currently up and running and holding around 7TB of multimedia data. But its just in test mode as I have other storage of the data on multiple usb 3TB drives. If I need to completely redo this I still can at this point.

My two fears and the reason I built the systems as I did was to help protect myself from ransomware attacks as well as drive failures. With intermittent and limited bandwidth internet cloud storage is not an option for backup. I have though maybe a tape backup could augment one of the 2 servers. Though server 2 might not be necessary with a good tape system. My original though was to have server 2 powered up once a week to make a backup of server 1 tank1. Maybe using snapshot or something like that on one or both of the servers. However Im not sure that would successfully work if a ransomware attack was successful since I have just under 2x capacity. The used data will grow at a 1TB or so per year. The only thing I have been doing at this point to protect the server 1 from that is to login to the server when needed and not map it as a network drive. Not the most convenient for sure.

What ideas and recommendations do any of you have on how to back up the data, should I use snapshot or not. Are there plugins I should be using to backup my engineering machines user data? How about an automatic or even manual backup of the files from Server 1 to server 2 using plugins or something else. At this point I have been using Microsoft sync toy for making copies of the data only one direction.

Thank you for your advice and guidance!!
Matt

 

[JohnDigital]

I think you are on the right track. All TN users should be having some form of auto snapshots being taken in parallel with level of paranoia. Protection from all out loss can be achieved with some of the large USB3 drives that are available, i havent seen a tape rig in some time, but if this is feasible to you, go for it. On this USB or tape is mission critical items, complex databases, documents, configs, your vinyl collection you painstakingly ripped to flac over a year of lockdown and dont forget your family photos. 2 Servers with several snapshots thru time is pretty good against ransomware or other data or deletion catastrophe, and even more so if the backup server isnt powered 24/7. Ransomware/virus tends to target windows networks which is also beneficial in a BSD world. A hot (and cold) spare on hand for each server could further help protect against drive failure.

Once you have a working verified backup and known good snapshots, I wouldnt worry so much about mapping the disks but for those paranoid folks, reconsider "saving" of the creds for connection, Make sure to have sane permissions set, with strong passwords all around.

Your also smart thinking to have the second server behind the first not otherwise connected to the main network or the internet period (once configured). Definitely adds layer of complexity for a would be intruder. This is easiest with two NICs in server1. One on your main net, and one on a different net only common to the two servers connected via switch or crossover cable.

As far as backing up your working user data, ive used a ROBOCOPY (rsync is also available for windows) auto task in windows task manager to sync all user data to Server1 every X amount of time, each user has a dataset which only their machine/user can access, then at X time the two servers sync their files via rsync task. I started doing this for all users when work did the switch to Win10 a few years ago. If all mapped disks are common a single batch file works great here. Im doing this remotely from several sites now and its great, and it would work really great when all machines are in the same location. You can have these tasks go every hour on the hour if you want, up to you and how much data is changing day to day, for us its not too extreme.

There are many different ways to go about this, this being just one. Theres no real need for 3rd party plugins or tools to accomplish this, but if its easier and makes more sense to you, do it. There are many plugins and such available. I find simple rsync to be easily selective of what is backed up, its otherwise "included" and straightforward. Also data that is unchanged is skipped, cutting down drastically what is actually transferred. Sometimes ours are just a handful of files, sometimes its many GB. Replication is also a very valid option, but ive been doing it this way for years, and it works. Then basically when I think about it or major things have changed I pull out my USB and do a "just in case backup" and store it back in the fire safe. Then I know I wont lose my Bruce Springsteen records in case theres a torcano or Godzilla attacks or something.

Hope this helps.

 

[Matt Rupert]

Hello John Digital thank you so much for the response.

I want to clarify on what would be a good strategy going forward.

Some form of snapshots on both servers? Are snapshots like a first in first out kind of archiving mechanism. In that the oldest data once disk space runs low is the first to be deleted off then back end to make room for the most current data?

Tape or usb disk backup. Im going to keep my eye out for an LTO7 tape drive. Does TN/FN support any tape backup mechanism with any kind of plugin or 3rd part app to be placed in a jail? Hardware wise I dont have room internally on the R720xd for a tape. Does any one have an idea how this might be hooked up to a TN/FN machine? Most likely via Ethernet.

You mentioned the Ransomware/virus's tent to target the windows networks. So as a general rule of thumb, not mapping my TN server to a network drive and logging into it as needed is the safest way to operate? If I ended up mapping the drives, with snapshots running could I run into a situation where the old good data is pushed off the back side and lost as new ransomware effected data is written to the drive. This is presuming with snapshots running that data of a certain age gets deleted when disk space starts to get low. Is there a ratio that I need to be concerned with as far as free space to used space foes with running snapshots?

I also have 4 cold spares drives to deal with drive failure. You suggest making one of those hot? With already established tanks running can I add a hot spares? Or will I have to delete and reestablish the tanks?

So on the second server connected only to the first via the cross over cable or switch, my servers have the dual GB rj45s as well as the dual GB fiber on each of them. (ix1, ix1, igb0, igb1)

To run Robocopy or Rsync I think will require there to be a mapped network drive from the windows to TN machines? When doing it from TN1 to TN2 Im guessing its configure the network and then Ill maybe need to map TN2 on to TN1? Or will Rsync connect via a port and not need a mapped drive?

Again John thank you for the help and guidance!!!

Matt

 

[Samuel Tai]

Are snapshots like a first in first out kind of archiving mechanism. In that the oldest data once disk space runs low is the first to be deleted off then back end to make room for the most current data?

Manual snapshots are immortal until manually deleted. Snapshot tasks can have an expiration date set.

 

[JohnDigital]

So as my buddy Samuel says snapshots last forever until destroyed or they expire. Also something to note about snapshots, they are tiny as long as lots of data doesn't change. From snapshot to snapshot they are small but a snapshots taken before or during radical data shifts can get huge. Old snapshots grow as data changes. The easy way to prevent this, is to set an expiry date on the snapshot. After which it's destroyed automatically and replaced with a new, small, current snapshot. How long will it take you to notice a ransomware attack? Wife deleted my Springsteen records? My cherished Windows 95 isos? Our family photos? A few days maximum likely? Snapshots to the rescue.

So snapshots set to go every day at noon and midnight set to last a week would give you the ability to go back to the exact way your data was up to a week ago. This can be shortened or lengthened commensurate with paranoia and free disk space (as they do use some space). And yes I'd have snapshots rolling on both machines. Because knowing what we now now, why not?

Think of a snapshot as exactly that, a hi-res "picture" or snapshot of the surface of the drive. If you are attacked by ransomware today and TN has a "picture" of the drive from yesterday, and just restore this picture over the current picture, voila ransomware fixed, at incredible speed. Snapshots accumulated and eventually will use a ton of space, but avoid that by setting some intervals. You will not need snapshots to last more than 2 weeks in my humble opinion.

I'd go far away from tape, but it's just me. Again if this is in your wheelhouse, swing away.

Yes, not mapping disks to a vulnerable windows machine is probably the safest. If a determined hacker is in your TN or windows box with admin clearance, not having a disk mapped isn't going to stop much. With 2 servers and identical data, a tape backup, snapshots, sane permissions, and strong passwords all wrapped into a low value target (presumably) such as yourself is unlikely to be targeted, but it's not to say you wouldn't be affected in some blanket attack (think email attachment). Professional hackers want money not my Win95 isos.

Tl;dr Map all your drives, strong impossible passwords all around (email account, router, windows, truenas). Don't open fishy emails!

Just having several new disks around is enough. YOU WILL HAVE DISK FAILURES so be prepared for it. By a hot spare I just mean a disk in the machine connected and powered, but idle. You can assign them, but its not neccassary. Don't forget to setup your email alerts.

You can do the transfers between all machines and servers without a mapped drive.

Run a fiber line between the 2 servers for ultimate bandwidth.

Hey, your welcome. My practices are likely not end all. But as I like to say, whatever you decide will be right for you.

Hollar out if ya need anything further.

 

[Matt Rupert]

Hello Samuel and John,

Thank you guy for the input. Ill get these suggestions going on these machines. Like you said Im most likely a low risk target. Ill probably have more issues with hardware failures then attacks, but being prepared for either will help me sleep better at night.

Thank you again both of you!!!
Matt