How to implement daily backup from Windows Server to TN hot-swap SATA drive that is taken offsite 5 days/week?

[TillmanR]

INTRO: Hello TrueNAS community, this is my first post, though I have been using the forum as my main source of tech support for TrueNAS Core since mid-2022. I’m still a TrueNAS noob with zero Linux background and no scripting experience and have barely scratched the surface of TrueNAS/ZFS, only using the most basic local file server/backup repository functionality to date. However, I love learning and I thoroughly enjoyed building, troubleshooting, and configuring my two TrueNAS home-office servers and setting up the iX systems MiniX+ I purchased in mid-2023 as an onsite backup repository for my client, a small law office.

ISSUE: Sometime in 2024 I’ll be replacing my client’s Dell T320 Server 2012-R2 domain server with a new Server 2022 box and I’d like to expand the role of their TrueNAS system if my application is feasible. Note that I am in the Bradenton/Sarasota FL area and thus offsite backups are a mandatory part of our contingency planning. The current offsite leg of our backup schema uses Dell 1TB HDD cartridges written via a DataVault RD1000 drive mounted in the server. 10 cartridges are rotated such that the nine most recent are offsite, with the owner bringing in the oldest backup each day and swapping it with the current backup cartridge each evening before she leaves the office. She brings the newest backup home each night while the oldest cartridge is written over with the coming night’s backups. Backup images are written with Acronis Backup running on the server. The RD1000 was a royal pain to initially get functional with no intervention, and if possible, I’d like to replace it using one of the MiniX+ 2.5” hot-swap bays.

I searched the forums and read the several threads I found on similar-sounding topics but they did not match my use-case, mostly because I need a solution that can be implemented daily by office staff. I need my client to ONLY have to remove the current backup drive/caddy and replace it with another drive pre-mounted in a hot-swap caddy; nothing more complicated for her would suffice. The MiniX+ has five 3.5” hot-swap bays (only two are in use) and two 2.5” hot-swap bays. I’d like to make use of the 2.5” bays because the drive/caddy would be small and easy to transport. Two ideas come to mind, though I believe both would require a time-of-day based script to run after the drive was swapped and before the first backup launched (unless it could be triggered on a drive removal/replacement?), with the second idea requiring a replication (or similar function) script after backups complete. Once configured, manual intervention must be limited to the hot swapping of a single drive/caddy assembly.

1-The first thing that comes to mind is a mirror of the two 2.5” bays as the backup target, with one of the bays being designated as the one to be pulled & swapped nightly. With only two 2.5” bays there couldn’t be a hot-standby, and even with an adapter for one of the 3.5” bays, a three-way rotation with mixed caddies would be too complicated for the owner. QUESTION: Would resilvering 5 days/week cause a reliability issue or would this be ok for long-term practice? Additionally, 80% of the drive capacity (using the current HDD cartridge size of 1TB) would also be re-written every night, so including the resilvering that’s 150-200% drive capacity writes 5 days/week. Using 2TB drives and/or using SSDs could be considered but we are definitely cost-sensitive. If HDDs are used I assume we should use 2TB drives since the 1TB drives would start slowing down as they approached 80% each night, if I understand that issue correctly with ZFS (if I don’t, please correct me).

2-The second thing that comes to mind is using a mirrored pair on two of the 3.5” bays as the backup target and after all backups are complete, replicating the backup dataset to one of the 2.5” bays that would be setup as a single-drive vdev. I haven’t done anything with single-drive vdevs so I don’t know the ins & outs of this other than the obvious lack of redundancy. But I believe it would still have higher reliability than our current setup, which writes from Windows Server to a removable HDD cartridge with no redundancy and none of the resiliency that ZFS provides. I assume this would require two scripts, one to mount the inserted drive after the current backup drive is swapped, and one to replicate (or ZFS send/receive?) the backup dataset to the single drive vdev in the 2.5” bay after all backups finish.

I was not considering the use of any sort of external USB drive; please let me know if I should indeed be considering this. I realize my limited TrueNAS experience renders me unfamiliar with the majority of capabilities and tools available, so perhaps I’m overlooking something obvious, and I’m open to any approach to achieving my objective. I don’t expect anyone to write scripts for me, but guidance and/or examples used for a similar purpose would be very helpful and much appreciated. I’m open minded to any & all ideas and guidance, even if it turns out to be “TrueNAS is the wrong tool for that application”. I apologize for the lengthy post. Thank you for your help!! Roger

Show : My TreuNAS Severs

Home 1

• TrueNAS Core 13.0-U6 | SuperMicro H8DME-2 | 2 x Opteron 2356
• 80GB DDR2 ECC | Corsair 600T chassis | 1000W Corsair PSU
• 9211-8i w/IT fw + MoBo SATA ports
• Mirrored SSD boot pool
• Internal: 12 x 4TB RAID Z3 data pool (mix of NAS, consumer, shucked HDDs, all CMR)
• External: 2 x 4-HDD RAID Z1 vdevs in 4-bay port multiplier boxes via 2-port eSATA PCIe card

Home 2

• TrueNAS Core 13.0-U6 | SuperMicro X10SDV-2C-TLN2F | Pentium D1508
• 64GB DDR4 ECC | SilverStone DS380 chassis & 450W SFX PSU
• SU-SA3004 (ASM1061 PCIe 4ch SATA card) + MoBo SATA ports
• Mirrored SSD boot pool | WD-Blue NVMe SSD jail pool
• 16TB Mirror (2 x Tosh MG08), 8TB Mirror (2x SG IronWolf), 8TB Striped Mirror (4x SG NAS)

Client

• TrueNAS Core 13.0-U4 | iXsystems TrueNAS MiniX+ | Atom C3758
• 32GB DDR4 ECC | MoBo SATA ports
• Mirrored SATA-DOM boot-pool
• 6TB Mirrored data pool (2x WD-Red+)

 

[TillmanR]

Claaaass?... Anyone? Anyone?... Bueller?... Jgreco? Chris Moore? Joeschmuck? Patrick? Arwen? Ericloewe? Samuel T? Anyone??...

Can anyone please point me in the right direction, either in response to the content of my question or as to the deficiencies in my post and ways I can improve so that next time I will be more likely to receive a reply? (Perhaps I'm just cutout to be a lurker...)

 

[HoneyBadger]

Hey @TillmanR

I'd say you've asked a complicated question with a lot of moving parts (figuratively and literally) so it requires a fair bit of time to sink ones teeth into for a response that does it justice. A short answer might be seen as dismissive, and we don't want to be rude. :)

The challenge with the 2.5" bays would be finding drives with enough capacity - unless you have only a pair of 2TB drives in your 3.5" bays, anything larger than that is going to be using shingled magnetic recording (SMR) which is the antithesis of "good performance and small backup windows" - or it's going to be an SSD, whereupon having ten of them in the 4/8TB range might prove to be a rather expensive venture.

Add to that the logistical challenges of creating a script to "check for completion of resilver, start replication, wait for completion" and the potential for things to go Very Wrong if the drive gets pulled out early during a resilver/replication job.

To be honest, in your shoes, I'd be looking at some combination of

• Multiple daily snapshots (opening/halfway through/close of business)
• A second TrueNAS unit at an offsite, secured location - with replication over a secure WAN link/SSH/VPN tunnel
• Cloud/distributed storage provider like iX-StorJ
• Some or all of the above

This also mitigates the risk of the physical backup drives being damaged, lost, or stolen during their time out of the array.

Hopefully this helps spark some discussion. I know you're far from the only one doing periodic replication, but the need to fully automate and have it be "very hands off" introduces a number of unknowns. Heck, even the backplane's drive port physically wearing out over time could be an issue - hotswap isn't really intended for daily insert/remove cycles year-round, rather "when a drive fails during its life, it can be swapped"

 

[Arwen]

Your post was a bit too long for me to read during the work week. (I suffer from FBS, Fried Brain Syndrome . Even on the weekend I tend to answering simpler, (and mostly shorter), questions.

I wrote a Resource of how I do backups;

How to: Backup to local disks

How to: Backups to local disks Overview: This subject comes up often enough that I thought I would start a post of how I do it. Specifically I back up my TrueNAS to a hot-inserted, locally attached disk, then when done, hot-remove and take...

www.truenas.com

But, it is not meant to be plug and play for others. Just an example read for other to modify as desired.

Backups are a tricky thing. Everyone has their own ideas. Yours of using 2.5" disks adds in complications that @HoneyBadger mentions. Larger HDD models would be SMR, not suitable for ZFS. And SSDs might be too small, (or expensive), for practical usage.

What I can say, is this;

• Repeatability is key
• Document how you do backups
• Periodically test the backups to make sure you can restore if needed, (ZFS scrubs can help with this)
• Expect backup media to fail eventually, so perhaps rotate through multiple sets
• If their is private information on the disks, protect them somehow, (though encryption can be problematic)

You might start a shorter post with a single thought, and hardware setup, then ask what people think. That way if others are suffering from FBS they may still be able to answer.