SOLVED Drive backup using mirroring: how to restore?

[Fire-Dragon-DoL]

Hello!
My setup is the following: 1 vdev with 3 HDDs, all 3 mirror of each other (so 3 x 3TB HDD, total available space = 3TB).
2 Drives are just going to spin all the time, mirror of each other, that's the redundancy. The 3rd drive, I'll plug it in once every now and then (say: once per month) to resilver.

My question is: let's say that all of a sudden the server explodes, killing the 2 HDDs (I have the config in my email), if I restore the config in a new server, do I just need to attach the old device to the pool and it will "resurrect"? What's the correct procedure for restoring backup from a mirror device? Will it cause problem if it's a bit outdated data-wise (but it's the only drive that survived)?

Is this a good approach for backup, or should I just use RSYNC (but that means using a second computer to sync all the data, which is annoying to say the least).

 

[Basil Hendroff]

In principle, it should work. The issue I see is that resilvering can be slow for large drives taking 1-2 days to complete. During this period, your system is vulnerable. Lose the primary drive and you've potentially lost two days of work. Over the period of a year, that's 24 days or close to a month that your system is vulnerable. On top of that, you have 12 x manual interventions throughout the year. If your drive bays are not hot swap, that's 12 times you've got to schedule server downtime. A second server doesn't seem so annoying anymore.

 

[Basil Hendroff]

If keeping the cost down is important, a compromise might be to use the technique you've described when there are key structural changes to your FreeNAS ecosystem eg. installing a new plugin, etc. What's really key though is your data. So maybe figure out an approach to copy your data off to an external location on a regular basis. This should help minimise any risk, downtime or intervention on your part.

 

[Fire-Dragon-DoL]

Hey @Seymour Butt , this is a home server, so in reality downtimes are not a problem at all. That being said, I bought a used server, so all 24 slots are hot swap!
Resilvering takes ~6.5 hours on 3TB if no data has changed (tested yesterday), which is acceptable.

I am considering the rsync alternative, but for whatever reason my laptop connection over ethernet crashes in case after a while and requires manual reset, so that requires using wifi (slower), which is annoying to say the least, or fixing it (more annoying, but I'd like to).

With rsync, I'm also not sure if everything will be backed up properly, since I might lose permissions/acl on some files (related to the jails mainly), not a big deal but worth considering.

Is there a version where I "resilver" across the ethernet cable? That would make my life easier

 

[Basil Hendroff]

With hot-swap, you won't have any downtime. So the technique you're suggesting, though somewhat unorthodox, is sounding pretty good now. Bear in mind though, your arrangement still remains vulnerable though during the three-way mirror, that's at least 12 x 6.5hrs or about 3 days in a year. The way around this is a rotation of two disks for the third drive. Getting back to your original question...

My question is: let's say that all of a sudden the server explodes, killing the 2 HDDs (I have the config in my email), if I restore the config in a new server, do I just need to attach the old device to the pool and it will "resurrect"? What's the correct procedure for restoring backup from a mirror device? Will it cause problem if it's a bit outdated data-wise (but it's the only drive that survived)?

I think it's just a matter of reconnecting the pool from the third disk https://www.ixsystems.com/documentation/freenas/11.2/storage.html#export-disconnect-a-pool after you've restored the config. It shouldn't cause a problem, though I'm not sure what happens if there have been major structural changes between backups eg. jail creation, new plugins added, etc. Maybe a good idea to do a three-way mirror when there are significant structural changes as well.
Of course, you should test the whole arrangement first rather than accept the theory prima facie. If it all works as expected, the only real issue that still exists will be the loss of data created between three-way mirrors. I'll leave that one for you to ponder further.

 

[Fire-Dragon-DoL]

Well, my concern is with reducing data loss. Files get added and they stay there for long time, but the "latest added" are usually on other devices too (laptop, pc, phone), so if i lose recent data, that's not usually a problem.

So, considering my pool has already live data, what can I do to test the 3rd hdd "restore", should I spin up a vm and restore my freenas config AND load that 3rd hdd into the pool?

 

[Basil Hendroff]

Unfortunately, I don't have any personal experience with 3-way mirrors, nor with what you're attempting to achieve. What you're suggesting sounds plausible though. I'd be curious, as would other folk, to know how you go and what issues you come up with.

I'm just not sure, when you detach the 3rd disk from the 3-way mirror, what happens when you attempt to import it into its own pool. This scenario isn't covered in the documentation.

 

[Fire-Dragon-DoL]

Unfortunately, I don't have any personal experience with 3-way mirrors, nor with what you're attempting to achieve. What you're suggesting sounds plausible though. I'd be curious, as would other folk, to know how you go and what issues you come up with.

I'm just not sure, when you detach the 3rd disk from the 3-way mirror, what happens when you attempt to import it into its own pool. This scenario isn't covered in the documentation.

Oh, I did that already (with the command line).

When I detached it, nothing happened. The command zpool status showed 2 disks instead of 3, but the pool wasn't degraded (I used zpool detach command to remove the drive).
Then, zpool attach to reattach the drive. The pool was considered degraded now and it started resilvering with a 6.5 hours estimated. Estimate was indeed perfect, and the pool was back to 3 drives.

Hence my guess, if I have a pool and add a drive to it, there is nothing to resilver (the pool is empty) and should start from there...maybe!

 

[Basil Hendroff]

Well, it's good to know now that a 3-way mirror can be reverted to a 2-way mirror. I wasn't sure about that. Now the act of reattaching the drive initiates a resilver, but I'm not sure what the act of detaching does. Does it, for instance, destroy partition tables on that drive? I guess we'll know more when you try to add it to its own pool. You've certainly aroused my curiosity now.

 

[Fire-Dragon-DoL]

Well, it's good to know now that a 3-way mirror can be reverted to a 2-way mirror. I wasn't sure about that. Now the act of reattaching the drive initiates a resilver, but I'm not sure what the act of detaching does. Does it, for instance, destroy partition tables on that drive? I guess we'll know more when you try to add it to its own pool. You've certainly aroused my curiosity now.

I'm guessing it didn't touch it, but the timing (6.5 hours) equals to running a full data copy at 100MB write speed on an HDD, so potentially, it did copy over all the data again.

I guess I have to experiment!

 

[pschatz100]

Hello!
My setup is the following: 1 vdev with 3 HDDs, all 3 mirror of each other (so 3 x 3TB HDD, total available space = 3TB).
2 Drives are just going to spin all the time, mirror of each other, that's the redundancy. The 3rd drive, I'll plug it in once every now and then (say: once per month) to resilver.

My question is: let's say that all of a sudden the server explodes, killing the 2 HDDs (I have the config in my email), if I restore the config in a new server, do I just need to attach the old device to the pool and it will "resurrect"? What's the correct procedure for restoring backup from a mirror device? Will it cause problem if it's a bit outdated data-wise (but it's the only drive that survived)?

Is this a good approach for backup, or should I just use RSYNC (but that means using a second computer to sync all the data, which is annoying to say the least).

I don't understand your whole strategy. This means you are always running your volume in a degraded state - which is not a good idea.

I would run a two-disk mirror in your system. You could mount the third disk as a separate volume in another bay or in an external enclosure and RSYNC or copy to it as appropriate. There is no need for a second computer.

 

[Fire-Dragon-DoL]

I don't understand your whole strategy. This means you are always running your volume in a degraded state - which is not a good idea.

I would run a two-disk mirror in your system. You could mount the third disk as a separate volume in another bay or in an external enclosure and RSYNC or copy to it as appropriate. There is no need for a second computer.

That's a good point too! I assumed the mirroring would be faster. I'm afraid of losing permissions (windows permissions) with rsync. I guerss I can live with it though, I care about the data really

 

[danb35]

I'm afraid of losing permissions (windows permissions) with rsync.

Have you thought about replication?

 

[Fire-Dragon-DoL]

Have you thought about replication?

I thought about it. Would that work with an HDD usb-connected to the server?

 

[Fire-Dragon-DoL]

Ok, I went with replication and it works great.
The major annoyance is that I need to point out what's the previous snapshot to perform the backup (and the next one), so I'll need to keep track of the previous one.
The cool part is that I will backup only from snapshot-to-snapshot, which means it will copy very little data effectively.

I manually run a script (no automation, since I unplug the drive from the server), something along the lines of:

Code:

zfs send -R -i "MARS" "MARS@wdblue_20190226" | pv | zfs receive -Fd "BMARS"

next time it will start from the "wdblue_20190226" snapshot instead of "MARS" (beginning)