Migrating from hardware raid 5 to ZFS RaidZ1

[Camofelix]

Hi Everyone!

To start off, younger me was a little bit silly. around 3 years ago I build my first FreeNAS using an HP ML 350 G6 server that I bought used. it's raid card, an HP smart array p410i, does not support Jbod mode/it mode. still wanting to use the backplane for convenience I decided to use the build in raid5 and have FreeNAS use that one drive a a 1 drive pool.

It's been running rock solid for years now, but it's now time for a storage upgrade. I've now ordered an LSI HBA and am waiting for it to come in the post. my question is how best to move from the current raid 5 to a RaidZ1.

The raid 5 is built of 3*3TB wd reds with a working capacity of ~5.6TB. Rest of hardware is at the bottom of the post

I'll be adding 3 10TB seagate IronWolfs to the setup once I have the LSI card.

I'm also running a few Jails (Plex, torrents) but nothing too intense
and Running docker for a development web server.

It goes without saying that I don't want to loose any data. Whats the best way to do this? Would I be plugging in (one of?) the new ten terabyte drives, then performing a sort of clone?



2 Xeons 5620 quad cores @2.4Ghz, 144GB ECC registered RAM, 2 16gb Kingston flash drives in a mirror for boot, a 256GB NVME portioned into slog, zil and L2arc with 50 GB left over for wear levelling, 2 gigabit nics in an LACP Lagg (running into a managed switch)

 

[MikeyG]

Are you looking to have a pool made up of 2X RAIDZ1 vdevs of 3 drives each? Or were you just going to set up the 3 10TB drives in a RAIDZ1 vdev and call it a day? If the latter, should be easy enough to connect the new HBA and drives, create a new pool, and then transfer all the data using snapshots and zfs send/receive.

If you are wanting to add all 6 drives into a new pool (which seems a little pointless to me given the disparity in drive sizes), I think you'd have to offload the data someplace and then destroy the pool/RAID5 array before adding drives back to the HBA for FreeNAS.

You might get the feedback on this that RAIDZ1 is not enough protection, especially for 10TB drives, and to think about getting a 4th drive and create a RAIDZ2 vdev, but in the end that's up to you.

The only way I can think of to re-use your old drives + the new ones in a new pool without transferring the data to additional storage (like an external USB drive) would be to copy all data to one of the 10TB drives, create a (preferably) RAIDZ2 vdev with the remaining 5 (2 10TB drives and 3 TB drives), copy all data over to the new pool, then once the data is copied over, replace one of the 3TB drives with your now redundant 10TB drive (after the copy). This would leave you with a 5 drive RAIDZ2 vdev, however, you'd be limited to the space of 5 3TB drives until you replaced all the 3TB drives, which of course takes you from buying 3 10TB drives to 5. In the end, you'd end up with a spare 3TB drive, since you can't expand a vdev through drive addition. It's a bit convoluted and risky but should work. If you don't have data backups, or are feeling nervous about the whole thing I'd go with the first method of simply creating a new pool with the new drives and transferring the data, then decide what to do with your old drives later.

 

[Camofelix]

Are you looking to have a pool made up of 2X RAIDZ1 vdevs of 3 drives each? Or were you just going to set up the 3 10TB drives in a RAIDZ1 vdev and call it a day? If the latter, should be easy enough to connect the new HBA and drives, create a new pool, and then transfer all the data using snapshots and zfs send/receive.

If you are wanting to add all 6 drives into a new pool (which seems a little pointless to me given the disparity in drive sizes), I think you'd have to offload the data someplace and then destroy the pool/RAID5 array before adding drives back to the HBA for FreeNAS.

You might get the feedback on this that RAIDZ1 is not enough protection, especially for 10TB drives, and to think about getting a 4th drive and create a RAIDZ2 vdev, but in the end that's up to you.

The only way I can think of to re-use your old drives + the new ones in a new pool without transferring the data to additional storage (like an external USB drive) would be to copy all data to one of the 10TB drives, create a (preferably) RAIDZ2 vdev with the remaining 5 (2 10TB drives and 3 TB drives), copy all data over to the new pool, then once the data is copied over, replace one of the 3TB drives with your now redundant 10TB drive (after the copy). This would leave you with a 5 drive RAIDZ2 vdev, however, you'd be limited to the space of 5 3TB drives until you replaced all the 3TB drives, which of course takes you from buying 3 10TB drives to 5. In the end, you'd end up with a spare 3TB drive, since you can't expand a vdev through drive addition. It's a bit convoluted and risky but should work. If you don't have data backups, or are feeling nervous about the whole thing I'd go with the first method of simply creating a new pool with the new drives and transferring the data, then decide what to do with your old drives later.

After everything is transferred, I was hoping to reuse the 3 3TB drives as another raidz1 (Pool 1), dedicated purely to TimeMachine/general backups and personal data.

The Raid Z1 (pool 2) of the 3 10TB hard drives would become a media hub/dumping ground for anything and everything that is not super important. I'd also like to somehow have a full mirror of Pool 1 held on Pool 2.

For now the 10tb will only be holding general data that I don't overly care for, so won't bother with a 4th drive for now.

 

[MikeyG]

I'd also like to somehow have a full mirror of Pool 1 held on Pool 2.

For this you could create snapshots and have them replicated between pools. I think this is made easier with 11.3.

 

[Camofelix]

For this you could create snapshots and have them replicated between pools. I think this is made easier with 11.3.

Sounds good! as for making sure my VM's transfer over (not sure If doing a "clone" of an entire pool is possible?) Same thing for my jails

 

[MikeyG]

Snapshots with zfs send/receive is the closest thing I know of to cloning as it will create a block for block copy. You can do this recursively on a top level dataset or zvol.

 

[Camofelix]

('m probably getting this wrong but:)
I would connect the 3 10tb drives to my FreeNAS externally (I haven't shucked the drives yet), configure each drive as a vdev, use those 3 vdevs to form a raidz1. I would then perform a zfs snapshot of the entire pool recursively. (A this point I would have 2 pools, one using the hardware raid 5, the other using raidz1)
I would then remove install my hba card into the system, connect it to my backplane, and boot into freenas. I would then do a snapshot from the 10tb array back to the 3tb array.

At this point I would have migrated in full with VM's (?) from raid 5 to raid Z1 on the 3tb drives. At this point, I would shuck the 3 10tb drives, connect them to the backplane for the first time, configure each as it's own vdev, create a raidz1 of the 3 of them, and use that raidz1 as a second pool in the system.

From there I would transfer all the non core information from the 3tb array to the 10tb array. Once I'm satisfied with that, I would implement a recurring snapshot of the 3tb pool onto the 10tb pool.

 

[MikeyG]

Not exactly. vdev types (like RAIDZ1, RAIDZ2) are made up of drives. If you were to configure each drive as it's own vdev, that would be the equivalent of a RAID0 stripe and any drive lost would take out the whole pool. Additionally, you'd have the complexity of multiple vdevs to make up that stripe, instead of just one vdev. Also, creating using drives attached via USB is not recommended, as the USB converter might introduce a layer between the drive and FreeNAS, preventing it from seeing the actual drive characteristics (similar to your existing RAID5).

I would recommend the following:

1. Shuck the drives and connect via to HBA. Create RAIDZ1 vdev. Create your pool on top of said vdev.
2. Transfer data from existing RAID5 based pool to new RAIDZ1 based pool.
3. Do whatever you want with old drives, including connecting to HBA, creating another pool, transferring data back again, etc.

 

[Camofelix]

Not exactly. vdev types (like RAIDZ1, RAIDZ2) are made up of drives. If you were to configure each drive as it's own vdev, that would be the equivalent of a RAID0 stripe and any drive lost would take out the whole pool. Additionally, you'd have the complexity of multiple vdevs to make up that stripe, instead of just one vdev. Also, creating using drives attached via USB is not recommended, as the USB converter might introduce a layer between the drive and FreeNAS, preventing it from seeing the actual drive characteristics (similar to your existing RAID5).

I would recommend the following:

1. Shuck the drives and connect via to HBA. Create RAIDZ1 vdev. Create your pool on top of said vdev.
2. Transfer data from existing RAID5 based pool to new RAIDZ1 based pool.
3. Do whatever you want with old drives, including connecting to HBA, creating another pool, transferring data back again, etc.

Perfect!

I'm still waiting on my hba to arrive. in the meantime could I connect/configure the drives using the build in sata ports on my motherboard? Being as the HBA is the same as those ports and only passes through the drive, would I be able to connect them this way, pull the data from the Raid 5, then once the HBA arrives, connect up everything through it?

 

[MikeyG]

Yes, as long as they are operating as just standard sata ports and you aren't creating any hardware RAID arrays, FreeNAS doesn't care where the drives are plugged in.

 

[adrianwi]

If you don't want to lose data, RAIDZ1 isn't really advisable with drives over more than a couple of TB, and certainly not 10TB!

If you have this data backed up elsewhere, then maybe its a gamble you're willing to take, but the likelihood of errors rebuilding a pool should a 10TB drive fail in a 3-disk RAIDZ1 is much higher than I'd be willing to go.

 

[Camofelix]

Yes, as long as they are operating as just standard sata ports and you aren't creating any hardware RAID arrays, FreeNAS doesn't care where the drives are plugged in.

Sounds good! Off to the store to grab some extra sata cables, will report back!

 

[Camofelix]

If you don't want to lose data, RAIDZ1 isn't really advisable with drives over more than a couple of TB, and certainly not 10TB!

If you have this data backed up elsewhere, then maybe its a gamble you're willing to take, but the likelihood of errors rebuilding a pool should a 10TB drive fail in a 3-disk RAIDZ1 is much higher than I'd be willing to go.

The 10tb array is all for media dumping So not overly worried about it!