Splitting a Vdev

[AVB]

I have a Pool comprised of an 8 drive Raid Z2 Vdev that grows in data by about 500GB a year. I'll call this Pool 2. I have about 18 months left before I hit 80% of fill for the pool. What I want to do is have two 4 drive Z1 Vdevs so I can upgrade only 4 drives to increase total Pool disc space instead of 8. If I take out the drives for my other Pool (Pool 1) in the server I have enough slots to do a copy of some kind to the 8 new drives internally in the server which will be MUCH faster than any other solution.

So the plan goes something like this:
Remove drives from Pool 1
Load 8 new drives (These are the same size as the old drives, upgrades will come later)
Make new Pool with 2 Vdevsl
Copy data From Pool 2 to new Pool

So my question is what is the most data safe way to do this? I know I can use the CP command, or I can use rsync or even snapshot but I don't know which of the 3 are the most accurate or if it even matters which I use. Any thoughts on this will be appreciated.

 

[Patrick M. Hausen]

I would not use RAIDZ1 with current size disks. It's generally discouraged.

 

[AVB]

I would not use RAIDZ1 with current size disks. It's generally discouraged.

Two questions come to mind, What size discs do you mean? The upgrade will be to 4TB drives which are considered rather small nowadays plus I have full backups. Do either of those affect your opinion?

 

[Arwen]

The general rule for avoiding RAID-5 & RAID-Z1, are disk sizes 1TB to 2TB. So, using 4TB disks with RAID-5 / RAID-Z1 in the future is also not really recommended.

The issue is that during a drive failure & replacement, you can have an un-recoverable read error on another disk. Thus, data loss. This has to do with bit error rate due to the amount of data. Remember, changing out a single disk in a RAID-5 / RAID-Z1, you have to read all the data on the other disks. (Though, in ZFS' case, only the used data, which could be much less than the total space.)

In the case of a full 4 x RAID-Z1 using 4TB disks, you would have to read 12TB, WITHOUT ERROR to recover a RAID-Z1 failed disk. But, the read error rate says you would likely get at least 1 failed block on one of the other 3 disks during that time. This does not mean YOU will get a failed block, just statistically you might.

To be fair to ZFS, your pool would not be 100% full. Plus, by default ZFS stores 2 copies of metadata irrespective of any vDev redundancy. So any metadata read that failed, could try again with the other copy. And critical metadata has even more copies. However, metadata & critical metadata are by their very nature generally far less amount of storage than actual data... Thus statistically less likely to have errors.

---

Anyway, you can see it's a complex topic. Some don't believe the "read error rate" and don't believe that RAID-5 / RAID-Z1 are dead with large disks.

Others of us prefer to error on the side of caution, (which is one reason why we use ZFS).

 

[AVB]

Thank you for the informative well thought out reply. My data is not critical as it is just media that can be replaced. I'll have to think about if having a video or audio glitch or two is worth the up-front cost to replace all the drives at once or if I can fit a drive every 2 months into my budget if I do one at a time.

Do you have any advice on the copy process?

 

[Patrick M. Hausen]

Copy process: nothing faster and more reliable than zfs send | zfs receive ... if you can have both pools online in parallel, that is.

 

[Arwen]

@AVB - I use ZFS for both OS pool & media pool on my miniature media server. The OS takes a small piece from each disk, (they are 1TB mSATA SSD & 2TB laptop drive), and mirrors them. But, the media pool is striped, no redundancy, (except the above mentioned metadata).

About once every 6 months I "loose" a media file. Twice a month ZFS scrubs check for, and if needed, tell me which file was lost. Since I have good backups, it is pretty easy, (though annoying), to restore the "lost" file.

That is the nice thing about ZFS. Not only can ZFS tell you about bad data, it helps you by telling you which file(s) to restore.

Once I even had an automatic repair. Took me a bit of time to wrap my head around HOW it could repair the faulty block automatically on a pool without redundancy. My conclusion was that the "lost" block was part of metadata, which has built in redundancy.