Recommendation for 4-bay drive setup.

[SeaFox]

I'm working on a new build right now. I already have motherboardboard/CPU/RAM and am reusing my existing chassis.
This is a 4 bay hot-swap case. I also can add two SSDs inside, but I'm really writing more about how to use the main storage bays.

I'm currently behind the times a bit (points at sig), and have not had to deal with ZFS at all.
My original plan was to do 4 x 4TB drive in a RaidZ1. But I've been hearing how the size of drives today makes using parity for an array a "dead end". The size of modern drives makes resilvering from a drive loss too taxing on the remaining disks. At the same time, I'm a bit annoyed at having to replace everything at once if I want to increase storage capacity.

Should I instead be dropping the "4x4" plan and look into doing two mirrored pairs, with the understanding I'll just be losing a flat half of total drive space for the mirroring? Should I even do four drives then? Would it be better to put my budget towards just two drives of higher capacity and then add two more later? From my understanding I can make a pool of just one vdev of two physcials drives, and then expand the pool with a second vdev later.

 

[Chris Moore]

But I've been hearing how the size of drives today makes using parity for an array a "dead end". The size of modern drives makes resilvering from a drive loss too taxing on the remaining disks.

That is not exactly correct. Just that you should not use single disk parity. Here is an article:

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

I use RAIDz2 (two parity drives) all the time and ZFS also allows RAIDz3 (3 drives of parity) so it is even more robust.

At the same time, I'm a bit annoyed at having to replace everything at once if I want to increase storage capacity.

How much capacity are you looking to get out of your system? You can use RAIDz2 on a four drive pool of 10TB drives and get 13TB of usable storage. The problem is that you get more 'bang for the buck' by using smaller drives that are less expensive and using more drives. Typically six to eight drives in RAIDz2 is the sweet spot, because of the cost of drives, it just takes a larger chassis.

Should I instead be dropping the "4x4" plan and look into doing two mirrored pairs,

Mirrored pairs still just give you one parity drive. It isn't better than a RAID with single drive parity. It is actually harder on the mirror drive that is being rebuilt from because the entire content of the drive must be read. The theory that more drives will fail at the same time is based on the theory that all the drives are old and about to fail anyhow and the extra stress of a rebuild will cause more drives to fail. It is best to plan to replace drives on some kind of schedule instead of waiting until they are of an advanced age and begin to fail at high rate from advanced age.

Should I even do four drives then?

I wouldn't

Would it be better to put my budget towards just two drives of higher capacity and then add two more later?

That depends on your tolerance to pool failure. If you just have a mirror, and one drive fails, all your data is on JUST ONE DRIVE. If that drive is in any way likely to fail, you could be out of luck. I just can't tolerate having a single drive of parity. In a RAIDz2 pool, if one drive fails, you still have a drive of parity, so you are still protected while the pool rebuilds the failed drive.

 

[SeaFox]

How much capacity are you looking to get out of your system? You can use RAIDz2 on a four drive pool of 10TB drives and get 13TB of usable storage.

I was looking for at least a dozen terabytes, and 4x10 would do that, but getting only 13 TB of usable space on buying 40 TB of storage seems a bit...

The problem is that you get more 'bang for the buck' by using smaller drives that are less expensive and using more drives. Typically six to eight drives in RAIDz2 is the sweet spot, because of the cost of drives, it just takes a larger chassis.

Yeah, in this case I'm trying to figure out what the best usage is for four-bay cases now. It's nice to have a 4U case filled with blocks of platters, but this is meant to be a small system. What does iXSystems do with the Mini now?

Mirrored pairs still just give you one parity drive. It isn't better than a RAID with single drive parity. It is actually harder on the mirror drive that is being rebuilt from because the entire content of the drive must be read.

Okay, so I guess RAIDz1 is still better than mirrored pairs then, since it's spreading the work out.

Should I even do four drives then?

I wouldn't

Maybe I misunderstood your response here. You would do just two (higher capacity) drives in a mirrored pair, instead of two pairs?
That's what I was asking. Saying you wouldn't do more drives just seems counter to the rest of your post saying that more drives are better. ;)

That depends on your tolerance to pool failure. If you just have a mirror, and one drive fails, all your data is on JUST ONE DRIVE. If that drive is in any way likely to fail, you could be out of luck. I just can't tolerate having a single drive of parity.

Probably a difference in how we use our NASes. For me the NAS is there for high availability with lower utility costs and better stability/security than leaving a Windows desktop PC on all the time. I don't expect the NAS to protect my data like a data center, because I never leave my only copy of anything important on the NAS (which is a mantra I've heard here a lot). If a drive fails, the hassle for me is the time recopying everything from the source machines. I would of course welcome something that makes the recovery process a little easier, but I furrow my brow at using the majority of my storage budget on redundancy protection.

I would be fine continuing to use UFS, but I need to upgrade FreeNAS itself for support-related reasons.

 

[Chris Moore]

I was looking for at least a dozen terabytes, and 4x10 would do that, but getting only 13 TB of usable space on buying 40 TB of storage seems a bit...

This calculator will let you see what kind of results you would get with different drive sizes and quantities of drives. It even lets you put the cost of the drives in to figure what the cost per TB of storage is:
https://wintelguy.com/zfs-calc.pl

What does iXSystems do with the Mini now?

For some situations, the Mini is still a fine system, but they also sell the Mini XL, with eight drive bays, for people that need more storage. I went through several cases as my storage needs increased over the years. My first system was a simple desktop computer with five 1TB drives in it. Sometimes, to grow, we must increase the size of the shell.

Maybe I misunderstood your response here. You would do just two (higher capacity) drives in a mirrored pair, instead of two pairs?
That's what I was asking. Saying you wouldn't do more drives just seems counter to the rest of your post saying that more drives are better.

I guess I wasn't clear. If I were building a storage pool, because I want to have two drives of parity, the smallest pool I would build would be a five drive RAIDz2 pool. One of the upgrades I made over the years, to my own storage, was to move from a desktop computer case that limited me to only five drives to a server chassis that allowed me to mount up to 12 drives. When I made that switch, I initially built a pool of six drives in RAIDz2 and later added a second vdev with an additional six drives to that pool.
If you remain with a server chassis that can only mount four drives, your options are limited. I would still choose a RAIDz2 arrangement as that allows a drive to fail while still protecting the data in the pool with one parity disk.

lower utility costs

Certainly, I have not had to pay for electricity in eight years, so low utility cost has not been a consideration for me.

I don't expect the NAS to protect my data like a data center, because I never leave my only copy of anything important on the NAS (which is a mantra I've heard here a lot). If a drive fails, the hassle for me is the time recopying everything from the source machines. I would of course welcome something that makes the recovery process a little easier, but I furrow my brow at using the majority of my storage budget on redundancy protection.

I think this is a difference also. I actually run two NAS systems one as the primary data-store and the second NAS as a backup to the first. I try to only keep important data on the NAS, not on the desktop or any other place because the NAS is highly redundant, and everything on it is copied onto another highly redundant NAS. My house would need to burn down for me to loose any of my data.

I would be fine continuing to use UFS, but I need to upgrade FreeNAS itself for support-related reasons.

Like I said before, it depends on your tolerance for risk. Based on what you have said, I think you would be tolerant of the risk involved in running your pool in RAIDz1 because you use your NAS not as a primary data-store, but as a backup where, if the storage pool did fail, you could reconstruct it and it would only be an inconvenience. If you did that, you could reach your target of 12TB usable with just four 6TB drives.

 

[SeaFox]

For some situations, the Mini is still a fine system, but they also sell the Mini XL, with eight drive bays, for people that need more storage. I went through several cases as my storage needs increased over the years. My first system was a simple desktop computer with five 1TB drives in it. Sometimes, to grow, we must increase the size of the shell.

True. But part of this build is a desire to keep things fairly small, efficient, and quiet. If I didn't care about any of these three things, I wouldn't have started this build because I already have something like that here you would approve of:

But that's 2008 hardware, and the Xeon D system I'm putting together now is even more powerful (on Passmark score at least) and doesn't sound like a hair dryer when it's idling.

Going back to that ZDNet article you linked, I noticed the part about rebuilding a RAID array and how with today's disk sizes you're likely to hit a non-readable sector in the parity of the rebuild process (which would make the rebuild fall flat on its face). This would cause a loss of the entire pool since the rebuild wouldn't be able to complete, the article says. If you were using mirrored pairs instead, wouldn't the effect be completely different? We wouldn't be rebuilding from parity then, we would just be copying files over from the remaining good drive to the new replacement. I would expect the results would just be the loss of the one file that occupied that unreadable disk sector, and not the whole burrito like with an array?

 

[Yorick]

In a pool of mirrors, if the drive you’re rebuilding from fails during the rebuild, your pool is down, and you will restore from backup. That’s because writes in a pool are distributed across vdevs.
Mirrors resilver faster than raidz - 1.5 to 3 times faster - and they give you much better IOPS, which is likely not a concern for you. You did say this will just hold backup files.

Because divergent opinions can be interesting, this guy advocates for pool of mirrors over raidz-anything: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

 

[gpsguy]

What the guy doesn't mention, is that 3 and 4-way mirrors are also possible.

Sure, most users wouldn't want to give up 2/3 of their storage to parity for a 3-way mirror. But it does allow for additional renundancy with mirrored vdevs.

this guy advocates for pool of mirrors over raidz-anything

 

[Chris Moore]

If I didn't care about any of these three things, I wouldn't have started this build because I already have something like that here you would approve of:

No. That is actually far too old, loud and power hungry for me to approve of. I like some used gear, all the hardware I have is used gear, but I try to stay in the three to five year age range.

If you were using mirrored pairs instead, wouldn't the effect be completely different?

No. The whole IDEA is that the stress of the rebuild will either cause another drive to fail, or a data error that had previously been undetected will be detected during the rebuild and the missing data caused by that data error will cause the rebuild to fail. If your pool disks have been being tested with short and long SMART tests and periodic scrubs have been being run, you should have no undetected data errors. A scrub does a full pool read in the same way that a resilver does, using the exact same code as a matter of fact, so the pool is being regularly exercised because you should do a scrub about once a month and I do long tests on all my disks weekly.
Anyhow, as an example, a RAIDz1 pool needs to read the portion of the data stored on the other disks to rebuild the parity disk. Lets say that the pool has 1TB of data stored in it and there are five disks. That would mean each disk contains approximately 256GB of data (actually a little more because of the checksum data, but lets not split hairs) and the resilver reads each of those disks in parallel. So, from the perspective of stress on the other disks, each source disk in the pool only needs to provide 256GB of data and they can't be read any faster than that data can be written to the disk that is being rebuild, so the rebuild target controls the rebuild speed, not the type of pool, that is important to remember because it is also true of rebuilding a mirror. Let me recap, in case I missed a point, each pool disk is only providing 256GB of data and they are not reading at full speed because the data can only be written at the speed that the rebuild target is able to write but, you only need to write about 256GB of data to the disk being resilvered because the data is spread among FIVE disks. If you have a very low power system that is memory and/or processor constrained, the calculation of parity data from those reads, might cause a slowdown, but the limit of rebuild speed should be the actual write to disk and you are only needing to write about one quarter of the total data to the disk being resilvered.
By contrast, in a pool made of a single mirror, which is what the OP is suggesting, the rebuild of a mirror, with 1TB of data in the pool, the ENTIRE 1TB of data must be read from the source disk, and the system still looks at the checksum data as the data is being read from disk, then that entire 1TB of data must be written to the rebuild target. The difference is that instead of reading 256GB of data from the rebuild source disks, you are reading 1TB of data AND you must write that entire 1TB of data to the destination disk. Writing 1TB of data (mechanically speaking) must take longer than writing 256GB of data.
It is my assertion that there are actually many cases where a rebuild of a RAIDZx pool will be faster and less stressful on the disks than a rebuild of a mirror. This is based on reasoning, logic and observations on my part. Not all things are equal, so there may be instances where a mirror will rebuild faster than a RAIDz, but I would not agree that it is always true that a mirror rebuilds faster and is less stressful on the disk involved and it is the stress on the source disk in a rebuild that is (in theory) the cause of the additional failure that might cause a rebuild to fail.
YMMV, no warranty.

 

[SeaFox]

No. The whole IDEA is that the stress of the rebuild will either cause another drive to fail, or a data error that had previously been undetected will be detected during the rebuild and the missing data caused by that data error will cause the rebuild to fail.

Okay. I was trying to find a way to mitigate the "this one sector failed, so your entire pool is unrecoverable" since I don't see why you can't just skip the data covered by the bad sectors and recover the rest at least. That's like refusing to read a book because you found an editing error in the first chapter.

Anyhow, as an example, a RAIDz1 pool needs to read the portion of the data stored on the other disks to rebuild the parity disk. Lets say that the pool has 1TB of data stored in it and there are five disks. That would mean each disk contains approximately 256GB of data (actually a little more because of the checksum data, but lets not split hairs) and the resilver reads each of those disks in parallel. So, from the perspective of stress on the other disks, each source disk in the pool only needs to provide 256GB of data and they can't be read any faster than that data can be written to the disk that is being rebuild, so the rebuild target controls the rebuild speed, not the type of pool, that is important to remember because it is also true of rebuilding a mirror.

So, rebuilding a RAIDz1 would be less likely to cause a stress-from-overwork situation on the other disks, because they are only having to do a quarter of the work now on the re-silver now (in the 5-disk example)?

So far I'm getting the impression 4-bay cases are obsolete for anything but small datastores.

 

[Chris Moore]

I was trying to find a way to mitigate the "this one sector failed, so your entire pool is unrecoverable" since I don't see why you can't just skip the data covered by the bad sectors and recover the rest at least.

Because the data in pool works kind of like a linked list. If it can't recover this, it can't access anything after that and it leaves the pool in an unrecoverable state. That is the reason that having good redundancy is needed for any kind of RAID array. Even hardware RAID arrays can fail and that is what the article was actually about. ZFS is somewhat more robust because the software is written better. I have seen cases where four drives out of seven were all failing and it was still possible to recover the pool with ZFS only loosing a file or two. A hardware RAID controller would have quit trying and everything would have been lost. ZFS is a fantastic tool, but it needs parity drives. A mirror is the bare minimum. If one drive fails, you lost your parity and there is no guarantee you can recover. That is why some people will run 3 way mirrors, so they can loose a drive and still have parity.

So, rebuilding a RAIDz1 would be less likely to cause a stress-from-overwork situation on the other disks, because they are only having to do a quarter of the work now on the re-silver now (in the 5-disk example)?

It just makes sense to me. Others may argue.

So far I'm getting the impression 4-bay cases are obsolete for anything but small datastores.

I feel that way.

 

[Yorick]

I feel that way.

What will this look like in 2026? Seagate will sell 50TB HDDs. That doesn't seem reasonable for raidz1. Is it reasonable for two mirror vdevs? Or for narrowly a three-way mirror with one slot unused.

 

[Chris Moore]

What will this look like in 2026?

That is five years from now, so it is difficult to say what may happen in that time. The rate at which SSD technology is increasing in capacity and becoming more reasonably priced may continue to the point that small, entry level systems, will be all SSD. With the reliability of SSD, we may be able to have a RAIDz1 of SSD in a 4 drive chassis. The future is difficult to see. I have some new dual actuator drives from Seagate that I am going to be testing in the near future. It is interesting technology for spinning disks because each 14TB drive shows up as two 7TB logical units on the SAS interface.

Here is a link to the published info on those drives: https://www.seagate.com/innovation/multi-actuator-hard-drives

I think mechanical disk drives will be around for a long time yet. I started my first FreeNAS system many years ago with only four 1TB drives and have expanded it year after year, building whole new systems many times now. Personally, I have not used less than twelve hard drives in a build for five years, even though I have access to high capacity drives, I am still using a larger number of 4TB drives in my system. I could easily have as much storage with far fewer drives, but I use the quantity of drives to obtain speed. I am looking at a reconfiguration to take my system from two vdevs to four vdevs in an effort to obtain better performance in my home system.
I see low drive count as a performance limiting factor, so I would not consider it. Not even if 50TB drives were available, unless some new and magically faster IO schema is developed, I would still want a large number of drives instead of a mirrored pair of very large drives. To me, it is about being able to get to the data quickly and that is currently limited by the number of IOPS which is controlled by the number of vdevs, which is limited by the number of disks. All things in harmony.

 

[Evertb1]

small, entry level systems,

A small system is not by default an entry level system but I get what you want to say. Anyway, I keep my spinning drives for 40000 to 45000 hours and then replace them. In fact about 10 months ago I replaced my WD Reds with Iron Wolfs. But if I was a betting guy I would bet that my next pool will be all SSD. But I don't have the need for huge amounts of storage.

 

[Yorick]

I keep my spinning drives for 40000 to 45000 hours

Very proactive of you :).

I am perversely looking forward to my first drive failure. It's been not even 2.5 years since I built my system, so another 2-3 years at minimum, I assume.

 

[Evertb1]

Very proactive of you :).

Just cold calculation I guess. I expect HDU's that survive the first couple of months, wil serve me for about five years. Though I have a couple of old drives that just refuse to die, going strong for ten years or more.

Anyway, I also expect that I will outgrow them at a certain point in time. So I buy drives of a size that is sufficient to form a pool of the size that I guesstimate will do for the coming five years. To me that is just common sense. My current pool is six years old and recieved its third set of drives the past year. I started it out with a bunch of old desktop drives (coming from my Windows Home Server) that I replaced with a set of WD Reds after 6 months.