Z2 or Z3 for SSD pool?

[Patrick_3000]

I'm in the process of migrating my pool on Scale from hard drive (3-way mirror of 12 TB drives) to SSD using 4 TB NVME modules. I'm deciding between Z2 and Z3. That is:

5 x 4 TB SSD modules @ RAID Z2 ==> 12 TB capacity; or

6 x 4 TB SSD modules @ RAID Z3 ==> 12 TB capacity.

I'd rather save the cost of buying an extra SSD module and go with Z2 if it's safe enough. The argument I've seen for Z3 is that after a failure, resilvering could take several days, and additional drives could fail during the resilvering process, and Z3 provides some cushion against that. However, those discussions mostly involved hard drives. I would guess that with SSD, resilvering would be much faster, probably a few hours. Does anyone know or have thoughts on this? (By the way, I also have a backup pool in another Scale server, so even if the pool failed, I'd likely be able to recover the data).

On the other hand, if there any strong arguments for Z3, I would be interested to hear them.

 

[NickF]

Not a direct answer to your question. Please describe your use case and your current hardware. This is important information to have to answer the direct question

 

[Patrick_3000]

Not a direct answer to your question. Please describe your use case and your current hardware. This is important information to have to answer the direct question

Hardware is ASRock Rack x570d4u-2L2T, Ryzen 7 Pro 5750G, 128 GB ECC memory. Use case is home storage but rather critical, includes not only personal data but critical work-related documents and a lot of my spouse's work-related data. Also runs two VMs.

 

[NickF]

Use case is home storage but rather critical, includes not only personal data but critical work-related documents and a lot of my spouse's work-related data. Also runs two VMs.

Just a general document repository?

 

[Patrick_3000]

Just a general document repository?

Yes, but the stuff is rather critical, especially the documents for my spouse's home-based business and for my job, and a bunch of other personal data. Also contains the virtual disks for two VMs.

 

[NickF]

I would say save the money regarding NVME. Use the savings and buy 6 SATA SSDs instead of 5 NVME drives.

 

[Patrick_3000]

I would say save the money regarding NVME. Use the savings and buy 6 SATA SSDs instead of 5 NVME drives.

There is no savings. They're virtually the same cost. Cheap NVME SSDs, like Teamgroup and Kingston, are down to the $160-$170 range for 4 TB, and Crucial is around $200. SATA is about the same.

No reason to go SATA. There's no cost savings. They're slower. And you've got to deal with power and data cables dangling all over inside the case.

But it sounds like you're suggesting 6 drives, i.e. Z3.

 

[NickF]

24 PCI-E lanes is alot of lanes for a consumer platform.

 

[Patrick_3000]

24 PCI-E lanes is alot of lanes for a consumer platform.

I don't know enough about motherboard design to know if there would be potential lane congestion, but there are two open M.2 NVME slots on board, and I have an open x16 PCIE slot that can handle a quad SSD card, so I doubt there will be any problems. Also, even if there is lane congestion, it would almost certainly be faster than SATA. Moreover, avoiding cables alone would make NVME worth it to me, not to mention the fact that NVME is more power efficient.

All of that being said, my question isn't NVME vs. SATA. My question is RAID Z2 vs. RAID Z3. Related to that, am I correct in assuming that after a drive failure, resilvering would be a relatively quick process with SSD? I'm guessing a few hours at most. If that's the case, then I think Z2 is likely adequate.

 

[NickF]

I understand the question and I get why you are asking it. I promise I’m trying to be helpful.
I am asking other questions that are related about your config choices because 5 or 6 NVME drives on a consumer board is not a typical deployment.

Nvme presents some challenges because it shifts the bottleneck from the disks in your system to your system itself.

A RAIDZ3 will perform worse than a RAIDZ2. The problem is due to the velocity of data that NVME is capable that additional parity calculation overhead is likely exponentially rather than linearly more expensive.

A RAIDZ3 will provide more data integrity sure, but at what cost? It’d be my assertion that the computation overhead costs of parity and compression on NVME on a consumer platform are great enough to question using NVME in the first place. If nothing else you will make your CPU work harder and consume more power than you really need it to.

Then let’s consider RAM throughput. In ZFS particularly with fast drives like NVME, the speed and latency of your ram will become a bottleneck. Where when and why depends on your use case.

 

[ChrisRJ]

There is no savings. They're virtually the same cost. Cheap NVME SSDs, like Teamgroup and Kingston, are down to the $160-$170 range for 4 TB, and Crucial is around $200. SATA is about the same.

If you mention these models because the intention is to buy them, that does not match with the repeated statement of how critical the data are, IMHO.

Apart from a really good, i.e. tested monthly, backup, I would go for high-end SSDs.

 

[Patrick_3000]

I understand the question and I get why you are asking it. I promise I’m trying to be helpful.
I am asking other questions that are related about your config choices because 5 or 6 NVME drives on a consumer board is not a typical deployment.

Nvme presents some challenges because it shifts the bottleneck from the disks in your system to your system itself.

A RAIDZ3 will perform worse than a RAIDZ2. The problem is due to the velocity of data that NVME is capable that additional parity calculation overhead is likely exponentially rather than linearly more expensive.

A RAIDZ3 will provide more data integrity sure, but at what cost? It’d be my assertion that the computation overhead costs of parity and compression on NVME on a consumer platform are great enough to question using NVME in the first place. If nothing else you will make your CPU work harder and consume more power than you really need it to.

Then let’s consider RAM throughput. In ZFS particularly with fast drives like NVME, the speed and latency of your ram will become a bottleneck. Where when and why depends on your use case.

Thanks for your input. The fact is that SATA is not going to work for me. First, it's a lot more hassle, just from a maintenance standpoint. Six SATA SSDs would be 12 cables to manage inside the case, since each one requires a data and a power cable. Sometimes cables become loose and need to be tightened or replaced. I deal with that now with three SATA hard drives, which is bad enough.

NVME is clean and easy to maintain. A 6 x NVME setup requires zero cables, instead of 12 cables. One of the reasons I'm switching to SSD in the first place is to eliminate cables inside my case and thus have easier maintenance and greater reliability.

Moreover, I have a big drive cage inside my case, which needs to be removed to detached reach certain parts of the motherboard. Once I switch to an all NVME pool, I'll be able to remove the drive cage permanently, which will make it easier to work inside the case on the occasions when that's necessary and will allow me to, down the road, install a large and thus quieter CPU fan since I'll have more space to do so.

The bottom line is that, at least for me, there is no question of SATA SSD. It wouldn't even be worth switching to it as it would be more of a hassle to maintain than the current setup I have with three HDDs. Sure, it would be faster, but it wouldn't be worth it given the maintenance hassle.

Also, I've seen plenty of threads in other place plenty where people with consumer boards have x5 or x6 NVME setups. I really don't think it will be a problem.

My choice, realistically, is between an NVME SSD pool and my current HDD pool, and If I go NVME, then my choice is between x6 (Z3) and x5 (z2).

I hadn't realized that Z2 would be faster than Z3 due to computational overhead. Thanks for pointing that out as it's not something I'd considered. Although Z3 certainly would have greater data integrity.

 

[Patrick_3000]

If you mention these models because the intention is to buy them, that does not match with the repeated statement of how critical the data are, IMHO.

Apart from a really good, i.e. tested monthly, backup, I would go for high-end SSDs.

I'm not sure I agree with this. Consumer grade Teamgroup and Kingston are fine. I've used both over the years and never had problems with them. I've also read reviews, and they're not as fast as a Samsung or Western Digital gen 4 or 5, but they're unlikely to fail in the first few years and are probably more reliable than any hard drive (though that last point is a guess). And in Z3 or even Z2, if one fails, which I think would be a fluke, it would just need to be replaced.

I do think, however, that it's best to spread out brands when building an SSD pool. One Teamgroup, one Kingston, one or two Crucials, and maybe one or two higher end brands like Samsung and Western Digital, is probably better than 5x or 6x of the same make and model, since it minimizes the risk of multiple failures at once.

 

[Etorix]

Regardless of brand, a cheap 4 TB NVMe drive is going to be QLC. That will work when filling it slowly, but I personally would NOT like to have my own critical data on a resilvering pool of consumer QLC SSDs… Loading a few TB in one long sequence is going to be painful for these drives—just like it is for SMR HDDs.

If I were to build a NVMe pool for critical data, I'd go for data centre grade SSDs in U.2/U.3 form factor for reliability (plus thermal management and higher capacity than M.2). Of course, that means a nice set of cables, and a different price level.

 

[NickF]

I disagree that NVME will be less of a hassle. M.2 is not a hot seappable format as an example. Which means you have to power down your system to perform basic maintenance operations. That’s a lot more hassle if you ask me.

In general the use case presented and the design goals you’ve shared seem at odds with another. It’s not a system that is balanced. But it should work fine albeit not to the potential of the storage and will use more power than sata under load.

I also agree QLC IS QLC and you are never going to get the most out of any drive you choose in your config. So just get whatever mix of same size drives you want. But YMMV with consumer ssds. They don’t have PLP and other enterprisey features. If you’re considering a raidz3 and aren’t considering those options, you are asking the wrong questions.

This is an example of just because you can do something doesn’t mean that you should. There are design considerations you are not willing to consider or compromise on, so I’m not sure how much help we are going to be. Particularly questions about performance will go unanswered

 

[NugentS]

Hardware is ASRock Rack x570d4u-2L2T, Ryzen 7 Pro 5750G, 128 GB ECC memory. Use case is home storage but rather critical, includes not only personal data but critical work-related documents and a lot of my spouse's work-related data. Also runs two VMs.

And the drives are?
As per forum rules, please post your entire hardware including what the drives are. It matters

 

[Patrick_3000]

And the drives are?
As per forum rules, please post your entire hardware including what the drives are. It matters

I question the relevance of the current hard drives I use in a thread regarding the RAID level for SSD pool I plan to switch to. However, for what it's worth, my current drives are 12 TB Seagate Ironwolf Pro.

 

[NickF]

I question the relevance of the current hard drives I use in a thread regarding the RAID level for SSD pool I plan to switch to. However, for what it's worth, my current drives are 12 TB Seagate Ironwolf Pro.

In regard to providing a holistic view of the problem it is relevant. As an example, what is the current raid topology of those drives? Have you had data loss? Do you have backups?

Would it maybe make sense to build a backup appliance instead given you have existing hardware and you are particularly worried about your data?

 

[Patrick_3000]

I disagree that NVME will be less of a hassle. M.2 is not a hot seappable format as an example. Which means you have to power down your system to perform basic maintenance operations. That’s a lot more hassle if you ask me.

In general the use case presented and the design goals you’ve shared seem at odds with another. It’s not a system that is balanced. But it should work fine albeit not to the potential of the storage and will use more power than sata under load.

I also agree QLC IS QLC and you are never going to get the most out of any drive you choose in your config. So just get whatever mix of same size drives you want. But YMMV with consumer ssds. They don’t have PLP and other enterprisey features. If you’re considering a raidz3 and aren’t considering those options, you are asking the wrong questions.

This is an example of just because you can do something doesn’t mean that you should. There are design considerations you are not willing to consider or compromise on, so I’m not sure how much help we are going to be. Particularly questions about performance will go unanswered

Again, I disagree with you.

My question was simple: is RAID Z3 worth the cost in an SSD pool versus RAID Z2? The answer to that question probably doesn't even depend much, if at all, on whether the pool is NVME or SATA.

You have mostly just taken this thread as an opprtunity to expound your philosophy which is in favor of SATA over NVME. I disagree with you on this point. SSD, even consumer grade QLC, is probably more reliable than HDD, which people have been using and discussing in this forum uncritically for years. That's essentially a guess, but it's an educated guess based on what I know about SSDs, what I've seen online, and the fact that there are no moving parts.

Moreover, NVME SSD, even consumer grade, is sure to be faster than anything SATA. As one example, on my 10gbe ethernet network, I routinely max out the network speed or come close (like 1000 or 1100 MB/sec) when transferring consumer-grade NVME SSD to NVME SSD (in a transfer between a Linux desktop system to a Windows desktop system), but I only get something like half that (around 500 MB/sec) when transferring SATA SSD to SATA SSD.

We are in a new era when it comes to SSD. Prices have dropped significantly in the past year, and going forward, SSD, particularly M.2 NVME SSD, are going to be in a lot of places where they weren't previously. SATA SSD, not so much given its maintenance hassle, larger size, and higher power consumption.

The lack of need for cable maintenance alone makes NVME SSD a no-brainer in my opinion. I currently have three NAS hard drives (Seagate Ironwolf Pro) in SCALE, and just a few weeks ago, one showed as offline, but when I opened the case and looked, it was a loose and failing SATA cable, which I had to replace. That problem would still occur with SATA SSD.

 

[Patrick_3000]

In regard to providing a holistic view of the problem it is relevant. As an example, what is the current raid topology of those drives? Have you had data loss? Do you have backups?

Would it maybe make sense to build a backup appliance instead given you have existing hardware and you are particularly worried about your data?

Thanks for the clarification. I actually mentioned the current topology in my original post, but it's a 3-way mirror of these 3 12 TB NAS hard drives currently, so a 12 TB pool.

I currently have a backup SCALE server that's also a three-way HDD mirror containing hourly backups of my pool and that I plan to leave in place after I upgrade my main server to SSD.