FreeNAS/TrueNAS SSD vs HDD and 'RAID' levels

[SubnetMask]

When working with spinning disks, traditionally, but not universally, RAID10 would outperform RAID5/6 (I had a Promise vTrak that according to them was optimized for RAID5, and throughput testing confirmed that RAID5 blew RAID10 out of the water), but on FreeNAS/TrueNAS, RAIDZx provides little to no performance increase, read or otherwise, while stripes or striped mirrors (or mirrored stripes) provide a performance benefit. Now, when talking about spinning disks, when talking about an absolute best of ~200 IOPs, I can certainly see the benefit of a stripe (or mirrored stripe) increasing performance - a 10 disk stripe could bring the IOPs to 2000 IOPs... but what about talking about SSDs that can push 22,000 WRITE IOPs? You'd need a stripe of 110 15K SAS disks to try to equal the write IOPS of that single SSD - that being said, assuming one had a bunch of enterprise grade SAS SSDs that are capable of tens of thousands of write IOPS and hundreds of thousands of read IOPS, What is the drawback to setting up, say, ten of them, in a RAIDZ2 or even Z3 config? I suspect that in many environments, any benefits that could be had setting up ten SSDs in a mirrored stripe vs RAID5/6 would be impossible to really see since each individual SSD is so much faster than any spinning disk. It seems to me that when talking about SSDs, in most cases, 'RAID level' is more about redundancy than performance.

 

[ChrisRJ]

Yes and no. If the question is whether or not a certain performance threshold can be reached, and the benchmark is a number of HDDs configured as mirror vdevs, you could very likely achieve that with SSDs. (It would require a really huge number of HDDs to outperform the SSDs, although this is not unheard of.)

If, on the other hand, the question is whether or not SSDs will perform better when configured as mirrors, the answer would likely be yes. I do not make an absolute statement here, because at the end in such a situation it all depends where the bottleneck is. Also, performance always comes with two aspects: latency and throughput. In my understanding the code path is different between mirrors and RAIDZ, with the latter being more complex.

As a sidenote, and to hopefully illustrate my point better, when SSDs first came out, quite a few people running high-end database servers were disappointed. I spoke with a guy who is a professor for computer science and specializes in databases. According to him the reason for this was the level of optimization that gone into the databases. They were optimized for the performance characteristics of HDDs to such a degree, that even if the SSD would return much more quickly from a call than the HDD, another components of the database would not be ready yet (simply because it had not made any sense for HDDs). So the carefully aligned bottleneck would not "match" anymore, thus overall no gain was achieved.

Edit: Of course databases have long been changed to utilize the advantages of SSDs.

 

[mav@]

While it makes economical sense to use small 3/5-wide RAIDZ vdevs of SSDs if top IOPS is not a priority, wider vdevs not so much -- cost saving is diminishing, while CPU and others overheads grow. If you are using default 128KB records (and you should not use smaller there), then 10-wide RAIDZ2 will split it into 8 I/Os, so that both ZFS and SSDs (unless ZFS manage to aggregate) will have to handle plenty of 16KB I/Os, that is quite stressful. For wide vdevs you need much larger block sizes to stay efficient, but that is not exactly a workload where SSDs are thought about.