Great Read Speeds | Horrible Write Speeds | 16x SSDs

[socalcompute]

Hello TrueNas!

I am a bit new to TrueNas and looking to get the most performance out of my SAN:

Dell PowerEdge R730XD
CPU: 2x Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
RAM: 64GB DDR4 ECC
NICs: 2x Mellanox ConnectX-4 @ 25Gbps - MTU: 9000 - Using VMware Round Robin policy on hosts with RR Policy set to 1
16x 800GB SAS SSDs for Data (HITACHI HUSMM118 CLAR80 Revision C250 - Not set for self encryption)
2x 200GB SAS SSDs for TrueNas Core 12.0-U7
1x 800GB SAS SSD for Hotspare

Storage Pool looks like this:
2x VDEV's with 8 SSDs each configured for RAIDZ1 (I've tried recreating with RAIDZ2 and RAIDZ3 and I get poor write performance with those options as well)
1x zvol provisioned at 5TB of space (non scratch) - Current Sync is disabled (I've tried with Sync on and off - same poor write speed)

Storage is delivered to a large VMware Cluster via iSCSI (currently only mounted on one host for testing).

Testing: I'm using a Windows Server 2022 VM running Crystal Disk Mark 8.0.1 with NVMe Profile:

Hosts that are connecting to TrueNAS are:
16x Dell Powedge R920's (Currently only 1 connected for testing)
4TB of DDR3 RAM each
2x 200GB SAS in RAID 1 for ESXi 7.0.3
2x Intel X520 Dual Port 10GB SFP+

This SAN was converted from a StarWind Virtual SAN running on top of Windows. Performance with StarWind using RAID50 (Dell PERC H730P with Cache) was 2500MB/s Read and 2500MB/s write.

Troubleshooting done:

Updated firmware on 25GB NICS
gstat reports drives are sitting at 10% utilization during testing
Added a LOG drive (4TB WD SN1500 NVMe drive - 2x 2TB NVMe Gen3 in RAID0 from the card) - Had no effect after 200GB of written data

Thank you so much for your time,

Christopher
SoCal Compute

 

[Ericloewe]

16x 4 TB of DRAM… That’s impressive…

Anyway, it’s very interesting that you’re so write-limited and it’s not sync writes causing it. Your sequential writes feel closer to spinning rust than any sort of SSD.
What SAS controller are you using? Dell HBA330 mini? That’s the first suspect, since I don’t think you mention it. I don’t see any smoking guns, either.

Do keep in mind that with RAIDZ you can expect the performance of a single disk from each vdev. So, with your setup, two SSDs’ worth of performance, which feels about right in your read numbers.

Typically, given pools that aren’t very full, writes tend to be faster than reads because they can just be logged and then dumped as a whole transaction group in one go, whereas writes tend to need data from disk (unless it’s in ARC).

 

[socalcompute]

Hey Eric,

Thanks for your speedy reply!

The SAS controller is Dell's PERC H710P in non-RAID mode. Could this RAID controller be the culprit for slow non-RAID writes? Interestingly enough it performs very well while in a RAID 50.

I also failed to mention that I tried with 8x VDEV's with 2x 800GB SAS SSDs in Mirror and I got about the same write performance as that screenshot. During testing the pool was empty except for a 60GB Windows Server VM where Crystal Disk Mark was being run from.

The SAS SSD's individually read / write at around 600MB/s

 

[socalcompute]

I just noticed that H730P was still in RAID mode, not HBA mode. Rebooting to see if that makes a difference.

 

[socalcompute]

Putting the H730P in HBA mode (not just selecting all disks as non-raid) was the trick! Seeing fantastic performance now.

 

[Ericloewe]

Non-HBA mode is absolutely, insanely terrible, at least for ZFS. As for HBA mode, it's less than optimal since it goes through the less well-tested mrsas driver. Ideally, you'd want to replace it with an HBA330 mini to take advantage of the more mature and well-understood mpr driver stack. You also keep all the iDRAC functionality intact (minus RAID management, obviously). You're still dealing with Dell's firmware, which is a bit of a mystery when compared to the stock firmware, but nobody's ever actually complained.