hi all,
As per the title, looking for some help troubleshooting some performance issues with a TrueNAS scale setup. First up, hardware specs:
Dell R740xd, Dell PERC H730P (in HBA mode), 2 x Xeon Silver 4214, 384 GB of RAM (all 6 channels populated per CPU), 10 and 25gbE, running VSphere / ESXi 7, 14 x HGST SAS2/12gig 10TB disks.
Config as tested:
H730P (in HBA mode) in full pcie passthrough mode to TrueNAS scale VM. 8 CPUs and 128GB of RAM were allocated to the scale instance. No other workload running on the hypervisor at the time. All 14 spindles were setup as a single RAIDz2 storage pool, 1mb striping (this is a media and backups storage pool - lots of large files, no need for high iops), tested with compression both on and off.
Turning compression off and trying something basic (IE: dd if=/zev/zero of=testfile bs=1M count=102400) yields around 350MB/s (+ / - 15MB/s) to the array. Experimenting with block sizes (in the array itself and dd) generally yield poorer results. Aware that dd is going to limit testing to raw streaming with no consideration of iops but that's precisely what I'm tuning for at the moment.
Tore down the whole system, switched the H730P back to RAID mode, and threw the disks into a H730P managed RAID6 and exposed them to a basic ubuntu VM on the same host/infrastructure and formatted as xfs (using the same exact VM definition - just swapped OS out). While the RAID is even still initializing I can handily write 1GB/s to the array so it's not an issue with the disks / sas controller / backplane / etc.
Also having suspected performance issues with another setup using 8 x 3.84TB NVMe in RAIDz1 in a R640 NVMe backplane again via VMWare (but have not done much troubleshooting on this one yet) so curious if there's some tuning I need to do to get scale to perform under VMWare? Am I underestimating how much CPU hardware I need to throw at RAIDZ for performance? I figure 8 cascade lake CPUs is sufficient and don't see more than one thread getting much use under load so I don't think it's that?
Advice on what to try next would be appreciated!
As per the title, looking for some help troubleshooting some performance issues with a TrueNAS scale setup. First up, hardware specs:
Dell R740xd, Dell PERC H730P (in HBA mode), 2 x Xeon Silver 4214, 384 GB of RAM (all 6 channels populated per CPU), 10 and 25gbE, running VSphere / ESXi 7, 14 x HGST SAS2/12gig 10TB disks.
Config as tested:
H730P (in HBA mode) in full pcie passthrough mode to TrueNAS scale VM. 8 CPUs and 128GB of RAM were allocated to the scale instance. No other workload running on the hypervisor at the time. All 14 spindles were setup as a single RAIDz2 storage pool, 1mb striping (this is a media and backups storage pool - lots of large files, no need for high iops), tested with compression both on and off.
Turning compression off and trying something basic (IE: dd if=/zev/zero of=testfile bs=1M count=102400) yields around 350MB/s (+ / - 15MB/s) to the array. Experimenting with block sizes (in the array itself and dd) generally yield poorer results. Aware that dd is going to limit testing to raw streaming with no consideration of iops but that's precisely what I'm tuning for at the moment.
Tore down the whole system, switched the H730P back to RAID mode, and threw the disks into a H730P managed RAID6 and exposed them to a basic ubuntu VM on the same host/infrastructure and formatted as xfs (using the same exact VM definition - just swapped OS out). While the RAID is even still initializing I can handily write 1GB/s to the array so it's not an issue with the disks / sas controller / backplane / etc.
Also having suspected performance issues with another setup using 8 x 3.84TB NVMe in RAIDz1 in a R640 NVMe backplane again via VMWare (but have not done much troubleshooting on this one yet) so curious if there's some tuning I need to do to get scale to perform under VMWare? Am I underestimating how much CPU hardware I need to throw at RAIDZ for performance? I figure 8 cascade lake CPUs is sufficient and don't see more than one thread getting much use under load so I don't think it's that?
Advice on what to try next would be appreciated!
