Seeking Assistance with Low Read Speeds on TrueNAS (dd test)

[Congles]

Hey All,

I'm getting significantly lower read speeds compared to write speeds, and I'm seeking insights or suggestions to resolve this.

System Setup:

• TrueNAS running as VM with 16 Cores (EPYC 75F3 32-Core) and 124gb RAM
• 4 VDEV's of 6 drives in Raidz2
• Dataset for testing: speedtest (created specifically for this purpose with compression turned off).
• ZFS Cache Setting: Set primarycache=metadata to disable data caching (only applied to the speedtest dataset).

Test Procedure and Results:

1. Write Test:
   • Command: dd if=/dev/zero of=/mnt/bender/speedtest/tmp.dat bs=2048k count=50k
   • Result: 107374182400 bytes transferred in 56.531521 secs (approx. 1.899 GB/s)
2. Read Test:
   • Command: dd if=/mnt/bender/speedtest/tmp.dat of=/dev/null bs=2048k count=50k
   • Result: 107374182400 bytes transferred in 254.130370 secs (approx. 0.422 GB/s)

Issue: While I'm unsure whether the write speeds are what you might expect for this setup, I believe read speeds are significantly lower than they should be.
Are there specific ZFS settings or TrueNAS configurations that I should check or modify to improve non-cached read performance?


Any assistance or guidance from the community would be greatly appreciated.


Thank you in advance!

 

[Arwen]

Full hardware, including disk / storage make, model and potentially firmware.

Further, your test file appears to be 107GBytes large. With 124GBytes of RAM and using Core, (aka FreeBSD), that means the writes might collect into memory making the writes appear "too fast".

So, reading at 422MBytes per second does seem reasonable for a single 4 disk stripe using hard disks. (6 disk RAID-Z2 has 4 data disks per stripe).

Remember, this is a single threaded test. That means a single vDev accessed at a time. Even if all the data is striped across all 4 vDevs, a ZFS transaction write is to one vDev.

 

[sretalla]

ZFS Cache Setting: Set primarycache=metadata to disable data caching (only applied to the speedtest dataset).

If your intention is to disable caching, you need to set that to disabled, not metadata, but I get where you might be wanting to see the impact of allowing metadata to cache without data cached.

Read Test:

• Command: dd if=/mnt/bender/speedtest/tmp.dat of=/dev/null bs=2048k count=50k

I would throw out that approach in general and use something more like this:
fio --name TEST --eta-newline=5s --filename=/mnt/bender/speedtest/tmp.dat --rw=read --size=150g --io_size=1500g --blocksize=1M --iodepth=16 --direct=1 --numjobs=1 --runtime=120 --group_reporting

Play with numjobs to see the impact of multi-user access (and caching).

 

[Congles]

So, reading at 422MBytes per second does seem reasonable for a single 4 disk stripe using hard disks. (6 disk RAID-Z2 has 4 data disks per stripe).

Thanks, that does make a bit of sense. I'll do some further testing as suggested by sretalla.

Full hardware, including disk / storage make, model and potentially firmware.

here's more info:

• Supermicro H12-SSL Motherboard
• Host OS is Unraid
• EPYC 75F3 32-Core with 16 cores isolated for TrueNAS Core
• Broadcom LSI SAS3008 PCI-Express Fusion-MPT SAS-3 passed through to TN on boot.
• 256GB RAM with 124 dedicated to TrueNAS Core
• HDDs:
  • Exos 16TB and 18TB drives (currently migrating everything to 18TB):
  • Models ATA ST18000NM000J & ATA ST16000NM001G
  • Unsure how to check firmware

Any other information you need let me know, happy to dig out anything specific to help.

fio --name TEST --eta-newline=5s --filename=/mnt/bender/speedtest/tmp.dat --rw=read --size=150g --io_size=1500g --blocksize=1M --iodepth=16 --direct=1 --numjobs=1 --runtime=120 --group_reporting

Play with numjobs to see the impact of multi-user access (and caching).

Thanks, I've given that command a go:

I'm not great at interpreting this information but looks like 642mb/s is the speed outcome. If that's all on one VDEV then I think that is about expected?


I ran it again with 4 jobs, I guess this is reflecting what 4 concurrent, non-overlapping reads would do?

Any help in interpreting these results is greatly appreciated!

 

[sretalla]

Unsure how to check firmware

smartctl -a /dev/da1 | grep Firmware

help in interpreting these results

With a single job, you're seeing the best case for a single user of the system. As you saw, 600+ MB/sec is OK. You're probably getting a little under 200 from each VDEV.

Almost exactly the same story for IOPS, which as we know is limited to what one drive can do per VDEV, so pretty much spot on.

With more jobs, you get some idea of what might happen in terms of system performance with that many users, so you could expect concurrent use to push your network to that speed.

 

[HoneyBadger]

Have a second terminal open and check your top results on the VM. I expect you'll see very low overall usage even during a test; but 16 cores for a VM is a significant amount and you may be putting unnecessary pressure on the KVM thread scheduler underneath to keep things in sync, even with 32 cores on the host. Counter-intuitively, you may actually benefit from reducing core count if the host scheduler is able to concentrate the workload on a smaller number of physical cores and allow them to boost higher.

 

[Congles]

smartctl -a /dev/da1 | grep Firmware

Did a few random drives, firmware is mostly SN02, but a few will be SN04 and SN01. No idea what that might imply for performance but I think based on the rest of your comment, by performance is doing just fine.

Have a second terminal open and check your top results on the VM.

I did as you suggest @HoneyBadger and certainly the SMBD process seems to be using a lot of resources. Thought to be fair, I see smbd using about this much frequently even when not running test.

Not sure if it's related in any way, but I have just noticed these messages in the console. No idea how to interpret them, I'm happy to ignore them for now if they're unlikely to be impacting much