Good NVME speeds but should be Faster?

[DenisInternet]

Hey folks, I've been running my self built TrueNAS scale system for a few months, overall I love it. Also this forum has been a huge help in figuring things out, can't thank enough all the awesome and detailed documentation.

Overall I am very happy with how my system is running, the one thing though that doesn't seem to add up are the read and write speeds I am getting on my system. Before purchasing more RAM, or Testing new cables, new NIC etc. I was hoping to get some pointers on what and how to test things, to better understand what is causing the "bottleneck" or if the speeds I am getting are in fact correct for my setup.

I read the ZFS STORAGE POOL LAYOUT guide a few times, and ran a few shell commands to test things, and it seems that either my network settings/switch is bottlenecking the speeds (in which case, I will go back to the MicroTik discord folks for help with that), or I am not understanding the results I am getting, which as a newbie/rookie is very likely.

root@truenas[~]# dd if=/dev/zero of=/mnt/Nautilus_01/Post_Main/test.dat bs=2048k count=10000​

10000+0 records in​

10000+0 records out​

20971520000 bytes (21 GB, 20 GiB) copied, 5.93856 s, 3.5 GB/s​

root@truenas[~]# dd of=/dev/null if=/mnt/Nautilus_01/Post_Main/test.dat bs=2048k count=10000​

10000+0 records in​

10000+0 records out​

20971520000 bytes (21 GB, 20 GiB) copied, 3.12151 s, 6.7 GB/s​

root@truenas[~]#​

These speeds above look correct to what my RaidZ1 NVME pool should be capable of, but when testing the speeds on my Mac Studio using "Blackmagic Speed Test" I am getting 717.8 MB/s write speeds and 1464.5 MB/s read speeds.

So from more reading on the forums, I start reading/learning about iperf3 and that I should use that to test my actual network connection, so I tried running:
root@truenas[~]# iperf3 -c 192.168.1.253
iperf3: error - unable to connect to server: Connection refused

Tried trouble shooting but no success so far. Do I need to disconnect/shutdown my SMB share to run this test, or is this my MacStudio's firewall stopping my from running the test?


Specs and additional details below.
CPU: AMD EPYC 7302p+
MOBO: Supermicro H12SSL-i
SSDs: x6 WD UltraStar DC SN630 7.68TB NVMe PCIe 3.1x4 U.2 (In Raid z1)
RAM: x4 Supermicro (Hynix) 16GB 288-Pin DDR4 3200 (PC4-25600)
Broadcom HBA 9500-16i Tri-Mode - storage controller
Switch: MikroTik CRS504-4XQ-IN
NAS NIC: Supermicro AOC-S25G-B2S Rev 1.01 25GbE 2-Port PCIe 3.0 x8 Network Adapter
Mac Studio NIC: Sonnet Twin25G Dual 25G SFP28 to Thunderbolt 3 Adapter


For anyone reading, I found this very helpful to explain/visualize things: https://jro.io/r2c2/

Thanks!

*Update 3.

1) Made a clean install of TrueNAS to make sure nothing funky was happening.
2) Upgraded to 256GB RAM to improve performance.
3) Purchased second NIC to for Windows Machine, to test speed on two systems (MacOS and WinOS)
4) I changed the pool from 1 vdev with 6 NVMEs in RaidZ1 to a 2 vdev 3 NVME in Raid z1, and finally 3vdev mirrored pairs and while my performance has improved slightly, it is still much slower than expected.

Is there any other setting I can use to improve performance? It seems I am getting a bottleneck somewhere.
I am not sure if the bottleneck is SMB, my HBA, or somehow my CPU?

 

[Davvo]

dd is not the most reliable in these kind of tests, use fio instead (don't know if it's available on SCALE though).

Are you able to sucessfully ping 192.168.1.253?

Also, I suppose your pools are empty but can you confirm it?

Bonus read:

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

Edit for spelling correction.

 

[Etorix]

Zeroes compresses excessively well. Do you have compression on the test dataset? (If so, forget about your dd test.)

As for iperf3, had you launched iperf3 -s on the target Mac?
And what gives the test in reverse? (server on the NAS, client from the Mac)

 

[DenisInternet]

Zeroes compresses excessively well. Do you have compression on the test dataset? (If so, forget about your dd test.)

As for iperf3, had you launched iperf3 -s on the target Mac?
And what gives the test in reverse? (server on the NAS, client from the Mac)

@Etorix Doh! My mistake, I hadn't launched "iperf3 -s" on the Mac, now it works! Thank you

root@truenas[~]# iperf3 -c 192.168.1.251​

Connecting to host 192.168.1.251, port 5201​

[ 5] local 192.168.1.253 port 37368 connected to 192.168.1.251 port 5201​

[ ID] Interval Transfer Bitrate Retr Cwnd​

[ 5] 0.00-1.00 sec 1.97 GBytes 16.9 Gbits/sec 2402 537 KBytes ​

[ 5] 1.00-2.00 sec 1.99 GBytes 17.1 Gbits/sec 2093 764 KBytes ​

[ 5] 2.00-3.00 sec 1.98 GBytes 17.0 Gbits/sec 1510 602 KBytes ​

[ 5] 3.00-4.00 sec 1.99 GBytes 17.1 Gbits/sec 1728 857 KBytes ​

[ 5] 4.00-5.00 sec 1.98 GBytes 17.0 Gbits/sec 1830 509 KBytes ​

[ 5] 5.00-6.00 sec 1.98 GBytes 17.1 Gbits/sec 1742 697 KBytes ​

[ 5] 6.00-7.00 sec 1.99 GBytes 17.1 Gbits/sec 1920 731 KBytes ​

[ 5] 7.00-8.00 sec 2.00 GBytes 17.2 Gbits/sec 1753 554 KBytes ​

[ 5] 8.00-9.00 sec 1.97 GBytes 16.9 Gbits/sec 1812 612 KBytes ​

[ 5] 9.00-10.00 sec 1.99 GBytes 17.1 Gbits/sec 2078 499 KBytes ​

- - - - - - - - - - - - - - - - - - - - - - - - -​

[ ID] Interval Transfer Bitrate Retr​

[ 5] 0.00-10.00 sec 19.9 GBytes 17.1 Gbits/sec 18868 sender​

[ 5] 0.00-10.00 sec 19.8 GBytes 17.1 Gbits/sec receiver​

​

iperf Done.​

Ok based on these results, it's much closer to what Blackmagic speed test was showing.

Compression Level: LZ4 , didn't realize it would hit my speeds so much, should I turn if off? I am in Video Production / Color Grading large 6k+ resolution raw files.

@Davvo Yep, can ping just fine, Pool is currently just under 60% capacity, I remember vaguely reading a post from @jgreco that it's not recommended to use more than 50% of a pool, so I am guessing its a combination of that, and compression that impacts my speeds?

Thank you both!

 

[Davvo]

@Davvo Yep, can ping just fine, Pool is currently just under 60% capacity, I remember vaguely reading a post from @jgreco that it's not recommended to use more than 50% of a pool, so I am guessing its a combination of that, and compression that impacts my speeds?

With that kind of computing power compression is likely not harming your real speeds and just falsing your tests.
As the pool's available space decreases so does performance; the 50% of jgreco is the optimal point for mainly the writes, you read it in the following resource.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

So, can you check your pool's fragmentation?
Are your writes sync, async or standard? Setting the sync value off will hopefully increase your performance.

 

[DenisInternet]

With that kind of computing power compression is likely not harming your real speeds and just falsing your tests.
As the pool's available space decreases so does performance; the 50% of jgreco is the optimal point for mainly the writes, you read it in the following resource.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

So, can you check your pool's fragmentation?
Are your writes sync, async or standard? Setting the sync value off will hopefully increase your performance.

Thank you @Davvo , is this what you are looking for?

root@truenas[~]# zpool list -v​

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT​

Nautilus_01 41.9T 25.4T 16.5T - - 0% 60% 1.00x ONLINE /mnt​

raidz1-0 41.9T 25.4T 16.5T - - 0% 60.5% - ONLINE​

fece7 6.98T - - - - - - - ONLINE​

a7e9 6.98T - - - - - - - ONLINE​

d40d 6.98T - - - - - - - ONLINE​

e339 6.98T - - - - - - - ONLINE​

1b83 6.98T - - - - - - - ONLINE​

a6a9 6.98T - - - - - - - ONLINE​

boot-pool 928G 22.7G 905G - - 0% 2% 1.00x ONLINE -​

mirror-0 928G 22.7G 905G - - 0% 2.44% - ONLINE​

sdb3 931G - - - - - - - ONLINE​

sde3 931G - - - - - - - ONLINE​

And I believe my Sync is set on standard. (Looking at dataset details on the datasets page)

 

[DenisInternet]

Updated top of the post, any advice is very much appreciated. Thank you!

 

[Davvo]

I missed your previous post, sorry. We can rule out fragmentation.

 

[DenisInternet]

Updated top of the post, any advice is very much appreciated. Thank you!

No worries, I think this is the speed of the NVME drives when using TrueNAS, and that's that. I guess I was expecting the speeds to improve significantly more when using a 2vdev raidz1 or 3vdev mirrors over a basic 1vdev RaidZ1, but I guess not. So maybe there isn't any bottleneck at all, but the system working as intended?

 

[Davvo]

Try fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=300g --io_size=1500g --blocksize=128k --iodepth=16 --direct=1 --numjobs=16 --runtime=120 --group_reporting inside the pool you want to test /mnt/poolname.

man fio (1): flexible I/O tester

man fio (1): fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. The typical use of fio is to write a job file matching the I/O load one wants to simulate.

manpages.org

 

[DenisInternet]

Try fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=read --size=300g --io_size=1500g --blocksize=128k --iodepth=16 --direct=1 --numjobs=16 --runtime=120 --group_reporting inside the pool you want to test /mnt/poolname.

man fio (1): flexible I/O tester

man fio (1): fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. The typical use of fio is to write a job file matching the I/O load one wants to simulate.

manpages.org

Thank you will try this in a bit. Unfortunately with my clean install, setting up my Apps seems broken, so dealing with that right now, and reading on forums for solutions. (UTC clock is correct, tried unsetting, deleting dataset, restarting, forcing save, and then creating pool) but still nothing...