Slow pool read performance

[fulder]

Hello,

I've been a little confused about my TrueNAS SCALE read performance after doing some simple tests. The tests were performed inside an isolated (1 GbE) VLAN with only one "client" machine and the TrueNAS SCALE machine, connected to the same switch (so all the traffic should never leave it).

TrueNAS

iperf3 -s

Client:

iperf3 -c <TRUENAS_IP>

Results:

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 958 Mbits/sec 0 441 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 936 Mbits/sec 0 554 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 943 Mbits/sec 0 583 KBytes
[ 5] 3.00-4.00 sec 113 MBytes 946 Mbits/sec 0 611 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 943 Mbits/sec 0 639 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 670 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 706 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 738 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 814 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 940 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender
[ 5] 0.00-10.04 sec 1.09 GBytes 936 Mbits/sec receiver

TrueNAS:

touch /tmp/test
iperf3 -s -f /tmp/test

Client:

iperf3 -c <TRUENAS_IP>

Results:

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 114 MBytes 956 Mbits/sec 0 594 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 943 Mbits/sec 0 652 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 766 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 888 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 974 KBytes
[ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 1.02 MBytes
[ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 1.07 MBytes
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 1.12 MBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 1.18 MBytes
[ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 1.23 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender
[ 5] 0.00-10.04 sec 1.09 GBytes 935 Mbits/sec receiver

TrueNAS:

iperf3 -s -f <STRIPE_POOL_FILE>

Client:

iperf3 -c <TRUENAS_IP>

Results:

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 5.89 MBytes 49.4 Mbits/sec 57 99.0 KBytes
[ 5] 1.00-2.00 sec 4.04 MBytes 33.9 Mbits/sec 0 124 KBytes
[ 5] 2.00-3.00 sec 3.38 MBytes 28.4 Mbits/sec 0 146 KBytes
[ 5] 3.00-4.00 sec 2.14 MBytes 17.9 Mbits/sec 0 154 KBytes
[ 5] 4.00-5.00 sec 4.28 MBytes 35.9 Mbits/sec 0 171 KBytes
[ 5] 5.00-6.00 sec 3.80 MBytes 31.9 Mbits/sec 0 188 KBytes
[ 5] 6.00-7.00 sec 3.86 MBytes 32.4 Mbits/sec 0 199 KBytes
[ 5] 7.00-8.00 sec 1.60 MBytes 13.4 Mbits/sec 0 170 KBytes
[ 5] 8.00-9.00 sec 1.43 MBytes 12.0 Mbits/sec 0 212 KBytes
[ 5] 9.00-10.00 sec 4.33 MBytes 36.4 Mbits/sec 0 218 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 34.7 MBytes 29.1 Mbits/sec 57 sender
[ 5] 0.00-10.05 sec 29.0 MBytes 24.2 Mbits/sec receiver

^ This is confusing as the speed seems to drop 30 times. The Stripe pool contains only one HDD (see my system) /dev/sdc and hdparm gives the following (expected) results:

hdparm -t /dev/sdc
Timing buffered disk reads: 402 MB in 3.03 seconds = 132.88 MB/sec

I've also tested this on my RAID-Z2 pool with the same weird drops in speed and am a bit lost what could be the cause of this decreased performance. Any ideas and help are much appreciated (please tell me if you lack any tests or data).

Best Regards,
fulder