Overkill TrueNAS server slow on read

[AndroidBot]

I got a bit of an overkill server

2x E5-2699A v4 with 500GB RAM

And here are the pools

My Network card is 25GBe and 9000 MTU

Everything works well when I transfer a file from Desktop to NAS, I get constant 1.1GB/s transfer.

The trouble starts when I try to copy file from NAS to my Desktop, then I get initially 1.1GB/s transfer speeds but after 2-3 seconds it goes down to 100MB/s

What am I missing? Any advices ? No misses in `arcstat`

 

[jgreco]

The Intel Ethernet cards generally do pretty well, but jumbo frames exercise code paths that are uncommon, and you might be better served by disabling jumbo frames and making sure that offload is properly configured instead.

 

[AndroidBot]

But transfer would never reach 1.1GB without the jumbo frames and the thing is this used to work both ways, I used to get 1.1GB/s upload/download, but now reading is not working properly somehow

 

[AndroidBot]

I just noticed one thing...

sysctl: unknown oid 'net.ipv4.tcp_available_congestion_control'

Am I missing something? I tried to follow this guide

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

 

[AndroidBot]

I should mention, I did iperf3 test, it's perfect. I kept it up for 5 minutes and speed is constant, 1.2GB/s

I tested the pool with `fio`, again perfect speeds.

Only thing that's not working properly in SMB

 

[jgreco]

Which card exactly do you have?

 

[AndroidBot]

Client PC: Intel X710T
TrueNAS server: Intel XXV710

I tested with iperf3 and fio very carefully. Nothing wrong with those. I also experimented with smb signing on/off, absolutely no difference. Just on copy from TrueNAS to my computer it starts strong, jumps to 1.1GB/s then after 5 seconds or so drops to static 100mb/s and keeps it there till the whole transfer ends.

Update: actually it slows down both ways, from my computer to TrueNAS or from TrueNAS to my computer. Just it shows slow speeds faster when I'm copying from TrueNAS, after 3-4 seconds it drops to 100mb/s but when I'm copying from my computer to TrueNAS it takes like 8-9 seconds to drop to 100mb/s

Here's iperf3 results:

[ 5] 0.00-1.00 sec 676 MBytes 5.67 Gbits/sec
[ 5] 1.00-2.00 sec 753 MBytes 6.32 Gbits/sec
[ 5] 2.00-3.00 sec 755 MBytes 6.33 Gbits/sec
[ 5] 3.00-4.00 sec 754 MBytes 6.32 Gbits/sec
[ 5] 4.00-5.00 sec 748 MBytes 6.27 Gbits/sec
[ 5] 5.00-6.00 sec 752 MBytes 6.31 Gbits/sec
[ 5] 6.00-7.00 sec 750 MBytes 6.29 Gbits/sec
[ 5] 7.00-8.00 sec 757 MBytes 6.35 Gbits/sec
[ 5] 8.00-9.00 sec 757 MBytes 6.35 Gbits/sec
[ 5] 9.00-10.00 sec 758 MBytes 6.36 Gbits/sec
[ 5] 10.00-11.00 sec 756 MBytes 6.34 Gbits/sec
[ 5] 11.00-12.00 sec 755 MBytes 6.33 Gbits/sec
[ 5] 12.00-13.00 sec 754 MBytes 6.32 Gbits/sec
[ 5] 13.00-14.00 sec 749 MBytes 6.28 Gbits/sec
[ 5] 14.00-15.00 sec 754 MBytes 6.32 Gbits/sec
[ 5] 15.00-16.00 sec 757 MBytes 6.35 Gbits/sec
[ 5] 16.00-17.00 sec 753 MBytes 6.32 Gbits/sec
[ 5] 17.00-18.00 sec 750 MBytes 6.29 Gbits/sec
[ 5] 18.00-19.00 sec 757 MBytes 6.35 Gbits/sec
[ 5] 18.00-19.00 sec 757 MBytes 6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-19.00 sec 14.1 GBytes 6.39 Gbits/sec receiver
iperf3: the client has terminated

fio test results:

root@truenas[~]# fio --filename=./rawrite --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=8k7030test --size=4G
8k7030test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=psync, iodepth=16
...
fio-3.28
Starting 16 processes
8k7030test: Laying out IO file (1 file / 4096MiB)
Jobs: 16 (f=16): [m(16)][100.0%][r=278MiB/s,w=118MiB/s][r=35.6k,w=15.1k IOPS][eta 00m:00s]
8k7030test: (groupid=0, jobs=16): err= 0: pid=9329: Mon Dec 25 14:54:25 2023
read: IOPS=43.7k, BW=342MiB/s (358MB/s)(20.0GiB/60002msec)
clat (usec): min=2, max=2651, avg=14.50, stdev=20.75
lat (usec): min=2, max=2651, avg=14.61, stdev=20.76
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 9],
| 70.00th=[ 10], 80.00th=[ 21], 90.00th=[ 44], 95.00th=[ 50],
| 99.00th=[ 73], 99.50th=[ 82], 99.90th=[ 119], 99.95th=[ 184],
| 99.99th=[ 668]
bw ( KiB/s): min=210904, max=1301635, per=100.00%, avg=350329.69, stdev=7627.49, samples=1888
iops : min=26357, max=162696, avg=43785.01, stdev=953.41, samples=1888
write: IOPS=18.8k, BW=147MiB/s (154MB/s)(8794MiB/60002msec); 0 zone resets
clat (usec): min=9, max=2927, avg=813.49, stdev=296.16
lat (usec): min=9, max=2927, avg=813.62, stdev=296.18
clat percentiles (usec):
| 1.00th=[ 21], 5.00th=[ 153], 10.00th=[ 478], 20.00th=[ 611],
| 30.00th=[ 668], 40.00th=[ 734], 50.00th=[ 807], 60.00th=[ 898],
| 70.00th=[ 1004], 80.00th=[ 1106], 90.00th=[ 1188], 95.00th=[ 1237],
| 99.00th=[ 1303], 99.50th=[ 1319], 99.90th=[ 1352], 99.95th=[ 1352],
| 99.99th=[ 1385]
bw ( KiB/s): min=99644, max=558695, per=100.00%, avg=150265.31, stdev=3251.28, samples=1888
iops : min=12447, max=69831, avg=18777.53, stdev=406.38, samples=1888
lat (usec) : 4=5.32%, 10=46.45%, 20=4.31%, 50=11.17%, 100=4.02%
lat (usec) : 250=0.21%, 500=1.79%, 750=9.55%, 1000=8.11%
lat (msec) : 2=9.07%, 4=0.01%
cpu : usr=0.92%, sys=7.75%, ctx=1073548, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2624309,1125639,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=342MiB/s (358MB/s), 342MiB/s-342MiB/s (358MB/s-358MB/s), io=20.0GiB (21.5GB), run=60002-60002msec
WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=8794MiB (9221MB), run=60002-60002msec

 

[AndroidBot]

Tunables

P2

 

[jgreco]

Well, as I said before, the jumbo paths in the drivers are kinda known to suck, you might want to try using 1500 and then relying on offload for a speed boost. Jumbo is basically a remnant of the early to mid 2000's when offload wasn't a thing and the CPU had to do it all, and the savings from fewer interrupts meant something. Between offload and interrupt coalescing, there's much less reason for jumbo now. You really need to crank everything up to "ideal" if you want to get maximum performance because the "weakest link" thing is real. My guess is that you've probably done something bad like a RAIDZ2 pool (you didn't really provide the system specs) which would easily explain five seconds of burst speed before the write throttle gets you, but there's enough moving parts in there that it is a bit hard to tell. ZFS is counterintuitive when it comes to large sequential file performance on RAIDZ.

 

[AndroidBot]

UPDATE:

Everyone, please ignore this thread, the issue was thermal throttling or failure in the driver (I installed latest) in Intel X710T, that's why initially it was copying fast and then dropping.

As soon as I switched it to another PCIE card from Starcom, everything started working perfectly, now I get 1.1GB transfers, with 9000 MTU

Thank you for your help @jgreco

 

[jgreco]

Yeah I guess I should have looked at you a bit funnier when you said E5-2699A. Merry Christmas.