Good benchmark results, but slow SMB speed

[caterpilar]

Hello,

I seted up my first TrueNas machine few weeks ago, and so far everything is fine except my SMB transfers, which are pretty slow.

Here are the specs of my server :

• AMD APU 3200G (4 cores)
• Asus Prime B450M-A
• 16 Go DDR4 non ECC
• Pool ZFS Raidz2, dedup off, lz4 on, atime off, synchronized standard: 6 x 6 To Toshiba N300
• Intel X520-DA2 (2x 10GB SFP+ ports), recongnized in TrueNas as " 10Gbase-Twinax ", jumbo frame on 9k
• Boot drive : Kingston KC2500 M.2 250 G M2, on a PCIe x1 card
• Switch Netgear S3300-28X with 10G ports, jumbo frame on 9k
• TrueNas Core 12.0-U8
• My home is 100% 10GB ready (EA / Cat 6A)
• Test computer : High-end workstation with 10GB Intel X550-T, Win 10 Home, jumbo frame on 9k

I spent days tweaking and testing everything I could find on the internet, but I struggle to find what is bottlenecking my SMB transfers. I tweaked both the settings of my 10GB NICs, and the record size of my pool, but without getting great results.

Here are the results :

1. DD write speed : 3152 MBytes/s
   
   Code:

   dd if=/dev/zero of=/mnt/TrueNas/Ressources/ count=1000010000+0 records in
   10000+0 records out
   10485760000 bytes transferred in 3.171790 secs (3305944133 bytes/sec)

   
   
   
2. DD read speed : 6502 MBytes/s
   
   Code:

   dd of=/dev/zero if=/mnt/TrueNas/Ressources/test bs=1M count=1000010000+0 records in
   10000+0 records out
   10485760000 bytes transferred in 1.537948 secs (6818021947 bytes/sec)

   
   
   
3. Iperf with -P20 on both ways : 9.83 - 9.86 Gbits/sec
   
   
4. SSH speed test : 303 MiB/s
   
   Code:

   yes | pv | ssh remote_host "cat >/dev/null"

   
   
   
5. Bonnie++
   
6. SMB speed : start at 650 -800 MB/s when it's in cache, then drop around 275/340 MB/s
   
   
   

In every tests, server' CPU is not stressed at all, barely reaching 30/40% :




I don't get it : DD test seems pretty eratic, showing unbelivable results. Bonnie++ show 1030 MB/s block speed, which is the raw ideal speed I can expect from my 6 x 6TB pool raidz2, Iperf show that bandwith is great and that I can saturate my network at full speed, but ssh speed test and smb are far below what I expect.
The CPU seems to not be the bottleneck neither.

What I miss ? I'm still in the learning process about networking, so I surely did something wrong.
Any guess or help will be apreciate, to find if it's configuration related or if it's the hardware that's the culprit.

 

[caterpilar]

As I can't edit my first post, here are some results I got from Atto and CrystalDisk (1Gb and 64GB) :

 

[xorus]

I experience the same issue, I tested on an simple setup, a PC with 32gb ram, 2.5gb (Realtek RTL3125B) connected directly to my PC, with an single SSD pool. Prior to install Truenas, I tested my setup with Windows server and I reach a stable 220mb/s write speed. Then I installed Truenas Scale (also tested with Truenas Core, same result). Unfortunately I'm able to get 2-3 great files transfer and then the speed drop drastically to around 50mb/s.

I don't know if it is a matter of cache or ARC, my knowledge of those aspect of ZFS is limited.

I do my test with the same 30gb file (.mkv) and most of the time, the transfer starts fast and drops after few seconds.

When I do the same transfer over a 1gb nic card, I reach a stable speed of 113mb/s.

It is very frustrating to get a more stable result (and faster) with an 1gb nic than with a 2.5gb nic...

In my understanding, a 2.5gb should not saturate the speed of a SSD pool...

There is something I'm not catching!

 

[MikeyG]

Aren't N300s spinning drives? Did you test DD speeds after disabling compression on the dataset? If it's fast reading from ARC then slows down, I would think you are hitting a drive speed limit.