Very slow SMB read speeds after TrueNAS 12 upgrade

[lonelyzinc]

SMB read performance over 10G has taken a huge dive after updating from 11.3-U5 to 12. Write performance remains the same. Anyone else experiencing this, know how to fix? Using MTU 9000 on both server and client.

UPDATE: You can read all the testing I did below, but have since had to roll back to 11.3-U5 to resolve this issue.

 

[x130844]

same here, I use to get 600MB/s. now only 250MB/s
Aquantia on clients and Chelsio on freenas

 

[lonelyzinc]

Even my USB 1G NIC is faster for reads. I tested another TB3 10G NIC I have, based on the Aquantia AQC107(not-B0) chipset, and it has the same problem reading from SMB on TrueNAS 12.

Using the same computer for all of these benchmarks in the screenshots. The SMB share is stored on a Samsung EVO 970 NVMe drive capable of speeds over 3,000MB/s

 

[MikeyG]

@lonelyzinc I have a ticket for a similar issue if you want to take a look: https://jira.ixsystems.com/browse/NAS-107593

 

[anodos]

How is performance over SMB if read is over loopback?
smbclient //127.0.0.1/<SHARE> -U <your user>
Once you have the SMB prompt, run "get large_file /dev/null"

 

[MikeyG]

For me I get: (average 1187794.4 KiloBytes/sec)

My issue is reading smaller files like photos. Large files seem to be fine. Is there an equivalent test for an entire folder @anodos ?

 

[lonelyzinc]

How is performance over SMB if read is over loopback?
smbclient //127.0.0.1/<SHARE> -U <your user>
Once you have the SMB prompt, run "get large_file /dev/null"

For some reason the username at the end defaults to a domain account, even though I have removed the binding to AD. What is the syntax for telling it to be a local account?

smbclient //127.0.0.1/<SHARE> -U [hostname]\[user]

Even the above seems to default to the AD domain.

 

[anodos]

For some reason the username at the end defaults to a domain account, even though I have removed the binding to AD. What is the syntax for telling it to be a local account?

smbclient //127.0.0.1/<SHARE> -U [hostname]\[user]

Even the above seems to default to the AD domain.

hostname\\username

 

[lonelyzinc]

hostname\\username

Seems normal there, but like I said, it's the SMB connection on our clients that is slow

getting file \centos8.iso of size 7554990080 as /dev/null (1741307.5 KiloBytes/sec) (average 1741307.5 KiloBytes/sec)

 

[lonelyzinc]

Appears to be related to using Jumbo frames at MTU 9000

If I have MTU 9000 configured on the server/client , the write speeds are around 800MB/s, reads about 29MB/s. If I configure them both to MTU 1500, then I get about 430MB/s write and 330MB/s read.

Are Jumbo Frames broken on TrueNAS 12? I'm not a networking expert, but my tests seem to indicate that. I had previously tuned the frame size for performance on FreeNAS 11.3-U5 and earlier, but those settings are not working well any longer.

 

[lonelyzinc]

After thoroughly testing speeds using MTU 9000/1500 and 5 different models of TB2/TB3 10G NICs over 4 different Macs -- have decided to go back to FreeNAS 11.3-U5. Fortunately, for my own sanity, it performs the same way I remember it -- very fast read/write over SMB using MTU 9000.

Hopefully others will flag this TrueNAS 12.0 problem down too. Not sure what it is about my setup that deserves this on the initial release. Just a basic SMB share and 10G NICs using MTU 9000.

 

[robinmorgan]

Hello all, I also tried dropping Jumbo Frames down to 1500. No change for me. I also tried the snmp fix with no luck. I believe this bug has a ticket against it... https://jira.ixsystems.com/browse/NAS-107593

Any have any brave ideas?

R

 

[anodos]

Anecdotally, since it's not my ticket, affected users have reported speed issues with multiple services (iSCSI, SMB, NFS, AFP). So likely not in the SMB service itself. You can try local transfers over loopback `smbclient //127.0.0.1/<SHARE> -U <smbuser>%<password>` then 'mget <large_file> > /dev/null' to isolate somewhat from your local network / NICs.

edit: sorry, noticed I already posted something like that here

 

[robinmorgan]

mget <large_file> > /dev/null

Anecdotally, since it's not my ticket, affected users have reported speed issues with multiple services (iSCSI, SMB, NFS, AFP). So likely not in the SMB service itself. You can try local transfers over loopback `smbclient //127.0.0.1/<SHARE> -U <smbuser>%<password>` then 'mget <large_file> > /dev/null' to isolate somewhat from your local network / NICs.

edit: sorry, noticed I already posted something like that here

I’ve tried this! Thank you... this is what I got but not sure if its good or bad news?

 

[Kuro Houou]

Looks like I ran into this issues as well and went down a rabbit hole thinking it was my hardware as I did a 2.5Gbps upgrade as well right after upgrading to TrueNAS 12. The odd thing is I have two servers, a backup nas and main nas, both were on TrueNAS 12 and only the main nas had this issue, which made me think it was hardware and not software related.. so not all system exhibit this issue.

Can’t even hit 2.5Gbps [SOLVED]

Thought I would do another copy from v02 to my PC and see how it does, as you can see, its only reading from 4 disks like v01.. but its reading at the full 280MB/s in total..

www.truenas.com

 

[x130844]

Looks like I ran into this issues as well and went down a rabbit hole thinking it was my hardware as I did a 2.5Gbps upgrade as well right after upgrading to TrueNAS 12. The odd thing is I have two servers, a backup nas and main nas, both were on TrueNAS 12 and only the main nas had this issue, which made me think it was hardware and not software related.. so not all system exhibit this issue.

Can’t even hit 2.5Gbps [SOLVED]

Thought I would do another copy from v02 to my PC and see how it does, as you can see, its only reading from 4 disks like v01.. but its reading at the full 280MB/s in total..

www.truenas.com

Same here, I went back to freenas 11. everything is back. the speed and reliability and options.
I wonder if the upgrade itself was not the problem. If I have time, I'll try to do a fresh install of truenas 12 instead of a migration.

 

[D.Ryzhkov]

same problem - slow speed after upgrade 12.0 or 12.1

https://jira.ixsystems.com/browse/NAS-108081

 

[anodos]

What if you turn off aio read / write path on the SMB share?

Code:

aio read size = 0
aio write size = 0

 

[lonelyzinc]

same problem - slow speed after upgrade 12.0 or 12.1

https://jira.ixsystems.com/browse/NAS-108081

As the OP, was going to say that I was pleasantly surprised to find that it's fixed for me with 12.0-U1!!

 

[lonelyzinc]

Actually, it's the fastest I've ever seen!! Never went above 800s for read on 11.3