Very slow SMB read speeds after TrueNAS 12 upgrade

[Kuro Houou]

Curious if this is really fixed or not.. one person said no the other yes...

 

[Kuro Houou]

Same here, I went back to freenas 11. everything is back. the speed and reliability and options.
I wonder if the upgrade itself was not the problem. If I have time, I'll try to do a fresh install of truenas 12 instead of a migration.

I did a fresh 12.0 install back when doing my testing.. didn't matter. I think 12.0 was just broken. The SMB aio setting was the temp fix.. but I just reinstalled 11.3u5 and am on that for both my servers now.. will not upgrade to 12 until I know for certain this issue is fixed..

 

[Kuro Houou]

Actually, it's the fastest I've ever seen!! Never went above 800s for read on 11.3

Curious did you go from 11.3 straight to 12.1? Or were you on 12.0?

 

[MikeyG]

12.0-U1 definitely did not fix the SMB speed issues for me. Neither did the AIO settings.

 

[tiberiusQ]

Does this perf. issue only happens on 10g+ connections or also on 1g or 1g lagg situations ?

 

[MikeyG]

I have only seen it on 10g connections for myself and others that have posted. It's possible that it's just less noticeable on 1g. My 1g connections are actually much faster than my 10g though when I switch over.

 

[anodos]

I have only seen it on 10g connections for myself and others that have posted. It's possible that it's just less noticeable on 1g. My 1g connections are actually much faster than my 10g though when I switch over.

Is this exclusively via SMB or are you seeing similar issues with other protocols (iscsi, NFS, etc)?

 

[MikeyG]

@anodos I'm seeing issues with iSCSI as well. In that case, it seems to be write speeds that take a big hit.

 

[lonelyzinc]

Curious did you go from 11.3 straight to 12.1? Or were you on 12.0?

I never said 12.1, it's 12.0-U1. I don't think there is an actual upgrade path per se, it just boots up directly to it like a network switch firmware AFAIK. I just booted from 11.3-U5 to 12.0-U1. You can always boot back to the old version if you don't upgrade your zpool.

 

[anodos]

I never said 12.1, it's 12.0-U1. I don't think there is an actual upgrade path per se, it just boots up directly to it like a network switch firmware AFAIK. I just booted from 11.3-U5 to 12.0-U1.

In 12.0-U1 I made several tweaks / bugfixes to samba's AIO, specifically in regards to behavior in overload situations. Big picture, AIO can be fine-tuned by modifying the values of the `vfs.aio.*` sysctls sysctl -a | grep aio.

Of particular note are vfs.aio.target_aio_procs and vfs.aio.max_aio_procs. These control the low-water and high-water marks for kernel threads handling AIO requests. You can view vfs.aio.num_queue_count to see the current queue depth for AIO requests. The kernel will create new aio threads as needed until it hits the high water mark. As we receive AIO requests, if the current queue is too deep, aio_read() or aio_write() will fail with EAGAIN.

Starting in U1 we fall-back to synchronous pread() / pwrite() if our aio is overloaded (we were passing EAGAIN back to clients originally, which provoked less than optimal responses from them). We also bumped up the values for target and max aio procs.

 

[ude6]

Can you detail the amendments of the aio settings?

This sounds similar to: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212128

 

[anodos]

Can you detail the amendments of the aio settings?

This sounds similar to: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212128

Not really a bug, it's just a matter of how you handle overload situations. We bumped target aio procs to 16 and max aio procs to 128. You can do further tweaking if you need to by setting different values for these sysctls.

 

[ude6]

This definitly helped my performance under smb. Went from below 30mb/s to above 80mb/s on 1gb lan.

 

[TallCoolOne]

I saw @lonelyzinc 's response that this was fixed for them in 12.0-U1 so tried that after having downgraded back to 11.3-U5. Still very slow read speeds over SMB (and other services are the same, also slow) over my 10Gbe interface.

The write speed (client side write operation) is a bit slower than before but not by much, but read speed is awful.
The 1Gbps interface works as expected.

@anodos I tried the local transfer over loopback, it gets around 200MB/s on smaller files and an NT_STATUS_IO_TIMEOUT error on larger ones. Clearly that's above the 25MB/s I'm seeing on my clients though. But with the local loopback that isn't going over the 10Gbe network interface right? Only that interface has this issue.

Note that AFP is equally slow over the 10Gbe, so it's not just SMB.

 

[TallCoolOne]

Also I saw someone mentioned changing the MTU helped, changing it to 1500 made no difference for me in the read speeds (still roughly 25MB/s), all it did was half my write speed also.

 

[nasbdh9]

Not really a bug, it's just a matter of how you handle overload situations. We bumped target aio procs to 16 and max aio procs to 128. You can do further tweaking if you need to by setting different values for these sysctls.

Have you tested the case of opening aio reading and writing plus smb encryption?
Now that both aio and smb encryption are turned on at the same time, it will cause transmission data errors, and if smb encryption is turned off, data errors will not occur.

 

[anodos]

Have you tested the case of opening aio reading and writing plus smb encryption?
Now that both aio and smb encryption are turned on at the same time, it will cause transmission data errors, and if smb encryption is turned off, data errors will not occur.

I'm not aware of issues with samba's AIO and SMB encryption. Can you provide a reference for this issue?

 

[hydrosaure]

Following this thread as I'm also suffering from slowdowns and now I'm starting to get SMB mounts disconnects with 12-U1.

pid 2026 (smbd), jid 0, uid 0: exited on signal 6 (core dumped)
pid 2131 (smbd), jid 0, uid 0: exited on signal 6
pid 2633 (smbd), jid 0, uid 0: exited on signal 6
pid 1854 (smbd), jid 0, uid 0: exited on signal 6
pid 16271 (smbd), jid 0, uid 0: exited on signal 6
pid 17322 (smbd), jid 0, uid 0: exited on signal 6
pid 17336 (smbd), jid 0, uid 0: exited on signal 6
pid 17552 (smbd), jid 0, uid 0: exited on signal 6
pid 21088 (smbd), jid 0, uid 0: exited on signal 6
pid 21286 (smbd), jid 0, uid 0: exited on signal 6
pid 21893 (smbd), jid 0, uid 0: exited on signal 6
pid 21899 (smbd), jid 0, uid 0: exited on signal 6
pid 21920 (smbd), jid 0, uid 0: exited on signal 6
pid 4836 (smbd), jid 0, uid 0: exited on signal 6
pid 27954 (smbd), jid 0, uid 0: exited on signal 6
pid 21965 (smbd), jid 0, uid 0: exited on signal 6
pid 79043 (smbd), jid 0, uid 0: exited on signal 6
pid 79059 (smbd), jid 0, uid 0: exited on signal 6
pid 79060 (smbd), jid 0, uid 0: exited on signal 6
pid 79079 (smbd), jid 0, uid 0: exited on signal 6
pid 1888 (smbd), jid 0, uid 0: exited on signal 6
pid 2515 (smbd), jid 0, uid 0: exited on signal 6
pid 8999 (smbd), jid 0, uid 0: exited on signal 6
pid 9262 (smbd), jid 0, uid 0: exited on signal 6

 

[ude6]

Might be this issue: https://www.truenas.com/community/threads/smb-connection-problem.89501/

 

[J-Lo]

We recently upgraded from FreeNAS to TrueNAS Core 12.0 and then to 12.0-U1 when it was released. We primarily used FreeNAS as a SMB store for our Veeam backups. After migrating to TrueNAS, Veeam backups would fail. Transfer speeds were also slow, maxing out at around 400mbps inbound via ethernet to TrueNAS. Veeam was indicating that it failed to write to the SMB share.

It appears we have "resolved" the issue by enabling System > Advanced > Autotune. This added a bunch of stuff in Tunables. We basically had nothing in tunables before autotune was enabled. (There were a couple of old entries for iohyve that I deleted.) After enabling autotune and rebooting TrueNAS, Veeam appears to be working. Inbound ethernet throughput seems to be back to normal with spikes above 1gbps.

Autotune does not appear to be recommended as a permanent / production solution, but it seems to be working for now.

If anyone else with these issues tries autotune and has success, please post back to this thread as it may help point to the problem.