MacOS Mojave and FreeNAS 11.1-U6/11.2-RELEASE

[Jason Keller]

Just a shout out there to see if anyone else has had the terrible experience I've had on my iMac Pro trying to write files to my SMB share. High Sierra worked fine on my old Mid-2011 Macbook Pro, but on iMac Pro/Mojave writing to the share is a miserable, stop/go experience that usually ends in an error. Reads and deletes appear to be unaffected.

It'll burst through anywhere from 50MB-2.5GB, then suddenly pause, during which time all networking on the Mac seems to stall (even in Chrome, the effects are felt far and wide as web pages refuse to load). This lasts up to a full minute, then network traffic seems to resume. This happens even on the latest 10.14.2 update just released for Mojave, under both FreeNAS 11.1-U6 and 11.2.

Solved! See below.

 

[anodos]

I noticed some significant changes in Mojave SMB behavior. net.smb.fs.version: 303000 One thing I noticed is that it's compounding SMB2 requests differently. I'm still in early investigation of it.

But before we get too far afield. Try setting the following sysctl tunable sysctl net.inet.tcp.reass.maxqueuelen=1448

Explanation of the issue is here: https://github.com/freenas/freenas/pull/2206/commits/74812ca86a794e07397a1a3547eef843f07a8217

 

[Jason Keller]

@anodos - After my brief attempt this morning, this did indeed improve but not entirely alleviate the situation. It brought the stalling/seesawing down significantly, to a few seconds at a time instead of almost a minute. Should I try increasing it further? I'll make myself as available as possible for testing to help get this resolved.

 

[anodos]

@anodos - After my brief attempt this morning, this did indeed improve but not entirely alleviate the situation. It brought the stalling/seesawing down significantly, to a few seconds at a time instead of almost a minute. Should I try increasing it further? I'll make myself as available as possible for testing to help get this resolved.

Feel free to increase the value of the sysctl. Perhaps try a wired connection to see if the issue goes away entirely.

 

[Jason Keller]

Thanks @anodos ; all of this has been occurring on a wired connection (the 10Gbe port on my iMac Pro, at 1Gbe speed due to the switch). I'll increase to maximum 11586 and retest.

 

[Jason Keller]

Most interestingly, even on the maximum setting on this sysctl, it does not alleviate it any further.

 

[Jason Keller]

After the third packet capture last week, Apple still has yet to get back to me on my case regarding this. @anodos have you found anything else on your end?

 

[anodos]

I'm preparing to test Samba 4.7.12 with FreeNAS 11.2 with a variety of issues I need to resolve. I'll see if behavior is different with upstream samba.

 

[troehl]

I had the same Problem, SMB was almost unusable.
Setting strict sync = no , solved the problem. Not seen any disadvantages so far.

 

[Jason Keller]

Solved!

The workaround/solution as follows...

• Set sysctl to allow a full TCP window to arrive out of order... thanks @anodos
  • sysctl net.inet.tcp.reass.maxqueuelen = 1448
• Set auxiliary parameter within the SMB service... thanks @troehl
  • strict sync = no

And as always, for Mac compatibility make sure you have the following VFS set on all SMB shares...

• zfsacl
• zfs_space
• streams_xattr
• catia
• fruit

Thus far, I can now write to the array at line-rate gigabit again. Once I get my 10 gigabit switch we'll see what it can really do.

 

[Johnny Fartpants]

I noticed some significant changes in Mojave SMB behavior. net.smb.fs.version: 303000 One thing I noticed is that it's compounding SMB2 requests differently. I'm still in early investigation of it.

But before we get too far afield. Try setting the following sysctl tunable sysctl net.inet.tcp.reass.maxqueuelen=1448

Explanation of the issue is here: https://github.com/freenas/freenas/pull/2206/commits/74812ca86a794e07397a1a3547eef843f07a8217

sysctl net.inet.tcp.reass.maxqueuelen = 1448 fixed it for me. Many thanks @anodos

 

[mjt5282]

this tcp sysctl seems to have sorted out writing over wifi to my NFS shares. Now showing 20-30 MB/s write throughput! Thanks for posting this!

 

[seanm]

Is this fixed as of the current release (11.2-U2.1)?

 

[Jason Keller]

Don't know if it'll be fixed in the vanilla update or not...check the sysctls and settings on a vanilla install/share setup. Regardless these are the settings needed to make it work well.

 

[seanm]

My system is pretty close to vanilla, and indeed `sysctl net.inet.tcp.reass.maxqueuelen` gives 1448, so I guess it's fixed in 11.2-U2.1.

 

[Jason Keller]

My system is pretty close to vanilla, and indeed `sysctl net.inet.tcp.reass.maxqueuelen` gives 1448, so I guess it's fixed in 11.2-U2.1.

That portion was already slated to be changed prior to this issue being filed; you still would have to check on the other settings to ensure a complete fix.

 

[nikbok]

Hi.

Running Mojave 10.14.3 (last update as of 15/3) and I'm on 11.2-U2.1, I have some SMB write issues.
Normaly when writing larger files everithing works fine but writing smaller files ends up in very very slow transfers.
Both sequential and random writes of 64KB to apx 1MB files are slow, other sizes are OK.

My settings on FreeNAS

• sysctl net.inet.tcp.reass.maxqueuelen = 1448
• strict sync = no
• zfsacl, zfs_space, streams_xattr, catia, fruit

are all in place.

SMB on Mojave

SMB on Sierra

AFP

AFP from same 10.14.3 as well as other Mac (10.13.x, 10.12,x, 10.11.x) client works fine,
SMB from windows 10 and windows 7 clients works fine,
SMB from other Macs (10.13.x, 10.12,x, 10.11.x) works fine as well.

Any ideas?

 

[anodos]

Try turning off samba's internal AIO (new feature in Samba 4.9 / FreeNAS 11.2-U2). This is a share-level parameter (aio write size = 0, aio read size = 0).
Also as point of reference try turning off fruit and repeating the test. If problems persist across these changes, begin by looking for errors in logs, and saturation (storage, CPU, network, etc).

 

[nikbok]

Turning off sambas internal AIO did not help, same with turning off Fruit.

Saturation:
On Mac side it is not even close to the full saturation on anything. Nothing i any log file...
On FreeNAS side there are no big jumps or changes, smbd proces is between 15-30% during benchmark and when it hits "brake" goes down to under 1%. No other processes visible to go up.
Logs:
log.smbd does not change under the test, should I change log level?
Any other log files I should look into?

Thax.

 

[anodos]

Turning off sambas internal AIO did not help, same with turning off Fruit.

Saturation:
On Mac side it is not even close to the full saturation on anything. Nothing i any log file...
On FreeNAS side there are no big jumps or changes, smbd proces is between 15-30% during benchmark and when it hits "brake" goes down to under 1%. No other processes visible to go up.
Logs:
log.smbd does not change under the test, should I change log level?
Any other log files I should look into?

Thax.

Check console on the Mac for SMB errors. Also check /var/log/samba4/log.smbd. Do note that those parameter changes require a full samba restart to ensure that settings have been cleanly applied.