AFP vs SMB 10GbE network speed problem in FreeNAS

[skyyxy]

Hi everybody here, I have a small problem.
My Nas server is base on Freenas 11.0U4 for 11.1U4 (different version just for test)
the hardware is:
intel E5-1620v3
Supermicro X10srl-f
32gb reg ecc ddr4 2400
sandisk 8u usb stick
lsi 9200-8i or 9300-8i
24 seagate EXOS 6TB HDD build raidZ2
Samsung 860evo 2TB for l2arc and no Zil
4*intel x520 dual ports 10GbE network adapter

Workstations all for mac osx user and running 10.11.x to 10.13.x and connect to Freenas server with 10GbE network and set mtu=9000 in MAC OSX settings.
AFP speed is good, around 1GB/S (aja speed test)
SMB speed is low, around 1XXMB/S (aja speed test)
I need smb connect for copy files,(the AFP link somethings will report error and stop work when I copy the folder has a lot of file or have a lot of child dirs. I dont know why)
Someone has idea with it ? Thanks advance.

 

[anodos]

Hi everybody here, I have a small problem.
My Nas server is base on Freenas 11.0U4 for 11.1U4 (different version just for test)
the hardware is:
intel E5-1620v3
Supermicro X10srl-f
32gb reg ecc ddr4 2400
sandisk 8u usb stick
lsi 9200-8i or 9300-8i
24 seagate EXOS 6TB HDD build raidZ2
Samsung 860evo 2TB for l2arc and no Zil
4*intel x520 dual ports 10GbE network adapter

Workstations all for mac osx user and running 10.11.x to 10.13.x and connect to Freenas server with 10GbE network and set mtu=9000 in MAC OSX settings.
AFP speed is good, around 1GB/S (aja speed test)
SMB speed is low, around 1XXMB/S (aja speed test)
I need smb connect for copy files,(the AFP link somethings will report error and stop work when I copy the folder has a lot of file or have a lot of child dirs. I don't know why)
Someone has idea with it ? Thanks advance.

Post contents of /usr/local/etc/smb4.conf

 

[Johnny Fartpants]

You need to disable SMB3 signing on your Mac clients otherwise you'll never get above 1Gb regardless of your Network speed.

From your Mac disconnect your share and and open Terminal and do the below.

sudo -s

echo “[default]” > /etc/nsmb.conf

echo signing_required=no >> /etc/nsmb.conf

exit

Reconnect to your share and enjoy your new 10Gb network.

 

[skyyxy]

You need to disable SMB3 signing on your Mac clients otherwise you'll never get above 1Gb regardless of your Network speed.

From your Mac disconnect your share and and open Terminal and do the below.

sudo -s

echo “[default]” > /etc/nsmb.conf

echo signing_required=no >> /etc/nsmb.conf

exit

Reconnect to your share and enjoy your new 10Gb network.

Sorry guys, I just back to home and tested your suggestion, some version is working and someelse is not. 11.0u4 working and 11.1u4 not. I dont know why.m What is your freenas version? Thanks a lot.

 

[Johnny Fartpants]

Sorry guys, I just back to home and tested your suggestion, some version is working and someelse is not. 11.0u4 working and 11.1u4 not. I don't know why.m What is your freenas version? Thanks a lot.

Ok, my guess is that you are now hitting another issue now on 11.1u4 with samba 'strict sync'. Your benchmark tool AJA is requesting sync writes and on 11.1 strict sync = yes by default in samba hence the slowness. On your FreeNAS system in the main SMB auxiliary parameters box enter strict sync = no and give that a try. Alternatively just try and copy some data over as that won't request sync and you should see your 10Gb performance.

 

[skyyxy]

Ok, my guess is that you are now hitting another issue now on 11.1u4 with is samba 'strict sync'. Your benchmark tool AJA is requesting sync writes and on 11.1 strict sync = yes by default in samba hence the slowness. In the main SMB auxiliary parameters box enter strict sync = no and give that a try. Alternatively just try and copy some data over as that won't request sync and you should see your 10Gb performance.

Really? so great, I will try that. Many thanks.

 

[dkusek]

Wanting to weigh in on this...

We cater to MacOS clients mainly so we have hit this issue. Long story short, if you want performance over 10G, DO NOT INSTALL ANYTHING OVER 11.0-U3.

We have added the strict sync = No auxiliary parameter to the smb global config. We then ran testparm -vvv to ensure that it was being loaded.

Regardless, MacOS still writes/read in BOTH AJA and with real data (11GB worth of 3FR RAW Images). This was confirmed and tested on MacOS 10.12.xx and 10.13.xx.

Another thing to note here is we have two Windows 10 clients in test. Prior to adding the "strict sync = no" aux param, these machine showed the SAME massive write/read speed drops. Once we added the aux param to the freeNAS smb global param, Windows 10 dramatically improved.

My end take... Windows and MacOS negotiate smb connections differently in that Windows actually listens to the freeNASserver. This is further confirmed with the whole SMB signing garbage that Mac introduced with 10.12.5 and above. During that time, we tried everything to mitigate that "security fix" on the FreeNAS side and were unable to do so. The only thing that fixed the issue was declaring a global default signing_required=no on the Mac OS client side.

My feeling is that we are going to have to dig into MacOS SMB code to find what the global setting for strict sync and set it on every MacOS client that is looking for 1G or 10G performance. ....Should be fun..

 

[kdragon75]

24 seagate EXOS 6TB HDD build raidZ2

Thats BALLZZY AF!

 

[kdragon75]

mtu=9000 in MAC OSX settings

Did you verify end to end MTU of 9k? If you don't have this set on all switches, MAC, and FreeNAS, your shooting yourself in the foot.

 

[dkusek]

Yes. MTU setting were at defaults. We also set them to jumbo frames. Same deal.

Verified with tcpdumb.

Not an MTU issue. But good idea to always check.

 

[skyyxy]

Did you verify end to end MTU of 9k? If you don't have this set on all switches, MAC, and FreeNAS, your shooting yourself in the foot.

 

[skyyxy]

Wanting to weigh in on this...

We cater to MacOS clients mainly so we have hit this issue. Long story short, if you want performance over 10G, DO NOT INSTALL ANYTHING OVER 11.0-U3.

We have added the strict sync = No auxiliary parameter to the smb global config. We then ran testparm -vvv to ensure that it was being loaded.

Regardless, MacOS still writes/read in BOTH AJA and with real data (11GB worth of 3FR RAW Images). This was confirmed and tested on MacOS 10.12.xx and 10.13.xx.

Another thing to note here is we have two Windows 10 clients in test. Prior to adding the "strict sync = no" aux param, these machine showed the SAME massive write/read speed drops. Once we added the aux param to the freeNAS smb global param, Windows 10 dramatically improved.

My end take... Windows and MacOS negotiate smb connections differently in that Windows actually listens to the freeNASserver. This is further confirmed with the whole SMB signing garbage that Mac introduced with 10.12.5 and above. During that time, we tried everything to mitigate that "security fix" on the FreeNAS side and were unable to do so. The only thing that fixed the issue was declaring a global default signing_required=no on the Mac OS client side.

My feeling is that we are going to have to dig into MacOS SMB code to find what the global setting for strict sync and set it on every MacOS client that is looking for 1G or 10G performance. ....Should be fun..

May be you are right, I will test to downgrade the system to 11.0u3. THANKS A LOT.

 

[RegularJoe]

AFP is designed different and as long as it is working it should ALWAYS be faster with a mac than SMB. 10gig is out there where a TrueNAS system would be a better way to spend time, i.e. less wasted time. Are you using MacPro's or 10gig thunderbolt adapters?

 

[skyyxy]

AFP is designed different and as long as it is working it should ALWAYS be faster with a mac than SMB. 10gig is out there where a TrueNAS system would be a better way to spend time, i.e. less wasted time. Are you using MacPro's or 10gig thunderbolt adapters?

No， Acturlly Im using MacPro 2010 with 10Gb pci-e network adapter.

 

[RKMStudios]

No， Acturlly Im using MacPro 2010 with 10Gb pci-e network adapter.

Did you every improve your SMB speeds over 10gbe? Facing a similar issue where AFP is near perfect, but SMB is less than half the potential throughput.

 

[anodos]

Did you every improve your SMB speeds over 10gbe? Facing a similar issue where AFP is near perfect, but SMB is less than half the potential throughput.

One problem with MacOS SMB clients is that they send an SMB2 Flush with _every_ write to the SMB share. This translates into a ZIL write. You can turn off the translation of flush to fsync() by setting strict sync=no. This can be defined on a per-share basis.

 

[RKMStudios]

One problem with MacOS SMB clients is that they send an SMB2 Flush with _every_ write to the SMB share. This translates into a ZIL write. You can turn off the translation of flush to fsync() by setting strict sync=no. This can be defined on a per-share basis.

Thank you, that definitely helped get SMB closer to AFP performance.

 

[anodos]

Thank you, that definitely helped get SMB closer to AFP performance.

Yeah, one more item on the "Macs do weird things" list. The source for the MacOS SMB client on apple's opensource site even explicitly states in a comment that fsync() is expensive and shouldn't be done all the time. I don't know enough about the OS and Apple's filesystems to say whether this is unnecessary.

 

[seanm]

Have you filed a bug with Apple? I could if you'd like. Do you have a link to that code and fsync comment?