Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Encrypted SMB3 speed questions

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

mledford

Newbie
Joined
Feb 12, 2015
Messages
2
I want to use SMB3 for encrypted connections however the performance hit is much more than I expected. I have searched high and low and it appears to me that it should be faster but I might be missing something.

My software:
  • FreeNAS-9.3-STABLE-201502110455
First my hardware:
  • ASRock C2750D4I
  • Intel(R) Atom(TM) CPU C2750 @ 2.40GHz
  • 2 x 8GB DDR3 PC3-12800 Unbuffered ECC
  • 6 x WD Red 3TB - WD30EFRX, raidz2, using GLEI

The connecting client is Mac OS X 10.10. I start out with having "smb encrypt = auto" so the connections are not encrypted. I have verified it is not encrypted by using wireshark. I repeatably transfer a 1 GB file from the client to the server in ~10 seconds. This works out to roughly 100 MB per second. If I go back and turn "smb encrypt = mandatory" the same file takes ~2 minutes 5 seconds. That works out roughly to 8 MB per second. I have verified it is encrypted by using wireshark. When I look at top in the console while encryption is turned on and the file is being transfered the samba process is peaking one of the cores at 100% so that appears to be the performance bottleneck. This is expected if you were to do the encryption without any acceleration. However, my CPU does implement AES-NI.

Now, I know that GLEI uses my hardware's AES-NI instructions to do fast encryption of the disk however it's not clear if samba also takes advantage of it. So I did some research and came across a Samba posting that said they offload AES to GSSAPI and that it is up to the vendors to implement the optimization of offloading it to the AES-NI instructions. I haven't been able to locate if FreeBSD's implementation does offload it or not. Given my performance it appears that it doesn't.

My questions are:
  1. Should I expect better SMB3 encryption speeds given my hardware and software?
  2. If so is there a configuration I am missing to enable the better performance?
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,392
In not entirely sure but i think the decrease in performance is from context switching from kernel space to use space where decryption happens and and back to kernel space for the write to disk. In a normal write to disk FreeBSD uses zero copy to transfer bytes off the wire straight to disk and doesn't do any processing on them in user space. This feature is a huge performance gain. When dycrypting packets you currently need to move data into userspace to decrypt then move it back to kernel so it can be written to disk.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I'm not sure I have an answer, and I tend to think that SweetAndLow is incorrect and that the problem isn't context switching. But I also don't have evidence to disagree nor do I have a much better idea. My guess is Samba isn't compiled by FreeNAS to support it, but that's just my feeling.

What I'd recommend you do mledford is put in a ticket and mention what you said in this ticket. Let the devs have a chance to either implement AES-NI properly in Samba if its not or tell you that it's because of whatever (context switching???).

I would like to think that the Avoton is more than capable of handling this workload (maybe not at Gb saturation speeds with encryption, but something like 50-75MB/sec).
 

mledford

Newbie
Joined
Feb 12, 2015
Messages
2
What I'd recommend you do mledford is put in a ticket and mention what you said in this ticket. Let the devs have a chance to either implement AES-NI properly in Samba if its not or tell you that it's because of whatever (context switching???).

I would like to think that the Avoton is more than capable of handling this workload (maybe not at Gb saturation speeds with encryption, but something like 50-75MB/sec).

Thank you cyberjock!

I actually have gone to the Samba technical list and inquired about this. It appears that they do not delegate this encryption off to another library. I have started a dialog about it and my hope is to work with them to get this implemented so my box, and many others down the road, will benefit from it. :)

I will work on submitting a bug report to FreeNAS soon with this information.
 

seanm

Senior Member
Joined
Jun 11, 2018
Messages
522
Anyone know the current status of this in current FreeNAS (11.1-U5)? On systems with AES CPU instructions (particularly those currently sold by iXsystems), does enabling "smb encrypt" cause a big slow down?
 
Status
Not open for further replies.
Top