SMB slower than iSCSI

[ColPanic]

I have an all in one server running FreeNAS 9.10 and esxi 6 and have been using iSCSI with great results. I decided to setup an SMB share to see how they compare and something doesn't add up.

If I create a disk across iSCSI in vmware and share it to windows, sequential reads and writes are 300/500 MB/s. Very reasonable. If I do an SMB share from Windows directly to FreeNAS it drops to 70-80 MB/s. Is this normal or have I done something wrong?

I tried changing the pool from sync=standard to sync=always but that didn't change anything. Its across the same virtual 10gb NIC.

The server in question:
Intel CP2600 - 2xE5-2670
106 GB RAM (32GB to FreeNAS)
ZFS volume is 8x WD Re 3TB drives in Raidz plus 2x HGST 400 GB SAS SSD as striped slog.
I also have another 400GB SAS SSD as L2ARC but I think I'll remove it.

Across SMB

iSCSI

 

[jdong]

This looks like sync write performance for a pool without a SLOG. Have you temporarily tried sync=disabled to see if that brings performance closer to iSCSI?

 

[ColPanic]

Thanks for the help. I just tried that and its the same. There is definitely a slog.

 

[ColPanic]

Interestingly I created an afp share and a Mac wrote to the save volume at its gigabit line speed. It seems to be just SMB.

 

[jdong]

Sorry I breezed through your system description too quickly. You're right, probably not SLOG related. Could you check the CPU usage while the transfer is in progress? SMB is single threaded and you might be hitting some limitations there. I'm not an expert at the mysterious SMB tuning knobs but hopefully someone else can chime in.


Sent from my iPhone using Tapatalk

 

[Ericloewe]

Weird antimalware solutions on the Windows side, perhaps?

 

[depasseg]

I also have another 400GB SAS SSD as L2ARC but I think I'll remove it.

You don't have enough RAM to support an L2ARC this large. You would need around 80-90G (1/5 - 1/8) for the plus the base 8G.
I would try without the L2ARC.

And an 8 wide RAIDZ1 is a disaster waiting to happen.

Is your windows machine a VM on the same ESX host?

@anodos any thoughts on the SMB performance?

 

[ColPanic]

The windows machine is on the same host but not the same datastore. Its on a separate SSD. Could be AV, I'll try that. Not processor related. I know the L2ARC isnt helping anything but that doesnt have anything to do write speed, does it? I also know that this is not a stable, long term RAID solution but for this particular situation its perfect.

 

[depasseg]

but that doesnt have anything to do write speed, does it

It might if it's trying to cache it. Which likely wouldn't apply to a zvol for the iscsi share. Not sure about the AFP share.

Also, the only other thing I can think of is that your transactions groups are filling RAM and therefore requires the writes to be flushed to the pool. This wouldn't apply to the datasets with Sync=forced, since those would get written to the SSD SLOG. So this could an edge case where sync writes could actually be faster than sync=disabled. But this is just a wild theory.

 

[Ericloewe]

Also, the only other thing I can think of is that your transactions groups are filling RAM and therefore requires the writes to be flushed to the pool. This wouldn't apply to the datasets with Sync=forced, since those would get written to the SSD SLOG. So this could an edge case where sync writes could actually be faster than sync=disabled. But this is just a wild theory.

Edit for misunderstanding:

That'd be very weird, more eloquent thoughts below:

 

[Ericloewe]

That kind of behavior would only make sense for bursty workloads, with enough time between bursts to flush the ZIL (the in-RAM one) to the pool. I really doubt that's happening here, unless some really weird stuff is going on with the networking (which seems incompatible with the iSCSI performance).

 

[ColPanic]

I removed the L2 cache and now it does this

iSCSI is still this

 

[depasseg]

It's hard to tell if it's related to the windows VM, ESX, or FreeNAS at this point.

Try a SMB connection from your Mac to FreeNAS (and compare against the AFP performance).

 

[ColPanic]

I also tried from a different windows VW and its the same. I'm starting to think its something network related on the Windows side. Because the iSCSI drive is presented by VMWare windows thinks its a local drive.

 

[ColPanic]

From as Mac OS VM on the same host over SMB (the previous Mac was a real one with physical NIC).

I don't know exactly how fast it was but its clearly close to iSCSI, so the problem must be on the windows side and showing up on two different VMs. Both have 10g NICs with jumbo frames connected to the same vswitch as iSCSI. (the mac VM just has a virtual gigabit nic without jumbo frames and is connected to the freenas management NIC but it doesnt give a f***) AFP and SMB are the same from the Mac side.

I also tried connecting windows exactly the same way as Mac (1gb NIC, MTU-1500, connect to management NIC) its the same slow speed SMB

 

[Ericloewe]

Jumbo frames? Start by getting rid of those.

 

[ColPanic]

Jumbo frames? Start by getting rid of those.

I also tried connecting windows exactly the same way as Mac (1gb NIC, MTU-1500, connect to management NIC) its the same slow speed SMB

 

[anodos]

That's weird. Although it's possible that SMB is being limited by a single thread or some synchronous operation, the 1gb cap on your performance is awfully suspicious. It looks like samba on FreeNAS is being limited to gigabit speeds. There seem to be people who can get a lot more than gigabit speeds out of samba+FreeNAS and so something about your situation is unique.

Post contents of /usr/local/etc/smb.conf.

My best guess is that there is some sort of networking / samba bug that is making it think that your virtual NIC / virtual networking is 1gb instead of 10gb. Have you tried using a physical host for your SMB client?

You'll probably need to set samba's log level to "debug", then capture the entries in /var/log/samba4/log.smbd that correspond to your client (1) establishing a connection to a samba share and (2) transferring a file. Feel free to post it here once you have separated the wheat from the chaff.

 

[ColPanic]

SMB from a physical host is normal. Also SMB from windows server 1 to windows server 2 which has its storage on FreeNAS via iscsi is normal.

Mac and Linux can both write via SMB at the same speed as iSCSI.

SMB from a physical client to FreeNAS (normal)

SMB from Win to Win (where the share is hosted via esxi and iscsi on FreeNAS)

smb.conf

Code:

[global]
	server min protocol = SMB2
	server max protocol = SMB3
	interfaces = 127.0.0.1 10.0.1.53 10.0.2.2
	bind interfaces only = yes
	encrypt passwords = yes
	dns proxy = no
	strict locking = no
	oplocks = yes
	deadtime = 15
	max log size = 51200
	max open files = 942851
	logging = file
	load printers = no
	printing = bsd
	printcap name = /dev/null
	disable spoolss = yes
	getwd cache = yes
	guest account = root
	map to guest = Bad User
	obey pam restrictions = yes
	directory name cache size = 0
	kernel change notify = no
	panic action = /usr/local/libexec/samba/samba-backtrace
	nsupdate command = /usr/local/bin/samba-nsupdate -g
	server string = FreeNAS Server
	ea support = yes
	store dos attributes = yes
	lm announce = yes
	hostname lookups = yes
	time server = yes
	null passwords = yes
	acl allow execute always = true
	dos filemode = yes
	multicast dns register = yes
	domain logons = no
	local master = yes
	idmap config *: backend = tdb
	idmap config *: range = 90000001-100000000
	server role = standalone
	netbios name = FREENAS
	workgroup = WORKGROUP
	security = user
	pid directory = /var/run/samba
	create mask = 0666
	directory mask = 0777
	client ntlmv2 auth = yes
	dos charset = CP437
	unix charset = UTF-8
	log level = 1
   

[400]
	path = /mnt/z400
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	vfs objects = zfs_space zfsacl aio_pthread
	hide dot files = yes
	guest ok = no
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare
   

[zShare]
	path = /mnt/ZFS/zShare
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	vfs objects = zfs_space zfsacl aio_pthread
	hide dot files = yes
	guest ok = no
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare

 

[depasseg]

Sounds like your issue is with your Windows VM. Which virtual adapter are you using and do you have vmware tools installed?