Performance Issues Exploring Other File Sharing Protocols

[Windows7ge]

Up until now I have been using SMB on FreeNAS and as many people have told me my 8 drive RAIDz2 (7200RPM disks) wasn't quite as fast as it should be as a network share (~500MB/s).

Well, I decided to build a 12x1TB SSD RAIDz2 and found that I had near identical performance using SMB.

Talking to people I was told that SMB/CIFS is heavily single threaded and not optimized on FreeNAS or Linux in general.

Someone found a guide that would optimize SMB a little bit and we increased the performance by ~100MB/s but that isn't nearly enough for a full SSD array with a 20Gbit network link and significantly higher internal performance. However, this at least told us that the bottleneck was SMB itself. For this reason we (as in me and the people currently helping me) are trying to explore with other file sharing protocols such as NFS & iSCSI however my efforts to set these up failed. I have no experience with either.

NFS - It had appeared to be pretty strait forward but disabling SMB and enabling NFS made the server completely drop off the network (excluding the WebUI that still worked). I could not locate the server let alone the NFS share. It was explained to me that by default Windows will use both TCP & UDP while NFS will only use TCP. I tried to disable UDP but couldn't locate a setting on the Windows NIC. FreeNAS had a NFS configuration option to enable UDP for clients that need it (ie. Windows) however this still did not make the server show up on the network.

iSCSI - Exploring this was a complete failure. From my current perspective it is quite intimidating with several pages that need to be configured and even then I couldn't figure out how to connect to it. I have been told though that this very well could be the most performance promising solution.

If you need system specifications let me know. I don't really see them necessary to list at this time.

 

[MatthewSteinhoff]

NFS - It had appeared to be pretty strait forward but disabling SMB and enabling NFS made the server completely drop off the network

NFS isn't something you'd browse to like you do SMB. NFS servers typically don't 'show up' on the network.

You should read up on how NFS works then make sure your clients can use NFS. Many versions of Windows do not. All UNIX variants should.

If you have any specific mounting questions, I'll be happy to help but do please read up on NFS first so we're speaking the same language.

Cheers,
Matt

 

[cobrakiller58]

Also a singal vdev pool is not going to be the fastest configuration as far as I understand zfs, if you had those 12 SSD's in 6 mirrored pairs I'm sure you would be able to get better performance on SMB than with RaidZ2.

 

[Windows7ge]

NFS isn't something you'd browse to like you do SMB. NFS servers typically don't 'show up' on the network.

You should read up on how NFS works then make sure your clients can use NFS. Many versions of Windows do not. All UNIX variants should.

If you have any specific mounting questions, I'll be happy to help but do please read up on NFS first so we're speaking the same language.

Cheers,
Matt

Informative.

My research shows Windows Home does not support it. Windows 10 Pro does (Control Panel > Programs > Turn Windows Features On or Off > Services for NFS) and Windows Enterprise also does. A little more research says I'll have to use command line to actually mount the share with a drive letter. As you stated it doesn't display itself as a network share. I'm comfortable with commandline and the setup looks fairly strait forward.

I will try to look up information when I hit the next roadblock. If what I locate doesn't help I'll present you what I tried to use and you can tell me how incredibly far off I was to the right answer. I'm in unknown territory here.

 

[Elliot Dierksen]

Up until now I have been using SMB on FreeNAS and as many people have told me my 8 drive RAIDz2 (7200RPM disks) wasn't quite as fast as it should be as a network share (~500MB/s).

You didn't say if you are trying to read or write from the NAS. That makes a big difference. The specs on the NAS would be helpful, particularly the info on the CPU and RAM.

 

[Windows7ge]

Also a singal vdev pool is not going to be the fastest configuration as far as I understand zfs, if you had those 12 SSD's in 6 mirrored pairs I'm sure you would be able to get better performance on SMB than with RaidZ2.

I tried 12x RAID0, I tried 3x 4 drive RAID5, I tried 2x RAID0, everything yielded ~500MB/s using SMB. Nothing made a difference. It's not the array. Something else is a limiting factor.

I should also mention these different configurations did yield different internal results running fio. The 12x RAID0 showed sequential write performance of 2.3GB/s when tested using a blocksize of 128k.

 

[Windows7ge]

You didn't say if you are trying to read or write from the NAS. That makes a big difference. The specs on the NAS would be helpful, particularly the info on the CPU and RAM.

The server is used on a primarily write basis. These performance numbers are writing to the array over a network share using a 9.5GB.mp4 file as a test subject.
The system is a dual socket LGA2011 motherboard with two LGA2011 E5-2670's for a total of 16 cores/32 threads. The RAM is 16x8GB Kingston DDR3 1600MHz Unbuffered ECC memory for a total of 128GB of RAM.
Every single SSD has its own SATAIII link. So we can eliminate that question.

If you want anything else more specific feel free to ask.

 

[Elliot Dierksen]

The server is used on a primarily write basis. These performance numbers are writing to the array over a network share using a 9.5GB.mp4 file as a test subject.

You might have a issue with synchronous writes. I haven't tried to do high volume transfers via CIFS/SMB, but I definitely have with NFS. I was getting in the 500M range over a 10G connection as well. I added an SSD as an SLOG (ZFS intent log). After adding that I was able to get ~= 4G write performance. This was with a pool with 2 vdevs of 8 x 1TB 7200 rpm SATA drives.

 

[Windows7ge]

You might have a issue with synchronous writes. I haven't tried to do high volume transfers via CIFS/SMB, but I definitely have with NFS. I was getting in the 500M range over a 10G connection as well. I added an SSD as an SLOG (ZFS intent log). After adding that I was able to get ~= 4G write performance. This was with a pool with 2 vdevs of 8 x 1TB 7200 rpm SATA drives.

Yeah the only problem there is that on ZFS a SLOG is only good for sync writes for applications such as a database or VM. A Network share is almost entirely or is entirely async so a SLOG will actually make the process even slower especially if you set sync=always

 

[Windows7ge]

@MatthewSteinhoff Progress. I enabled the NFS service in Windows. With the service enabled I can search the network via the servers IP address. It shows me the share folder labeled /mnt/{shareName}. I ran into an issue with permissions though. Regardless of my configuration the server thinks I'm "Other" meaning I can't keep the share permissions strictly under my account. To temporarily test the connection I enabled Other for rwx and I was able to enter the share.

However now I have a bug. Attempting to write a file I receive an error saying that an identical file name already exists. If I quit, a file pops up in the directory with a size of 0KB. If I try to overwrite it it stalls and doesn't write anything. I suspect it's related to the UDP issue on the Windows size that I was warned about as I was told NFS uses strictly TCP by default.

 

[MatthewSteinhoff]

You've still got a permission problem. Double check the user and group mappings for your NFS share as well as your file permissions. Something is out of alignment.

(Unless you have 'Serve UDP NFS clients:' checked (it should be unchecked), UDP doesn't play any part of this problem.)

Cheers,
Matt

 

[Windows7ge]

You've still got a permission problem. Double check the user and group mappings for your NFS share as well as your file permissions. Something is out of alignment.

The dataset permissions are my username and group, rwx permissions for me only, permission type Unix
The NFS share mapall user my username, mapall group my group
Everything looks correct to me. Configured this way I'm not allowed in the share at all.

 

[MatthewSteinhoff]

What command are you using to mount the share? Maybe a locking issue?

Try this horribly permissive mount...

mount -o nolock,mtype=soft,fileaccess=7 \\freenas\mnt\Volume\TestShare\ Z:

Local Remote Properties
-------------------------------------------------------------------------------
Z: \\freenas\mnt\Volume\TestShare\ UID=-2, GID=-2
rsize=131072, wsize=131072
mount=soft, timeout=1.6
retry=1, locking=no
fileaccess=755, lang=ANSI
casesensitive=no
sec=sys

We had a similar problem to yours from Windows Server 2012 running our security camera software and writing to FreeNAS. It would mount fine but the cameras only created zero byte files. We could delete files and create file stubs but never change - even to rename - files. The same mount command worked fine from Windows Server 2008. This was back in 2014. I think we fell back to SMB at the time.

Just checked the server and we're back at NFS so I'm not sure what happened in the meantime.

Cheers,
Matt

 

[Windows7ge]

What command are you using to mount the share? Maybe a locking issue?

The command I used very simply was
mount \\[server-name]\[directory-to-share] Z:
With that it said the share was successfully mounted to drive letter Z:. Though it did not show up in My Computer(This PC). That's when I navigated to Network and found the share directory then mounted it as a network drive.

I've always had permissions issues with FreeNAS. Everything looks perfect and exactly how it should be and then it still doesn't let me in, or write to it. I wonder if it has something to do with the fact I'm using a Windows machine

While waiting, to save time, I decided to explore iSCSI and after managing to locate a guide it proved to go together easier than NFS is currently and it gave me the performance I was looking for. Backing up the data got a little more complicated but other than that it's working OK. We are currently trying to get Multi-path going but with it "configured" we're unsure as to weather or not it's actually working since NIC2 isn't showing any activity.