NFS Share + Mac OS X Encoding Issues

[Juliano Arantes]

Hello guys!

A little backstory first, feel free to jump after the break if you don't feel like reading.

I've been using FreeNAS for a while now, since version 9.0, and setting it up to run smoothly with a Mac OS X client has always been a nightmare. Nothing wrong with FreeNAS, now that I have some experience the Windows clients work like a charm, no messing around needed. The problem seems to be Apple and its quest to complicate things...

I've spent the last week testing my setup and trying to get a decent sharing performance. Before anyone jumps in to tell me I need more RAM, I know I should and will add an extra 16GB soon, but right now that's not the issue. AFP was quickly ruled out as obsolete, since our clients currently run Sierra 10.2.6! SMB seemed to be the best option, and worked perfectly on Windows. After some tuning, I managed to get more than 100MB/s read/write speed on the Macs, but real world usage showed huge issues on the Macs when we had too many files in the same folder. Folders would take ages to load, file count would be wrong and randomly change, files disappearing, trying to move or delete files would cause the client to crash and disconnect. Believe me when I say I've tried every possible tweak available, truth is Apple's SMB implementation has always been flawed beyond repair.

I decided to go with NFS, and it has been ROCK SOLID! :) Performance is not that great as SMB, but it's enough and reliability more than compensates it. Folders load instantly, deleting takes seconds. We were able to work for three days without issues. But then someone uploaded a folder with accents and special characters to FreeNAS...
----------------------------
We live in Brazil and having folders named like "Decoração" and "Cerimônia" is fairly common. They upload just fine to FreeNAS, but appear empty to the Macs... After some researching I found out all the files were there, and the issue was a conflict between UTF-8 NFC and UTF-8 NFD. Added the "nfc" option to the mount command and now when clicking a folder it's content blinks on screen and Finder goes back to the previous folder... Click multiple times in a row and you end up inside the folder.

Read somewhere I should use convmv to convert older files to NFC, so I run:

Code:

convmv -f utf-8 -t utf-8 --nfc -r --no-test

But it doesn't find anything to convert, even files created after adding the "nfc" mouting option suffer from the same issue. Tried checking the files with "file -I" and the result is "charset=binary". Should be something else, right? That's when the system doesn't know the charset?

Adding "noac' to the mounting options fixes it, but that disables caching and brings a performance penalty. I do not want to use it! Does anyone have any idea how to make the Macs compatible with my NFS shares? Everywhere I read said adding the "nfc" option should do it. It improved my situation, but I still have issues with the attributes...

The mount command I'm currently using:

Code:

sudo mount -t nfs -o rsize=65536,wsize=65536,soft,intr,nolocks,locallocks,rdirplus,retrans=3,readahead=16,proto=tcp,resvport,timeo=600,noac,nfc 10.0.1.4:/mnt/Storage/Arquivos /Users/juliano/Documents/Servidor/

Hardware Specs:

FreeNAS-11.0-U4 (54848d13b)
Supermicro X9SCM-F
Intel Pentium CPU G2020 @ 2.90GHz
2x 8GB Kingston KVR16E11 1600MHz DDR3 ECC
5x 4TB Western Digital Reds

 

[Juliano Arantes]

Just updated one of my clients to OS X High Sierra and confirmed that SMB is still awful. Tried using NFSv4 instead of 3 but couldn't get it to work. Client connects, but the share is empty! Any ideas on how to get NFSv4 to show my files? That or making v3 work without the "noac" option...

 

[Juliano Arantes]

Still researching, hopefully someone will be able to give me some input! Just found out that "actimeo=0" also makes it work, just like "noac" with the benefit that the Finder window stops randomly refreshing. But that also means no cache, right?

Ran nfstat -c and despite being a complete newbie (first time I ran this command), I found the misses too high, and also the RPC TimedOut! Any inputs on that?

Code:

iMac-Juliano:~ juliano$ nfsstat -c

Client Info:

RPC Counts:

	Getattr	  Setattr	   Lookup	 Readlink		 Read		Write

	2683471		21912		98205			0		58474	   591632

	  Create	   Remove	   Rename		 Link	  Symlink		Mkdir

	  12840		  964			4			0			0			0

	  Rmdir	  Readdir	 RdirPlus	   Access		Mknod	   Fsstat

		  0		   40		  102	   116975			0		 5433

	  Fsinfo	 PathConf	   Commit

		  16		   16		 9095

RPC Info:

	TimedOut	  Invalid	X Replies	  Retries	 Requests

	  31157			0			1			1	  3614729

Cache Info:

  Attr Hits	   Misses	Lkup Hits	   Misses	BioR Hits	   Misses

	1266758	  2623352	  1659667		91023	   660444		58474

  BioW Hits	   Misses	BioRLHits	   Misses	BioD Hits	   Misses

		6420	   591632			0			0		 6869		 5469

  DirE Hits	   Misses

		4860		   41

 

[dlavigne]

Did you make any progress with this?

 

[Juliano Arantes]

I've tested AFP and can now say for sure that in addition to being deprecated it's slower than NFS, not an option. I came to the conclusion that NFS is currently the best option for Macs.

I've also messed around with the mount options to get rid of unnecessary stuff and this gives me the best performance with fewer options:

Code:

mount -t nfs -o rsize=65536,wsize=65536,intr,nolock,locallocks,rdirplus,readahead=128,noatime,nfc,actimeo=0 10.0.1.4:/mnt/Storage/Arquivos /Volumes/FreeNAS

I'd like to try NFSv4 but can't make it work, and I'd like to remove the "actimeo=0" option but that breaks everything...

The other issue I'm having is I can't transfer a folder from Macs to FreeNAS if they contain an "alias". Finder gives me error -50. Funny thing is it works if the alias is broken, missing file for example. But if the alias is working, it gives me an error, and Final Cut Pro libraries have a lot of them...

 

[EGP]

Hello!
I also struggled with the same performance issue with SMB. Try these commands on the client.

"defaults write com.apple.desktopservices DSDontWriteNetworkStores true"

and

"printf "[default]\nsigning_required=no\n" | sudo tee /etc/nsmb.conf >/dev/null"

credit here: https://dpron.com/os-x-10-11-5-slow-smb/

I run 10gb fiber to FreeNAS. NFS and SMB were pretty close in comparison but I use SMB for ease of use.

 

[nojohnny101]

I'm a bit confused about reading your thread. The fact that you are having widespread speed issues (with AFP, SMB, etc.) indicates some larger problem.

Your processor seems a bit underwhelming. Are you seeing these speed problems when just one person is accessing at a time? Have you monitored your CPU usage and hard drive usage during these transfers when it is slow? What kind of speeds are we talking about? I'm assuming you are using gigabit?

Some more info with hard numbers might help us help you.

 

[mjt5282]

I am in no way a Mac OS X or FreeNas expert, but generally when I create ZFS file systems for my Mac clients, I add "-O normalization=formD -O casesensitivity=insensitive -O atime=off" to the zpool create string. the Finder in Mac OS X expects formD normalization. Your example mount command is very much appreciated!

 

[EGP]

I am in no way a Mac OS X or FreeNas expert, but generally when I create ZFS file systems for my Mac clients, I add "-O normalization=formD -O casesensitivity=insensitive -O atime=off" to the zpool create string. the Finder in Mac OS X expects formD normalization. Your example mount command is very much appreciated!

I found your old thread regarding this. Interesting reading! Thanks!

 

[Juliano Arantes]

Hello!
I also struggled with the same performance issue with SMB. Try these commands on the client.

"defaults write com.apple.desktopservices DSDontWriteNetworkStores true"

and

"printf "[default]\nsigning_required=no\n" | sudo tee /etc/nsmb.conf >/dev/null"

credit here: https://dpron.com/os-x-10-11-5-slow-smb/

I run 10gb fiber to FreeNAS. NFS and SMB were pretty close in comparison but I use SMB for ease of use.

I'm sorry it took me so long to answer, but I guess better late than never. I feel like it's important because there's such a lack of Mac OS X info on this forum...

I had already tried that on SMB, didn't help! Just adding some more info, I can get decent SMB performance when transferring large files, let's say 5-10GB. But I'm a photographer so I usually deal with thousands of small files 30-60MB. That's the scenario that brings SMB to a crawl!!

 

[Juliano Arantes]

I'm a bit confused about reading your thread. The fact that you are having widespread speed issues (with AFP, SMB, etc.) indicates some larger problem.

Your processor seems a bit underwhelming. Are you seeing these speed problems when just one person is accessing at a time? Have you monitored your CPU usage and hard drive usage during these transfers when it is slow? What kind of speeds are we talking about? I'm assuming you are using gigabit?

Some more info with hard numbers might help us help you.

Hey there! I guess my notification e-mails were all going straight to the spam folder, didn't realize the thread had been answered... Sorry about that!

I've been monitoring CPU usage closely, it's usually really low (30%) and never saw it go higher than 60%. I've turned off compression on the entire system since my files are not compressible anyway... Maybe that's helping. I've done every test with a single user accessing at a time, but even with four users (the entire studio) the CPU is not a problem. Hard drive performance is also not a problem.

I do have a RAM issue however, and I'm the first to admit it. But right now I must deal with what I have, happy to inform that another 16GB of RAM are arriving before the end of the month! :)

I'm using gigabit and I can get 100MB/s on Windows clients using SMB, but that's just for testing purposes, we don't have any Windows clients here. SMB on the iMacs is a nightmare, we're talking 5-10MB/s when it doesn't crash. The solution was to use NFS, with that I can get 60-80MB/s on the iMacs, and it's rock solid.

The "larger problem" you indicated is OS X, it's awful SMB implementation and my use scenario... The system works fine on Windows, using SMB, NFS or whatever. But Apple has already deprecated AFP, and SMB is half baked, so I guess I'm stuck with NFS! Haven't tested it on OS X 10.3 though, and that's what we're using now, maybe there was some improvement... Will test as soon as I have some time.

About my use scenario, it's what I just told EGP on my last post. I deal with thousands of small files on the same folder. A single wedding can have 5.000 files on a single folder, each around 20-60MB. I do not have issues with SMB when transferring large files. I've tested with some full hd movies for example. The problem seems to be that OS X has a hard time negotiating every file on SMB, so even a large file starts slow and than gets up to full speed. But when you have thousands of small files, they never get to full speed. It crashes all the time, even browsing folders on the server.

If you have any ideas, I would be happy to test!

 

[Juliano Arantes]

I am in no way a Mac OS X or FreeNas expert, but generally when I create ZFS file systems for my Mac clients, I add "-O normalization=formD -O casesensitivity=insensitive -O atime=off" to the zpool create string. the Finder in Mac OS X expects formD normalization. Your example mount command is very much appreciated!

You're welcome! :)

Had never heard of that normalization string! Thanks a lot for the input, I'll do some research into that. I guess I can only choose that when creating the volume right?! Is there a way to change that on the fly without having to destroy the pool and creating a new one? Unfortunately I do not have enough hard disks to backup everything, destroy it and create it again...

Before anyone says I should have a backup, I do. But it's online and downloading everything again is not an option, would have to ask for a hard disk and it would be shipped from the US and would take ages to arrive in Brazil, etc.

 

[jssilva]

I have the exact same problem as you, same setup.

But, full of hope, I inserted your nfs mounting options in /etc/nfs.conf and tested.

But it still doesn´t work. When I click on the folder named "Zé", its contents are shown briefly, some times for less than a second, other times a few seconds, but then finder goes back and shows me the same "Zé" folder again.

If you are interested on a deeper description, I took the issue here:
https://discussions.apple.com/message/33825166
and someone, not apple, helped by saying I should take one the workarounds, either renaming or making symlinks to all the folders!!! Thousands of them!

As I dual-boot MacOS and Linux, almost always the later on a daily basis, I think I'll let MacOS go to sleep again for a few more years until they solve this problem.

From MacOS I only miss the much better power management, and consequently less heat on my lap and longer battery life.

Regards,
jss

 

[Juliano Arantes]

Not sure what else you could do, and by reading your other thread you seem to be a much more technical guy than I am. But's let's give it a try. About my mount options, as I wrote here before, it's "noac" or "actimeo=0" that solves my problem, without one of them I have the same problem you do... Have you tried both? "actimeo=0" works better in my opinion.

Another input, even using the "actimeo=0" the issue is not completely solved. When browsing into folders, the folder list will show up immediately without the subfolders that contain special characters and than quickly refresh to show them all. That happens within a second, so I got used to it "blinking".

Another detail is that Finder does not update it's horizontal stripping in list mode to reflect the changes, so it looks messed up, with double gray lines:

It feels like it couldn't load the subfolders with the special character on the first attempt, but managed to do it on a second attempt. Maybe as if it had some sort of cache that doesn't support the special characters... So it shows the cached subfolders and than refreshes to also show the ones with special characters.

The only real problem is that if I'm transferring a file to/from the server and I browse into one of this folders that contain special characters in subfolders, my transfer crashes when finder blinks to update the list... Do you know if there's any log I could check? A way to find the reason behind it?

Not sure if it helps, but I'm also running this commands on every client machine before it connects to the server for the first time, gives me a performance improvement:

Stop .DS_Store:

Code:

defaults write com.apple.desktopservices DSDontWriteNetworkStores true

Disable ARP Requests Validation and Change Delayed ACK to 0:

Code:

sudo nano /etc/sysctl.conf
net.link.ether.inet.arp_unicast_lim=0
net.inet.tcp.delayed_ack=0
sudo chown root:wheel /etc/sysctl.conf
sudo chmod 644 /etc/sysctl.conf

 

[jssilva]

Caro Júlio,

Thank you for replying and the obvious time you spent with my issue.

Frankly, I took the ant tactic, that is, if you can't push the rock out of the path, just go around it, don't waste your life trying to push it.

And so, that's what I did, I gave up importing NFS shares to my macOS, although I keep them for the other Linux machines on my network, and overlapped the ones I need to access from my macOS with SMB shares. It works perfectly, the issue is gone. Furthermore, it seems that Apple is dropping NFS.

But your issue seems slightly different from mine because when I click the troubled "Zé" folder, it opens briefly showing the contents, like one second sometimes more, then it closes showing the folder again. In your case I understand it stays open.

Anyway, I'm not spending more time with this, I know I can't use NFS shares with my macOS, end of story.

Cumprimentos,
jss