Slow NFS speeds to OS X

[neils]

Server:
45-drive storage pod
4.0 upgrade to 2 x Highpoint Rocket R750 PCI SATA cards
OS: FreeNAS 9.3 STABLE
NIC: Gigabit

Client:
iMac, Mac Pro, Mac Mini
OS: 10.9 and 10.10
NIC: Gigabit

Client:
various SuperMicro and Dell servers
OS: CentOS 6.x
NIC: Gigabit

server zfs raidz6 volume NFS exported to Mac and linux clients.

Mac 'rsync -av' from NFS mount of FreeNAS volume to local disk:
~ 5MBs
CentOS 'rsync -av' from NFS mount of FreeNAS volume to local disk:
~64 MB/s

Mac 'rsync -av' <user>@<server>:/filepath <local disk path> (ie., with ssh encryption overhead)
~54 MB/s
CentOS (ditto)
~63 MB/s

I am certain we weren't getting this NFS difference between the OS's at FreeNAS 9.1.

Any suggestions?

 

[neils]

Also, I can say we are getting these xfer rate differences with other zvols on this FreeNAS server.

 

[neils]

And I have tested and got the same results on the Macs with manual mounts:
sudo mount -o rw,nfsvers=3 <server>:/mnt/<zvol> /mnt
sudo mount -o rw,nfsvers=4 <server>:/mnt/<zvol> /mnt

 

[dlavigne]

Were you able to figure this out? Also, rsync isn't a fair test as it's not built for speed...

 

[neils]

No. Still broken.

And rsync is just a sample network exercise.
The critical observation is the drastic difference between OS X NFS clients and Linux clients.
And no matter how much slower rsync is from the next option, under no circumstances would I assume 4 MB/s is acceptable over gigabit networking.

I sure would like to know what's going on here.
Would anyone have suggestions on debugging methods?

 

[maglin]

Are you running a 45 drive RAIDZ6 i.e. all your drives are in a single vdev?

Can you try the same tests using a CIFS share? Is FreeNAS behind a hypervisor like ESXi and if so how much CPU and RAM does it get? NFS is CPU dependent as is rsync. Together if under provisioned could be the cause.

Also how did you get the Rocker RAID card to work as a HBA? I thought I saw several attempts at people using them with no luck.


Sent from my iPhone using Tapatalk

 

[neils]

Are you running a 45 drive RAIDZ6 i.e. all your drives are in a single vdev?

Can you try the same tests using a CIFS share? Is FreeNAS behind a hypervisor like ESXi and if so how much CPU and RAM does it get? NFS is CPU dependent as is rsync. Together if under provisioned could be the cause.

Also how did you get the Rocker RAID card to work as a HBA? I thought I saw several attempts at people using them with no luck.


Sent from my iPhone using Tapatalk

Thanks for the note --

The volume in question is a RAIDZ6 of 13 x 3 TB drives, with FreeNAS installed on 'bare metal'.

I don't recall doing anything special to get FreeNAS to recognize the (2) Rocket R750's and then importing the existing zpool. What had you heard?

 

[Nick2253]

Just FYI, there's no such thing as RAIDZ6. I'm assuming you mean RAIDZ2.

I'm curious: what makes you think this is a FreeNAS problem? Based on what you've provided, I would assume that it's an OSX problem. From the following tests, it looks like OSX is pulling data from a direct connected NFS share just fine, but is only having a problem when it pulls data from a mounted share. My immediate guess is that OSX's rsync is treating a mounted NFS share differently than directly connecting.

Mac 'rsync -av' from NFS mount of FreeNAS volume to local disk:
~ 5MBs
CentOS 'rsync -av' from NFS mount of FreeNAS volume to local disk:
~64 MB/s

Mac 'rsync -av' <user>@<server>:/filepath <local disk path> (ie., with ssh encryption overhead)
~54 MB/s
CentOS (ditto)
~63 MB/s

rsync is a poor comparison test because Apple's rsync is not the same as what you'd find on CentOS (or *nix in general). From a brief search around the web, I have found a few stories of rsync problems on OSX, that were all fixed when installing the ported rsync from MacPorts (https://www.macports.org/).

What happens when you just cp a file from locally-->NFS mount or vice versa? Do you get similar speed problems?

 

[neils]

Just FYI, there's no such thing as RAIDZ6. I'm assuming you mean RAIDZ2.

I'm curious: what makes you think this is a FreeNAS problem? Based on what you've provided, I would assume that it's an OSX problem. From the following tests, it looks like OSX is pulling data from a direct connected NFS share just fine, but is only having a problem when it pulls data from a mounted share. My immediate guess is that OSX's rsync is treating a mounted NFS share differently than directly connecting.



rsync is a poor comparison test because Apple's rsync is not the same as what you'd find on CentOS (or *nix in general). From a brief search around the web, I have found a few stories of rsync problems on OSX, that were all fixed when installing the ported rsync from MacPorts (https://www.macports.org/).

What happens when you just cp a file from locally-->NFS mount or vice versa? Do you get similar speed problems?

RAID6 ... RAIDZ2 ... when your managing multiple storage monsters of varying setups, and factoring in that your old as dirt, things tend to run together at times.

Thanks Nick2253,

cp gets roughly the same speed as rsync - maybe a M/s faster. This for both OS X and linux NFS clients; which still differ by a factor of 20.

As I mentioned, this is a backblaze 45 drive storage box, the first version, that was running FreeNAS 8.x and serving the same zpools (or mostly the same) without this drastic reduction in NFS service to OS X. I did the backblaze v4 backplane/SATA card swap out. Ran it for a while at that FreeNAS OS vers, again not seeing xfer difference, before upgrading to 9.1.2, then to 9.3. I tried a 9.10 upgrade but there seemed to be a hardware conflict. I'll have to retry it to refresh my memory.

Sure could use some suggestions on what logs or diagnostics might be helpful.