Slow NFS reads (maybe directory reads)

[Michael Eager]

I have a iXsystems FreeNAS-mini sever with a point-to-point 10Ge (MTU=9000) connection to a workstation. The FreeNAS has 16Gb RAM, 4x4Tb disks, RAIDZ. When I run iperf, the transfer rate from the server to the workstation is an outstanding 942Mb/s.

I have a directory mounted from the FreeNAS box to the workstation using NFS. Mount lists the following: (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.xx.xx,local_lock=none,addr=192.168.xx.xx)

After clearing the disk cache on the workstation, copying (cp -r) a 2.4Gb directory with 5822 files from the server to a local drive takes 51:18 minutes. Copying a tar file of the same directory takes 1:05 min.

It appears that the transfer rate declines as the transfer progresses, dropping to 500KB/s or much less toward the end of the copy. While this is going on, the FreeNAS seems to be lightly loaded. CPU is around 10%, load is 1.5, and the network traffic is minimal. (There is a network spike to 700Mb/s when the transfer starts, but that rapidly declines.)

Anyone have any idea what is causing this problem? Or how to fix it?

 

[JohnDigital]

A transfer of that size on that network hardware in my estimation should be well under 10 seconds. However, Solandri states regarding read/write on modern hard discs:

"Sequential is the speed at reading/writing big files. So if you were copying a bunch of 1GB movie files (which weren't fragmented), the sequential speed is what you'd expect. For a modern 7200 RPM drive it ranges from about 80-160 MB/s today.

4k is the speed at reading/writing really small files (4 kB in size). On many larger drives this is the smallest cluster size, so it's the lower speed limit. There's OS and drive overhead in locating the file, then the time associated with moving the read/write head to the proper location and waiting for the right section of the platter to spin under the head. For a modern drive it's about 0.7-1.3 MB/s."

Of course if we are talking SSDs its a different story. But this sounds to me like exactly what you are experiencing. Lots of tiny files. You should see rates about 160MB/s for a large 2.4gb file. But tons slower for small files. I have noticed this frequently when I copy my music around. Pop an SSD or two in that workstation and you should see rates increase to whatever the FreeNAS box can read at, theoretically. If it already has SATA3 SSDs in it, then more inspection is needed.

 

[Michael Eager]

The long delay is not caused by HD access times. Copy times disk-to-disk on the FreeNAS system is 6sec for the tar file (that's probably cache read, not disk read); 1:05 for the directory.

It appears that the problem is caused by allocation failures for jumbo frames. Setting MTU=1500 on workstation and FreeNAS improved directory copy to 2:20. (Thanks to iXsystems Tech Support!)