NFS writes blasting fast, NFS read extremely slow

[chuber]

Hello everybody,

I have a strange probleme here running a FreeNAS 9.2.1.2 box as NFS datastore for ESXi 5.5. Knowing that there are issues regarding write performance with ESXi and NFS, I have setup a RAIDZ2 with an SLOG device for ZIL. Now I am seeing really good write speed at the datastore but read access is extremly slow. iostat shows only minor activity when reading data, when writing the IO activity of the SLOG Device and the RAIDZ2 jumps to 30 to 60 Megabytes per second per device which is good taking into account that FreeNAS is running on an old hardware.
Event accessing the dataset via CIFS show high read rates (I am able to saturate the 1Gb NIC of the box) and writes are fast to.

Only NFS reads are extremly slow (with low disk activity and low CPU load, so I do not see any bottle neck here).

Has anybody an idea how to get close to cause of this behavior ?

Thank you very much and best greetings,
Chris

 

[eraser]

How much RAM is installed in your FreeNAS server? What brand of NIC do you have installed on your FreeNAS server and on the client?

 

[aufalien]

Well usually its the other way around being that NFSv3 is stateless is CIFS is statefull. At any rate, what are your client mount options?

 

[chuber]

Hello eraser, hi aufalien,
thanks for the response.

the system has 8GB RAM, NICs are broadcom based (BCM5708 ) (the whole system is a Dell PE2950 Xeon E5310), the client is using a BCM5719 .

I have one some additional testing and have to correct my observation. With CIFS reads are slow to, while writes are fast. I would have expect it to be the other way arount (write issue and high read speeds). SoI more and more think tat it must be something NIC/network related, but I have already replaced the NICs at the NAS and the ESXi host and used different cables and switch ports.

I am using ESXi as client, so there are no mount options to configure, only some NFS general params.

Thank you for your help.

 

[eraser]

Have you tested your network performance yet? You can use iperf to do this. During an iperf run, it is the client that pushes data to the server (seemed backwards to me when I first realized that). iperf doesn't exist for ESXi, but you can run the iperf server from inside a VM for a quick sanity check. Try using a TCP Window size of 64 KB (iperf's -w option).

Have you tested disk read performance directly from the FreeNAS server itself? To rule out ARC caching effects, you should either disable the ARC (requires a reboot) or use a test file that is larger then your ARC ( 7/8 of installed RAM by default). Hmm... I suppose you could also write a new file and then only read it one time (since the first time you read a file it shouldn't be in the ARC yet).

How many disks do you have in your ZFS RAIDZ2 Pool? I know that on my home ESXi lab using a NFS datastore the first boot of a VM seems much slower then I would expect. I chalk that up to the limited disk IOPS available in my FreeNAS system.

=-=-=

Apologies if you already know this....

With ZFS (almost) all writes end up as sequential writes to the underlying disks, and disks are fast at sequential IO. Every 5 seconds or so, a new txg (transaction group) is created and the old txg is (eventually) written to disk as one big chuck of sequential writes. (As you have discovered, sync writes mess this up unless you install a SLOG or set your ZFS dataset to "sync=disabled".)

ZFS reads can be either random or sequential, but often end up being random. Even if a client thinks it is making a sequential request for data, once it hits the ZFS pool the data could be scattered all over the place depending how it was first written and you end up with random read performance. You just have to wait while your disks seek to the correct place. The total time to satisfy the request depends on how many physical disks you have installed on your system and the ZFS Pool configuration (mirrored, RAIDZ1/RAIDZ2/ RAIDZ3 and number of disks & vdev). From my research, I believe that most companies combat this latency by having dozens of fast physical disks installed and configured in multiple vdevs.

ZFS also tries to help read speeds by using system RAM as a cache (also known as the ARC). However, the first time you read data you still have to wait for the physical disks. After that the data is stored in the ARC and future requests of the same data is much faster. FreeNAS disables the ARC on systems with less than 4 GB of RAM, which is why I asked how much RAM you have installed.