zpool iostat bandwidth much higher than network throughput

[BrentI]

Hello

I have FreeNAS 9.2.1.5 running in a ESXi5.1 VM with 12 Drives.
When there is NFS throughput going through what zpool iostat outputs in terms of read bandwidth is around 100MBps, yet the network throughput is only around 10MB/s. which seems odd to me.
This is what zpool iostat shows

capacity operations bandwidth
pool alloc free read write read write
pool1 15.1T 6.32T 899 0 111M 0
pool1 15.1T 6.32T 957 74 118M 2.43M
pool1 15.1T 6.32T 875 73 108M 1.78M
pool1 15.1T 6.32T 774 109 95.6M 4.34M
pool1 15.1T 6.32T 720 73 88.8M 2.37M
pool1 15.1T 6.32T 866 69 107M 2.31M
pool1 15.1T 6.32T 762 119 94.3M 5.64M

You will see in the attached graph from the interface traffic that the TX on the NIC is only around 70Mbps (9MBps).
This seems to be a wild difference and I have checked and prefetching is disabled.

The output from arcsummary is as follows:
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
05:21:52 0 0 0 0 0 0 0 0 0 11G 11G
05:21:53 1.7K 769 44 42 4 727 100 10 1 11G 11G
05:21:54 2.2K 1.2K 51 29 2 1.1K 100 12 1 11G 11G
05:21:55 2.2K 1.0K 46 16 1 1.0K 100 9 1 11G 11G
05:21:56 1.3K 645 47 29 3 616 99 5 1 11G 11G

And this is my zpool config
pool1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/2ab18178-7c00-11e2-95eb-000c2976f274 ONLINE 0 0 0
gptid/25cf095d-8620-11e2-a143-000c2976f274 ONLINE 0 0 0
gptid/2badff45-7c00-11e2-95eb-000c2976f274 ONLINE 0 0 0
gptid/2c2f92ad-7c00-11e2-95eb-000c2976f274 ONLINE 0 0 0
gptid/2c771923-7c00-11e2-95eb-000c2976f274 ONLINE 0 0 0
gptid/2cbf1e95-7c00-11e2-95eb-000c2976f274 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/5cd38309-89e7-11e3-a2b6-000c2976f274 ONLINE 0 0 0
gptid/79e65895-a14e-11e2-a143-000c2976f274 ONLINE 0 0 0
gptid/00d79ab4-7c01-11e2-95eb-000c2976f274 ONLINE 0 0 0
gptid/01505fb7-7c01-11e2-95eb-000c2976f274 ONLINE 0 0 0
gptid/01c84423-7c01-11e2-95eb-000c2976f274 ONLINE 0 0 0
gptid/0240dbec-7c01-11e2-95eb-000c2976f274 ONLINE 0 0 0

Is what I am seeing normal? I have checked VMware and there is around 100MBps hitting the disks which backs up the zpool iostat output.
It appears as though the system is reading a lot more data than it is sending to the NFS client.

Does anyone have any ideas?

 

[L]

ZFS is odd in the way it measures. Because of the transactions are processed, some iops may not appear realtime. But I would trust the iostat and the vm that you are consistantly doing 100.

 

[BrentI]

Thanks for the response.
It sounds like you are saying to trust the iostat and VM, however I know for sure that there isn't 100MBps being sent to the NFS client.
Therefore something must be amplifying the reads as I am sure this is not normal. Unless it is something to do with the size of the blocks that are read from the disk. i.e having to read a 1MB stripe to be able to retrieve 128KB of data from it.

 

[Norleif]

You are running FreeNAS inside a VM...?

 

[BrentI]

Yes that is correct. However presented 12 individual disks via RDM so ZFS can manage the disks. The performance is great in general. Just that in the above example it seems to be reading a huge amount from disk.
In further testing I think it may be due to the 128k recordsize and the fact that the application that is reading from it via NFS may be doing much smaller requests.
In that case am I correct in saying that ZFS will read in a full 128k even if it was only requested to send 1k?
I have decreased the recordsize to 32k and that appears to have made a huge improvement. I am now contemplating whether we are better off to give it an extra 10GB RAM, or add a SSD for L2ARC.
The dataset is about 10TB so I assume there is only so much that RAM or an SSD can do.

 

[depasseg]

You might want to investigate the use of RDM. There are some pretty drastic failure scenarios. The recommended approach is to passthrough the HBA into the VM.

 

[mav@]

In that case am I correct in saying that ZFS will read in a full 128k even if it was only requested to send 1k?

Yes, you are correct here. ZFS never reads/writes less then a configured recordsize, unless whole file is smaller then that. If your application does many small random I/Os, then you may wish to reduce the recordsize to reduce read/write amplification. But too small recordsize will penalize large sequential operations and compression efficiency. It's a hard choice.

 

[cyberjock]

Yes that is correct. However presented 12 individual disks via RDM so ZFS can manage the disks.

Queue the thread "zomg my pool just disappeared in 5...4...3...2..."