*READ* performance slow on ESXI from FreeNAS (NFSv3)

[skek]

Hi Everyone,

Summary: on ESXI host itself (not a vm) read performance over NFSv3 from Freenas, when reading a file completely from ARC, is very slow (13-17MB/s) over 1GbE link.

Hoping to pick the brains of some experts on this one. I'm relatively new to the forums, but not to FreeNAS. Basically, new build in testing, FreeNAS 11 RC3 on baremetal Xeon E1230v6, 32GB. NFS share from SSD mirror (samsung 850 EVO) with Intel S3520 SLOG, and mounted on both ESXI 6.5 and 5.5 (ie, tried this from 2 different servers).

Read performance of a file off this share is very slow at 13-17MB/s. The file is in ARC, there is no disk activity when reading the file and dumping it to /dev/null. Same test from a vm client on the same esxi boxes gives close to line speed at 111MB/s (ie, NFS mounting within the VM and dumping to /dev/null, so no ESXI NFS mounted datastore involved, but exactly the same network/NIC/path etc). ESXI 5.5 performance slightly worse. Nothing moves much past idle on either ESXI host, or the freenas server in terms of CPU/mem etc. Boxes are not doing much of anything else.

I could move to ISCSI (have not tested performance in comparison yet), but was hoping NFS would perform better than this, at least for reads. Has me a bit baffled. This is not a sync/async issue. I've not done definitive test writes. They are slow as expected with sync, but I suspect they are around 30-60MB/s (which was maybe double what they were with no SLOG) - so they may even be faster than reads. I'll test write performance if I can solve the read issues!

Why would reading a file over NFS be slow, especially when the file is cached in freenas memory? I've not done any tuning and the NFS server on freens is pretty much out of the box, esxi hosts just mounted with defaults and NFSv3. ESXI is accessing the NFS server over dedicated NIC (in failover with another) with only storage network traffic on it on its own VLAN. Just one switch connecting boxes, so low latency. this VLAN is on dedicated NIC on freenas also. Clearly the problem is with ESXI and how its NFS client implementation interacts with freenas.... just not sure if this is solvable. I've googled and found similar problems, although complaints seems to be centered around performance closer to the 60-70MB/s mark.

Any ideas or advice appreciated. Ideas for tuning or further testing to narrow down problem welcome, although if someone else has done the hard yards and advises that FreeNAS NFS server and ESXI NFS client don't go well together for some reason, I'll consider moving on to ISCSI.

Thanks for your time reading.

Regards
Dave

 

[joeschmuck]

I'm a bit confused so I need a little more clarification...

1) I think you said that you have one FreeNAS system with NFS shares on a bare metal system but you did not give complete hardware details, so provide these details.

2) I think you said that you have an ESXi 6.5 server (again no real details on the physical makeup) and you are trying to (I made this stuff up after this note becasue I'm making an assumption) copy data to a datastore on ESXi using an NFS share you somehow attached or browed for.

My suggestions, provide more details and describe how someone could recreate your steps of transferring the data. I'm making assumptions and that is a bad way to start out.

Also, have you done any troubleshooting like attaching the FreeNAS system directly to the ESXi server to ensure something in yoru network isnt' causing the issue? Also ensure you replace the Ethernet cable if you get the same results, it's just another troubleshooting step to rule an item out.

Have you tried SMB protocol?

Lastly, it does appear that NFS is slow in general, or should I say that there are a lot of complaints with ESXi and FreeNAS that is can be, but mine seems to be working just fine however my FreeNAS is a VM on my ESXi and I'm using the VMXNET 3 driver.

 

[gpsguy]

In addition to what @joeschmuck said, I would remove the 2nd "failover" NIC from the equation.

 

[skek]

ok. perhaps I should have been more rigorous in my testing. I inadvertently was using small block sizes. My mistake. Turns out NFS on esxi performs very poorly with small block sizes (no surprise). throughput went up to 67MB/s (crappy but as expected) by about 128k blocks, less than that and performance tanks. As I was testing in stages I was trying to test NFS performance of the mounted datastore before proceeding to test the performance of said datastore from within a vm. If I'd skipped ahead I would have found what I was looking for performance wise.

So in summary in case others stumble across this thread,

ESXI 6.5 host, mounting a datastore shared from freenas 11 RC3 with NFSv3 (tests done directly on esxi host):

NFS read from file in ARC with BS < 128K = bad, down to about 15MB/s
NFS read from file in ARC with BS >= 128K = 67MB/s

In a VM client on that host and testing on a mounted virtual vmdk disk sitting on that NFS datastore:

NFS read from file in ARC = 112-118MB/s (regardless of block size) (naturally this goes even faster once caching in the OS VM occurs :)

I am not completely sure why pulling data off an NFS share directly from the esxi cli is slower and block size dependent compared to inside the VM but can think of a range of explanations for this, and as its not the performance I care about (on a VM is) I'm not going to delve deeper.

Note all tests do not care about underlying ZFS architecture as all NFS transfers are from cached data. I was trying to unit test here to ensure NFS performed well, independent of disk testing.

Thanks Joe and gpsguy for your thoughts, not pointing out something obvious made me re-look at my testing and after some good sleep be more thorough. Time to move onto write testing and SLOG or not :)