Slow NFS read performance over 10 GbE

[pyakex]

Its actually not caching but network'ed IO that is slow, so yes anything remote is slower than the same operation locally.

Now why its so much slower in TNC vs FN is an interesting question - I'd recommend to open a ticket about that.
They can tell you why and if its to be expected or what could be done.
I am sure this is of interest for a wide variety of users...

Okay, I see. I have hard to understand why this is but I know far too little about the details to know.

Thought: Has the driver changed for the NIC?
--Edit--
Thought 2: Could there be additional tunables between FN and TN?

So I made some further tests on the TrueNAS vs FreeNAS performance:
1. I installed TrueNAS 12.0-U1.1 and imported the pools i created in FreeNAS-11.3-U5 and got similar performance as in FreeNAS (~570 MB/s)
2. I upgraded the pools, still get about the same (~570 MB/s)
3. I destroyed the pools, recreated them in TrueNAS, recreated the test files, and now it's back to being slower. (~320 MB/s)

I will look into creating a ticket for this.

 

[Newfoundland.Republic]

@pyakex - That really seems like there is something with the pool creation. I agree that a ticket seems like the best approach.

I took a quick look at the change logs, but nothing popped out on ZFS.

One more thing you could test: If you can, destroy and recreate the pool in FN 11.3 and see how it performs under 11.3.

 

[pyakex]

@pyakex - That really seems like there is something with the pool creation. I agree that a ticket seems like the best approach.

I took a quick look at the change logs, but nothing popped out on ZFS.

One more thing you could test: If you can, destroy and recreate the pool in FN 11.3 and see how it performs under 11.3.

That is what I did in this post: https://www.truenas.com/community/threads/slow-nfs-read-performance-over-10-gbe.90759/post-629087

If I do the comparison again, how could I compare the configurations? Is there anything in addition to zfs get all pool i should check?

 

[hpc_developer_007]

Hi @pyakex, any update on your situation?
I am facing a similar issue, with the following system:

24x 12TB Seagate enterprise SATA drives
LSI 9300-8i HBA
128GB RAM
ZFS running on Ubuntu 20.04

I have the disks configured as a pair of 12-disk RAIDz2 sets. Locally I can get 1.3GB/s continuous read for actual data files for my workload.
Over NFS, mounted from another Ubuntu 20.04 node over 10GbE, I get only 650MB/s read for files not in the server's ARC, vs 990MB/s read for files that have been read once on the server and are in ARC (but are not cached on the client).
I've also tried Samba and I get only 280MB/s read for uncached files vs 990MB/s for cached files.
iperf3 shows that the 10G link is running at full speed full duplex. I can also confirm that in our own HPC apps the nodes can communicate at full 10G speed.

Is there just something wrong with NFS that it can't saturate the 10G link even though the local filesystem is fast enough? Where could I find NFS tuning parameters that might help?

 

[pyakex]

Hi @pyakex, any update on your situation?
I am facing a similar issue, with the following system:

24x 12TB Seagate enterprise SATA drives
LSI 9300-8i HBA
128GB RAM
ZFS running on Ubuntu 20.04

I have the disks configured as a pair of 12-disk RAIDz2 sets. Locally I can get 1.3GB/s continuous read for actual data files for my workload.
Over NFS, mounted from another Ubuntu 20.04 node over 10GbE, I get only 650MB/s read for files not in the server's ARC, vs 990MB/s read for files that have been read once on the server and are in ARC (but are not cached on the client).
I've also tried Samba and I get only 280MB/s read for uncached files vs 990MB/s for cached files.
iperf3 shows that the 10G link is running at full speed full duplex. I can also confirm that in our own HPC apps the nodes can communicate at full 10G speed.

Is there just something wrong with NFS that it can't saturate the 10G link even though the local filesystem is fast enough? Where could I find NFS tuning parameters that might help?

Hello,
No, I haven't made any progress on this. Please let me know if you found anything that improved your situation.

 

[Elliot Dierksen]

I can tell you from firsthand experience that you are likely being bound by your pool construction. I was able to get 16Gb read speed across my 40Gb storage network from a 2 x 8 disk RAIDZ2 array, but only because I have 256GB of RAM so most of it gets buffered. I could only sustain about 3Gb writes even with an Optane SLOG. I just restructured my main pool as 8 x 2 disk mirror, and I get close to 4Gb write now consistently. The "as short as I can make it" version of that is if iperf tests between hosts get 95% or more of the link speed as throughput, it isn't the network. It is your pool construction.