10G NFS performance from ARC cache

[sandaco]

Hi FreeNAS gurus,

I'm testing freenas on following machine :
Dual Xeon CPU X5687 @ 3.60GHz
128 GB Ram.
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection dual port + twinax cable.
Intel P3700 1.6TB NVME SSD as the storage device ( single device ZFS volume)

My test consists of reading a 40GB file from another freenas installation over NFSv3 using dd, and sending it to /dev/null. When I repeat the test and monitor the disk activity on the server, I see zero disk activity meaning the file is being served directly from the ARC cache, but I don't seem to be able to exceed 450 MB/s on average . it has few moments when it reaches the 850 MB/s but quickly falls back to 400-450 MB/s .

I can not use jumbo frame in our setup and currently using 1500 byte MTU, iperf reports 9.7 Gbps bandwidth between the server and client machine.

Please advise.
-Ali

 

[JustinClift]

Does this help?

https://forums.freenas.org/index.php?threads/10gb-tunables-on-9-10.42548/

 

[Donny Davis]

What does the read op look like on the local machine? More than 400M?

 

[sandaco]

Does this help?

https://forums.freenas.org/index.php?threads/10gb-tunables-on-9-10.42548/

I'm reading through and I've implemented most of the suggestions without any noticeable effect.

 

[sandaco]

What does the read op look like on the local machine? More than 400M?

The volume is mostly idle, few occasional reads and writes in the 10-20 KB range, while I'm reading the file over NFS at said speeds.

This is arcstat results :

~# arcstat.py 1
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
20:00:10 0 0 0 0 0 0 0 0 0 59G 115G
20:00:11 640 31 4 31 5 0 0 30 4 59G 115G
20:00:13 234 1 0 1 0 0 0 1 0 59G 115G
20:00:14 153 2 1 2 2 0 0 2 1 59G 115G
20:00:16 174 2 1 2 2 0 0 1 0 59G 115G
20:00:17 718 17 2 17 2 0 0 16 2 59G 115G
20:00:19 183 2 1 2 1 0 0 2 1 59G 115G
20:00:20 328 1 0 1 0 0 0 1 0 59G 115G <- I started reading the file
20:00:22 1.9K 39 2 39 3 0 0 39 4 59G 115G
20:00:23 10K 2 0 2 0 0 0 1 0 59G 115G
20:00:25 11K 3 0 3 0 0 0 2 0 59G 115G
20:00:26 3.5K 1 0 1 0 0 0 1 0 59G 115G
20:00:28 8.8K 22 0 22 0 0 0 22 1 59G 115G
20:00:29 10K 1 0 1 0 0 0 1 0 59G 115G
20:00:30 10K 2 0 2 0 0 0 2 0 59G 115G
20:00:32 10K 23 0 23 0 0 0 23 1 59G 115G
20:00:33 10K 1 0 1 0 0 0 1 0 59G 115G

 

[SweetAndLow]

I don't think I have seen an example of the Intel NICs going any faster than what you have seen. Anyone seen or gotten theirs to move data faster?

Sent from my Nexus 5X using Tapatalk

 

[sandaco]

Hi FreeNAS gurus,

I'm testing freenas on following machine :
Dual Xeon CPU X5687 @ 3.60GHz
128 GB Ram.
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection dual port + twinax cable.
Intel P3700 1.6TB NVME SSD as the storage device ( single device ZFS volume)

My test consists of reading a 40GB file from another freenas installation over NFSv3 using dd, and sending it to /dev/null. When I repeat the test and monitor the disk activity on the server, I see zero disk activity meaning the file is being served directly from the ARC cache, but I don't seem to be able to exceed 450 MB/s on average . it has few moments when it reaches the 850 MB/s but quickly falls back to 400-450 MB/s .

I can not use jumbo frame in our setup and currently using 1500 byte MTU, iperf reports 9.7 Gbps bandwidth between the server and client machine.

Please advise.
-Ali

Wanted to add that server is 90% idle , spends 1.2% cpu on interrupts according to top as the test is running.
client is 95% idle, spends 1.5-1.9 % on interrupts during the test run.

 

[Donny Davis]

What I was asking is run the same read op directly on the freenas machine

 

[sandaco]

What I was asking is run the same read op directly on the freenas machine

using the same dd command on the machine that the file resides, I get something between 5.8-6.8 GB/s , 40G file is read in less than 8 seconds.

 

[Donny Davis]

How about the client system. Can you run the OP locally.

Are you bound to using NFS? Have you tried via SSH or zfs send?

It's best to isolate everything and try to narrow down the list of possibilities.

 

[sandaco]

How about the client system. Can you run the OP locally.
Are you bound to using NFS? Have you tried via SSH or zfs send?
It's best to isolate everything and try to narrow down the list of possibilities.

I'm running these tests to make sure I can build a robust NFS server for our environment so NFS performance is key.
iperf tests have established that the network stack is capable of passing data at line-rate. I'll test the SSH and zfs send as well.

 

[Donny Davis]

If using something that isn't NFS yields the performance you are looking for then you know it's an NFS issue. From there you can t/s NFS

 

[JustinClift]

Hmmm, there seem to be quite a few threads on this forum about NFS speeds, with various solutions being found. It's probably worth doing a search for suitable keywords and trying things out. :)