Any tips for XCP-ng ---> TrueNAS 12 NFS VM data store

[Brezlord]

Hi all I'm looking for any tips for improving performance of NFS as VM backing for XCP-ng 8.1.0. I feel that my setup should be much faster. I was previously running vmware over fibre channel which was lightning fast. Below are the specs of the hardware I'm using.

Storage
Version:
TrueNAS-12.0-RC1
Hardware:
Dell PowerEdge T320
Intel Xeon E5-2470 v2 10 cores @ 2.40GHz
96GB DDR3 PC3L 1333MHz (6 x 16GB DIMMs)
1 x Intel X520-DA2 10GbE (Various vLANs for Management and network shares)
1 x Dell PERC H310 flashed to IT mode (LSI SAS9211-8i HBA)
1 x Dell H200 HBA flashed to IT
2 x EMC KTN-STL3 DAE 15 x 3.5" enclosure
Storage Pool 3: VM storage
12 x HGST HUSML4040ASS600 400GB SAS SSD (6 x 2 way mirror)
log 1 x Intel Optane 900p 280GB SLOG

XCP-ng Server
XCP-ng 8.1.0
Dell PowerEdge R720
2 x Intel Xeon CPU E5-2660 v2 10 cores @ 2.20GHz
128GB DDR3 PC3 1866MHz (8 x 16GB DIMMs)
1 x Intel X520-DA2 10GbE (Various vLANs for VM Traffic, Management and IP Storage)

Switch
Ubiquity EdgeSwitch 16 XG

I know the following tests are not the best but they do give indication of performance.
I ran the following on a test data set on the pool that holds the VMs. time sh -c "dd if=/dev/zero of=testfile bs=1900K count=1K && sync". I get 1327.733912 MB/s, which is good.

Running the same command in a CentOS 7 VM with guest tools installed i get;
815 MB/s for 2G
695 MB/s for 4G

Using iperf I get the following.
VM --> TrueNAS 870 MBytes 7.29 Gbits/sec
TrueNAS --> VM 569 MBytes 4.77 Gbits/sec

Windows VM

Does anyone have any ideas on tuning TrueNAS network stack. I have the 2 10G ports in LACP but NFS can only use 1 connection so I doubt network topology will be the issue. When will pNFS be coming to TrueNAS Core?

 

[Brezlord]

Would changing the TrueNAS nic from Intel X520 to Chelsio T520-CR or T540-CR improve performance or is that wish full thinking?

 

[Brezlord]

I have changed the NIC to a Chelsio T520-CR and with no tunnig I'm getting much better performance, what I'd expect actually. Not sure why the Intel NIC would not preform as well.

Same VM used as above
iperf in both directions 8.40 Gbits/sec

 

[Brezlord]

With some tuning I get 22000 R/W IOPS in a windows 10 VM.

 

[Brezlord]

iperf between TrueNAS 12 and bare metal pfsense.

Code:

Connecting to host 192.168.10.40, port 5201
[  5] local 192.168.10.254 port 21115 connected to 192.168.10.40 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1005 MBytes  8.43 Gbits/sec    0   2.54 MBytes       
[  5]   1.00-2.00   sec  1.09 GBytes  9.35 Gbits/sec    0   2.54 MBytes       
[  5]   2.00-3.00   sec  1.09 GBytes  9.33 Gbits/sec    0   2.54 MBytes       
[  5]   3.00-4.00   sec  1.09 GBytes  9.35 Gbits/sec    0   2.55 MBytes       
[  5]   4.00-5.00   sec  1.09 GBytes  9.33 Gbits/sec    0   2.55 MBytes       
[  5]   5.00-6.00   sec  1.09 GBytes  9.34 Gbits/sec    0   2.55 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.35 Gbits/sec    0   2.55 MBytes       
[  5]   7.00-8.00   sec  1.09 GBytes  9.33 Gbits/sec    0   2.55 MBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.35 Gbits/sec    0   2.55 MBytes       
[  5]   9.00-10.00  sec  1.09 GBytes  9.34 Gbits/sec    0   2.55 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.25 Gbits/sec    0             sender
[  5]   0.00-10.31  sec  10.8 GBytes  8.97 Gbits/sec                  receiver

Bellow are the sysctl tuning parameters I'm using.

Code:

kern.ipc.maxsockbuf=16777216
net.inet.tcp.cc.abe=1
net.inet.tcp.initcwnd_segments=44
net.inet.tcp.minmss=536
net.inet.tcp.mssdflt=1460
net.inet.tcp.recvbuf_inc=65536
net.inet.tcp.recvbuf_max=4194304
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.sendspace=65536

 

[kspare]

Try going back to 11.3u5.

you may see better speeds.

 

[Brezlord]

I believe you are correct, but I am happy with 12 and am happy to help test and raise bugs if I find any.

 

[kspare]

I believe you are correct, but I am happy with 12 and am happy to help test and raise bugs if I find any.

Fair. I want to use 12 as well, but with iscsi not working and this mysterious network issue...can't keep running it in production.

 

[Brezlord]

I'm only using 12 in my home lab so no real issues. I had iSCSI working, what's not working for you?