Need help with R/W performance

[Phase]

Hi,

I'm looking at some low figures and I do not know what to make of them.

iperf gives me just under 15 GbE between two TrueNAS 12.0-U6 servers, but file transfers are 15 to 20 times slower.

Code:

iperf3 -i 10 -w 1M -t 60 -c 192.168.4.7
Connecting to host 192.168.4.7, port 5201
[  5] local 192.168.4.5 port 62313 connected to 192.168.4.7 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  16.7 GBytes  14.4 Gbits/sec    0   1.77 MBytes
[  5]  10.00-20.00  sec  16.2 GBytes  13.9 Gbits/sec    0   1.77 MBytes
[  5]  20.00-30.00  sec  17.3 GBytes  14.8 Gbits/sec    0   1.77 MBytes
[  5]  30.00-40.00  sec  17.1 GBytes  14.7 Gbits/sec    0   1.77 MBytes
[  5]  40.00-50.00  sec  16.3 GBytes  14.0 Gbits/sec    0   1.77 MBytes
[  5]  50.00-60.00  sec  16.5 GBytes  14.2 Gbits/sec    0   1.77 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec   100 GBytes  14.3 Gbits/sec    0             sender
[  5]   0.00-60.22  sec   100 GBytes  14.3 Gbits/sec                  receiver

In the chart below you see 5 bumps. They represent:

1. iperf reporting just under 15 GbE between the two TrueNAS servers
2. (2 SSDs mirrored) to (2 SSDs mirrored) -> 0.76 to 0.59 Gb
3. (2 SSDs mirrored) to (5 14TB HDD in Z2) -> 1.10 to 0.98 Gb
4. Repeat of #2 above -> 0.72 to 0.61 Gb
5. (9 16TB HDD in Z3 ) to (5 14TB HDD in Z2) -> 0.77 to 0.62 Gb

Notes:

• All datasets in the above have no compression
• All transfers were done over NFS via a cp command of a single 35GB file
• cc0 is a dedicated 40 GbE interface and was only used for communication between the two servers (card to card)
• em0 is the regular 1 GbE interface
• The HDDs use LSI SAS9211-8i card(s)
• The SSDs use motherboard SATA connectors
• Both servers have 32GB of memory, one has an i7-11700K and the other has an i7-3770K
• 14 GB drives are MG07ACA14TE (248 MiB/s)
• 16 GB drives are MG08ACA16TE (262 MiB/s)

The graphs on the other server mimic the ones below.


Any idea why the performance is this low?

Thanks!!

EDIT: Adding storage configuration for both servers. OVERWATCH and OVERVOLT are simple SSD mirrors. OVERWATCH1 and OVERVOLT1 are HDD in 9 x Z3 and 5 x Z2 respectively. Test #2 above is the transfer between the two SSD mirrors.

 

[sretalla]

If your "working set" is 35GB and you have 32GB of memory, you're not going to be getting any benefit at all from ARC.

You might want to look at your MTU size.

For large single-threaded file transfers, you would usually benefit from MTU of 9000 (or just over).

 

[Phase]

I tried to reply earlier and the software would not let me. Apologies for that.

I did not realize that the network settings could affect the transfer rate given that the network is currently 15-20 times faster than the file transfer. I'm currently working with Chelsio to increase the performance as these are 40 GbE transceivers and iperf is only showing 15 GbE. I'll certain look into the MTU once we get the network a bit more dialed in and report back.

Still strange that file transfers are 20x slower than the network. Any other thoughts? Thanks!

 

[Samuel Tai]

Slow file transfer speed

So i have a bit of an issue. I have a 10Gbs network on this network i have two computers one is a windows computer and one is a Truenas computer. This issue is when ever i upload to the TrueNas computer i get on at top end 300MB's< i have tested the cables the NIC switch all of it. so i tried a...

www.truenas.com

 

[Phase]

Thanks for the reply/link. Unfortunately, or fortunately, there is no windows or smb involved in the above. These are TrueNAS to TrueNAS transfers over nfs, one of which (test #2) is an SSD mirror in server 1 to an SSD mirror in server 2. Thoughts?

EDIT: added the 2 pics of the storage screens to the 1st post.

 

[Samuel Tai]

NFS is a slow protocol with significant overhead. Have you tried rsync instead?

 

[Phase]

No Joy :(

Similar results with RSync. CC0 and CC1 and 40GbE interfaces. See below

Server 1

Server 2

 

[Samuel Tai]

First, let's rule out a problem on-box. What's the performance if you replicate pool to pool on the same box, without using the network?

 

[Ralms]

No Joy :(

Similar results with RSync. CC0 and CC1 and 40GbE interfaces. See below

...

You seem to be focusing on the network aspect of things, have you checked your disks usage of the respective Pool during the file transfer?
Such as Disk Busy, Disk I/O or Pending I/O?

I feel the first step will mainly be if there is any obvious bottleneck somewhere.

 

[Phase]

Cool, I run a few more tests. All are TrueNAS replications. The tests were

1. Server 1 SSD mirror to Server 1 SSD mirror (same server, same disks)
2. Server 1 SSD mirror to Server 1 HDD 9xZ3 (same server)
3. Server 1 SSD mirror to Server 2 SSD mirror
4. Server 2 SSD mirror to Server 2 SSD mirror (same server, same disks)
5. Server 2 SSD mirror to Server 2 HDD 5xZ2 (same server)

Disks on Server 1

Disks on Server 2

 

[Phase]

And Network