Slow Transfer on SSD Pool

[user298978324792374]

Hi,

I recently build a new NAS system with Truenas on it. It has quite good specs: dual 10 core intel xeon 2690v3, 128gb RAM, and a Mellanox Connect-x4 dual 100gbit/s card. I have 2 pools in the system. one consists of 16 2tb WD RED SA500 SSDs and it is a mirrored pool 8 x MIRROR | 16 wide. I have another Server (in the following called client) with the same Mellanox card and they are both connected to a Mikrotik switch via a 40gb link. I have tested the Fiber link between the servers with iperf and it is (almost)40gbit. see picture below

I have run this iperf test a few times and always get the same result within a little variation.

The SSDs should each be capable of ca. 500mb/s reads and writes. So in a pool of 16 drives in a mirrors it should give me in theory 8x500mb/s = 4Gb/s of Read/ Write speed. This is just theoretical so I am not expecting that kind of speed.
When I then Try to create a NFS share on the Truenas and mount it on the second Server (which I will Client with the 40bg link) and copy some isos between them I only get around 300mb/s of transfer speed from the Truenas to the Client.

When doing it the other way around (Client to Server) it is a little faster and I get around 500-600mb/s

The Storage on the client is also an nvme SSD (Samsung pm9a3 u.2) which should easily handle at least 3gb/s reads and 1,5gb/s writes (benchmarked on linux)

Any idea on this? can I still tweak any settings for the Network cards or truenas?
Any input is appreciated.

Thanks

 

[Davvo]

Units terminology please.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

The 530-560 are sequential speeds; look at the fio command and jgreco's solnet array to test your drivers and pool performance, but it's normal for mirrors to have write speeds lower than read's.

Please read the following resource:

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

I would say it's not a networking issue.