SOLVED Slow transfer speed on the 10g network -> Poor Encryption performance in SCALE only

[Antoshka]

Hello folks,

I recently switched from TrueNAS Core to Scale and found an issue with the transfer speed. I have a 10g network between NAS and two desktops. Synthetic speed tests look ok. However, the real file transfer (SMB) gives 150 Mb/s only. If I start 2nd file transfer, the speed goes down to 75 Mb/s each.

iperf3 tests (both directions results are almost the same):
1 parallel gives 4.6 - 6.1 Gbits/sec
5 parallels give 9.3 - 9.4 Gbits/sec

OpenSpeedTest shows similar results as iperf3.

Disks are attached as network drives to PCs, so I can run Crystal Mark tests, and here I see interesting things:

Explorer (download and upload are the same speed):

Speed is ok for a few second at the beginning, but after that it goes down till the end.


NAS is virtualized, HBA and NIC are passthrough :
1) Ryzen 5800X - 8 cores for TrueNAS
2) 128 Gb RAM - 64Gb for TrueNAS
3) NIC - Intel X540-AT2
4) HBA - LSI 9305-16i
5) Pool #1, RAIDz1 - 4x Micron 5200 MAX 960Gb
6) Pool #2, RAIDz1 - 4x Micron 520 ECO 1.8Tb

Switch: Unifi Flex XG

Desktop NICs: Marvell AQC113, Marvell AQC111C

I didn't have this issue with TrueNAS Core after small tweaks from here. TrueNAS Scale has cubic protocol by default, and I don't see any Tunables from the Core version, so I don't know what I can do.
The disks look ok, fio shows ~480 Mb/s speed, so I don't expect that both SSD pool are bad.

Does anyone have any ideas?
Thank you in advance!

 

[sretalla]

So what's important is to first look at iperf (iperf3).

You need to run that on client and server in the respective mode each and see if you can get 10G or approaching that just from network alone.

If you can't, then you need to look into why that is.

It seems you've already said the disk performance meets your expectation, so there's only really one place to look now (unless we need to further narrow it to something like the SMB service).

 

[Antoshka]

Hi @sretalla,
I did an iperf3 first. The results are in the post. Let me duplicate them one more time:
1 parallel (-P 1) gives 4.6 - 6.1 Gbits/sec
5 parallels (-P 5) give 9.3 - 9.4 Gbits/sec
Also, as I mentioned already, I tested in both directions (NAS is server, and NAS is client).
In addition, I haven't changed anything from a network perspective when I switched from Core to Scale, so it's doubtful that's something w/ the network.

 

[Antoshka]

Hello everyone,
I find out a root cause - encryption.
Unfortunately, I was losing up to 80% of my speed due to encryption. This is another example.
Alright, it means I have two options: move back to Core OR recreate my pools without encryption.