Very slow 10Gbe copy performance

[agomendes]

Hello

I'm using Freenas (now TrueNAS) since long time ago and without any big problem.

Some days ago, I decided to change my NICs to 10Gbe because I need better file copy performance. However, the results are really bad :(

Here is my configuration:

Freenas:
AMD A4-5300 APU (dual core 3.4Ghz)
Gigabyte GA-F2A85XM-D3H (with 8 native SATA3 ports)
20 Gb RAM (2x8Gb + 2x2Gb)
NIC Chelsio N320E-SR 10Gbe (dual port SFP+ card)
Samsung SSD 120Mb for boot
POOL 1 - RAIDZ2 (4x Seagate IronWolf 4Gb 5900 RPM)
POOL 2 - MIRROR (2x Seagate Barracura 2Gb 7200 RPM)

Workstation:
AMD Ryzen 7 3700x Zen2
Asus X570 PRO
64Gb RAM (4x 16Gb)
WD 1TB Black SN750 NVMe SSD M.2
NIC Mellanox ConnectX-3 (single port SFP+ card)

• Both machines are connected with a SFP+ Cisco cable
• MTU is set to 9000 in both cards

I ran some tests and the results are the follow:

Iperf

Crystal Disk Mark (64Mb and 1Gb file sizes)

These seems very reasonable, but in real world the results are really bad. I can’t reach much more than 200Mb copying from NAS and 120Mb copying to NAS.

In the graph you can see a peak that is intentional (about 800Mb, which looks +- the iperf test) that matches a file that was cached by freenas (was copied just few seconds before).

It doesn’t matter the pool that I’m using because the results are the same (pool 1 have 5900RPM disks and pool 2 have 7200 RPM disks)

This must be a bottleneck somewhere, but I’m not figure out where it could be. First time I thought in the processor, but then I realise that the “Highest Usage” is almost the time under 25% and only few times reach about 75%.

Any help? Anyone as an idea where it could be the problem?

Many thanks for your help.

 

[agomendes]

Above, when I say Gb to my HDDs capacity, I would like to say Tb

 

[Hellione]

What speed do you expect? You have a pool with 2 data and 2 parity drives and only 5900rpm.
And the second pool is a mirror, so only the speed of 1 standalone drive.
I would say 200 read and 120 write is reasonable for this setup. Not much more you could expect.

 

[agomendes]

What speed do you expect? You have a pool with 2 data and 2 parity drives and only 5900rpm.
And the second pool is a mirror, so only the speed of 1 standalone drive.
I would say 200 read and 120 write is reasonable for this setup. Not much more you could expect.

@Hellione,

Many thanks for your answer.

Thinking a little deeper in the issue, yes, you are right. Indeed, it’s hard to find HDDs that can reach more than 300 MB/s. For example, Seagate Exos X18 has a maximum of 270MB/s sustained transfer rate; WD Ultrastar DC HC650 has 250MB/s; etc… (and these are enterprise / data center solutions).

Without upgrading to an SSD solution (that will be too expensive), can I change something to improve the final performance?

• Adding 2 additional drives into the POOL1? (with this I’ll have a pool with 4 data and 2 parity drives)
• Replacing the onboard SATA controllers by a HBA solution (for example LSI 9211-8i controler)?
• Adding an SSD for L2ARC cache (probably 1TB)?

Basically, I use the NAS to edit photos (Adobe Lightroom or similar applications) or videos directly into my workstation (all pictures / videos are in the NAS and databases are locally)

 

[Hellione]

I will do my best do give some advice :)

A HBA will make no difference at all. Those are only used when you need more drive ports then the mainboard has. Or if you want to virtualize, and passthrough the HBA. Or you reach the bottleneck between the cpu-chipset interlink, (which is between 3,9GByte/s PCIe 3.0 4x *intel consumer platform* and 7,8GByte/s with the newer Ryzen 3000/5000 PCIe 4.0 *consumer line*). You will only reach this limit with many SSD oder some NVMe drives. If you put your 10GBe NIC to the x16 (graphics) slot, this will be directly connected to the cpu, and will not use chipset lanes. So you have all bandwith for your drives.

More drives in a Pool will give you not more IOPs but more throughput, yes. For example, i use 6 drive raidz2 with those 7200rpm enterprise drives, and i can reach about 600-700MiB/s copy to the NAS. Large files only, small files it will rapidly slow down, for sure. Read is a bit slower, 350-450MiB/s.

I do not use l2arc at all, first you have to use memory, then mor memory, and then much more memory :) With less than 128GB RAM, i wouldn´t think of l2arc at all. and, l2arc is for reading same files multiple times, you won´t benefit if you read different files all the time. then everything has to be read from your slow drive pool.

So i would say, get faster drives, get more drives. Afterwards get more memory.

 

[Hellione]

Oh, and don´t rely on the maximum sustained transfer rate of drives all the time. If they are empty, they will reach those limit for a short time. But the more the drives fill up, they will become slower. Think of: One rotation of the outer side of a platter (where it begins to write) can hold maybe 3 times the data as the inner side. So you will drop down to maybe 150 or 100MB/s when nearly full. Hope you understand what i mean. English is not my primary language :)

That is one reason, i decided to make my pool double the size i need for the next 2-3 years. So each drive will not exceed 50% of its used space.

 

[Samuel Tai]

You may also need tunables to get better throughput.

10GbE performance issue in one direction

what is your MTU size? And are you seeing a lot of retries?

www.truenas.com