10GbE slow read/write speed

[Fastline]

Hello guys,

New here and i have the following setup.

Motherboard: GIGABYTE B460M D3H
CPU: Intel Core i3 10400
RAM: Corsair 4x16GB DDR4 3200MHz
Storage: Seagate 4x16TB EXOS (SATA 6Gb/s onboard)
Network: ASUS 10GbE Base-T and the same adapter on the client computer (Connected via CAT6 Cable)
Switch: MikroTik Cloud Router Switch (CRS312-4C+8XG-RM)
PSU: Antec 850W Gold

Currently, there is no cache (L2ARC) or SLOG device configured. I tried but the effect is the same. The issue is no matter i read or write, initially, the speed is full 10Gb/s but once 10GB of data is written, the speed is near about 5-6Gb/s sometimes 7Gb/s. I have tried almost everything. When doing iperf, i get straight 10Gb/s. The same thing is with UNRAID. At a point of time, i tried to configure Cache and SLOG, and vdev as a PCIe NVMe SSD but still the same situation. Of course, i plan to increase the RAM, switch to SPF+ and move to Xeon (C621/C621A) but want to know what's the bottleneck here.

Please let me know your suggestions guys.

Thanks

 

[Arwen]

Write speeds to disks tend to be limited to the disks themselves. The RAM acts as a bit of asynchronous write buffer, (aka ZFS write transaction space), but whence full, delays & slowdowns occur.

I did check the network card's chip set, and it looks good;

Intel® X710-AT2 network controller

You don't specify how the 4 x 16TB disks are configured, as in RAID-Z1, RAID-Z2, Mirror pairs or simple, un-redundant stripe. This makes a serious difference in how many disks are used for data.

 

[Fastline]

Write speeds to disks tend to be limited to the disks themselves. The RAM acts as a bit of asynchronous write buffer, (aka ZFS write transaction space), but whence full, delays & slowdowns occur.

Yeah, that makes sense. But does this means no one is able to achieve full actual 10Gb/s speeds?

I did check the network card's chip set, and it looks good;

No, it's AQC-107.

You don't specify how the 4 x 16TB disks are configured, as in RAID-Z1, RAID-Z2, Mirror pairs or simple, un-redundant stripe. This makes a serious difference in how many disks are used for data.

Oh, sorry, i completely forgot. It's configured in RAID-Z1.

As per the hardware recommendations guide, it suggests using 1GB per 1TB so that means 5x16TB=80TB which means 80GB RAM whereas i have 64GB. Is that what making the difference? Do note that if i use 1GbE or 2.5GbE or even 5GbE, i get same Read/Write speed but when i switch to 10GbE, it writes at full 1Gb/s speed and as soon as 10GB of data is written, it saturates and starts giving like 300-400Mb/s which is quite low for me. Any suggestions on this?

 

[HoneyBadger]

Yeah, that makes sense. But does this means no one is able to achieve full actual 10Gb/s speeds?

It's closer to the opposite, actually - thanks to the "buffering" of writes into the ZFS transaction groups, everyone is able to make use of that 10Gbps speed for some period, until their pool disks bring those pesky laws of physics in to ruin the fun.

Writing at 10Gbps requires that your back-end pool be able to continually ingest those writes at that speed; this is a combination of the type of device (HDD vs SSD), the pool configuration (mirrors vs RAIDZ) and the number of them ("more vdevs are better" is the general rule)

 

[Fastline]

It's closer to the opposite, actually - thanks to the "buffering" of writes into the ZFS transaction groups, everyone is able to make use of that 10Gbps speed for some period, until their pool disks bring those pesky laws of physics in to ruin the fun.

Writing at 10Gbps requires that your back-end pool be able to continually ingest those writes at that speed; this is a combination of the type of device (HDD vs SSD), the pool configuration (mirrors vs RAIDZ) and the number of them ("more vdevs are better" is the general rule)

Hmm, i get that. But for a typical 10Gbps NAS setup, how much vdevs and RAM to add so that the speed does not saturate even if you try to copy a 1TB file on a 100TB NAS box?

 

[Fastline]

@HoneyBadger If that's the case, i tried to use PCIe NVMe having 3500/3000 speed as a vdev but the same thing happens. So, i think if using the PCIe NVMe SSD with that much of speed, it should be able to constant transfer with the speed of 1Gb/s or am i wrong here?

 

[HoneyBadger]

Hmm, i get that. But for a typical 10Gbps NAS setup, how much vdevs and RAM to add so that the speed does not saturate even if you try to copy a 1TB file on a 100TB NAS box?

RAM benefits read speeds more than writes. I'd suggest at least twelve HDDs would be necessary to get close - and that would be with a purely sequential write workload of copying files without any competing reads.

@HoneyBadger If that's the case, i tried to use PCIe NVMe having 3500/3000 speed as a vdev but the same thing happens. So, i think if using the PCIe NVMe SSD with that much of speed, it should be able to constant transfer with the speed of 1Gb/s or am i wrong here?

Advertised numbers from storage devices are often written for peak performance, not sustained - depending on the device, it may have a much smaller space where it can write "fast" and then it again slows to the speed of its underlying NAND. What model is the NVMe SSD?

 

[Fastline]

RAM benefits read speeds more than writes. I'd suggest at least twelve HDDs would be necessary to get close - and that would be with a purely sequential write workload of copying files without any competing reads.

Oh, damn, and do you have any idea about the file size? I mean currently, I can copy 10GB file and it copies with full speed for a minute and then slows down to half of the speed.

Advertised numbers from storage devices are often written for peak performance, not sustained - depending on the device, it may have a much smaller space where it can write "fast" and then it again slows to the speed of its underlying NAND. What model is the NVMe SSD?

I used SK Hynix PC801 in a Gen3 NVMe slot. Now, if this was regarding cache which tells us insufficient vdev to maintain write cache, so I think having an NVMe SSD with a good write speed should fix that issue but still, i encounter it. So, I suspect something else. Cause, technically, the problem is of too less vdevs which is insufficient to maintain a write cache but installing a NVMe drive with a good write speed should fix that. Or am I wrong here?

 

[Arwen]

The 1GB of RAM per 1TB of disk suggestion is just that, a suggestion. And an old one too. Whence you get to 64GB of RAM it is less important, though more RAM does help for higher speed network cards. However, you likely don't need more RAM for your use case.

With only 3 data drives, (4 x 16TB in RAID-Z1), that limits your performance to what they can ingest, continuously. ZFS will create write transaction groups in memory, before flushing them to disk. Thus, higher speed initially for large writes.

Some NVMe drives only get maximum performance in specific, limited configurations. That would be a whole other discussion.

Now on to the network card. If you are truly using a Marvell Aquantia AQC 107, then that is part of your problem. Lower end, (aka lower cost), network chipsets tend to have reduced throughput than well known brands, like Intel, Chelsio, etc... Even lower cost cards from Intel, Chelsio and such, do perform better than other brand's cards.

I don't know this is a limiting factor, but it has been such for cheaper network cards in the past.


One last note. While I think ZFS, (and TrueNAS), are great, ZFS is not the fastest file system, volume manager and RAID out their. For most of us, data safety & manageability supplied by ZFS out weighs the few extra percentage points that could be achieved by using other software. A certain youtuber did a a show on a specialized file system designed for NVMe and such. It produced fantastic speed results, but the management and data safety were not covered as well as what is known about ZFS.

 

[Fastline]

The 1GB of RAM per 1TB of disk suggestion is just that, a suggestion. And an old one too. Whence you get to 64GB of RAM it is less important, though more RAM does help for higher speed network cards. However, you likely don't need more RAM for your use case.

Yes, i get this totally.

With only 3 data drives, (4 x 16TB in RAID-Z1), that limits your performance to what they can ingest, continuously. ZFS will create write transaction groups in memory, before flushing them to disk. Thus, higher speed initially for large writes.

Actually, it's 5x16TB in RAID Z1. Please note that i have tried 4xPC801 512GB and still have the same performance as on my HDD Pool. I do get that less the vdevs, less speed on a higher network such as 10GbE, the connection will saturate as the write is much faster to the RAM, and due to less vdevs, it cannot copy the data from the RAM to the disk. Is that correct? However, using the PCIe NVMe SSD with that much speed, it should be able to constantly transfer with the speed of 1Gb/s or am I wrong here?

Now on to the network card. If you are truly using a Marvell Aquantia AQC 107, then that is part of your problem. Lower end, (aka lower cost), network chipsets tend to have reduced throughput than well known brands, like Intel, Chelsio, etc... Even lower cost cards from Intel, Chelsio and such, do perform better than other brand's cards.

Unfortunately, I'm running AQC107 chip from ASUS Base-T. After testing several things, i have also narrowed down that it's my NIC. But if i want to upgrade, I'm really not sure what to pick. I mean if Base-T gives me 1Gb/s speed constantly, i would be really happy provided that the NIC can be bought for cheap, should run cool, and should be compatible with macOS. Today, i was reading again about SFP+ and Fiber and they seem quite interesting.

Any recommendations?

I don't know this is a limiting factor, but it has been such for cheaper network cards in the past.

I think its the only factor but I do not have any other 10GbE NIC to test.

 

[Arwen]

...
Unfortunately, I'm running AQC107 chip from ASUS Base-T. After testing several things, i have also narrowed down that it's my NIC. But if i want to upgrade, I'm really not sure what to pick. I mean if Base-T gives me 1Gb/s speed constantly, i would be really happy provided that the NIC can be bought for cheap, should run cool, and should be compatible with macOS. Today, i was reading again about SFP+ and Fiber and they seem quite interesting.

Any recommendations?


I think its the only factor but I do not have any other 10GbE NIC to test.

No, sorry, the 10Gbps Ethernet Resource, (some listed below), and sticky threads are the only additional information I have;

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

10 Gig Networking Primer

This document is now available in the Resources section! Use the tabs above to navigate to the document itself. The current version also follows inside the spoiler tag: Also, if you'd like to repost this elsewhere, I'd appreciate it if you would have the courtesy to ask. And to credit the...

www.truenas.com

 

[Davvo]

No, it's AQC-107.

Not good (see 10Gig primer).

For pool layouts:

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

TL;DR: you want mirrors.

For testing:

solnet-array-test

Back in the late '90's, I was managing a bunch of large whitebox storage servers. For the largest of these, I had the pleasure of building and deploying a massive storage server, 8 shelves of 9 drives each, Seagate ST173404LW 73GB drives, a...

www.truenas.com

Also, what is the distance between the nas, the switch, and the client? Cat6 should need max 50m for best performance iirc.

The tuning guide @Arwen posted is really useful, make sure to read it.

 

[Fastline]

Not good (see 10Gig primer).

For pool layouts:

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

TL;DR: you want mirrors.

For testing:

solnet-array-test

Back in the late '90's, I was managing a bunch of large whitebox storage servers. For the largest of these, I had the pleasure of building and deploying a massive storage server, 8 shelves of 9 drives each, Seagate ST173404LW 73GB drives, a...

www.truenas.com

Also, what is the distance between the nas, the switch, and the client? Cat6 should need max 50m for best performance iirc.

The tuning guide @Arwen posted is really useful, make sure to read it.

Its all under 5 meters setup via CAT7.

Basically, i do understand that i have less RAM (64GB) and also less vdevs. But the question if I'm using NVMe (2xPC801 512GB) in Stripe as a NVMe vdev, why the speed still saturate? I know this is not an enterprise grade NVMe having low latency or high endurance or the IOPS, but still has 7000MB/s Read/Write and it should be able to maintain 10GbE when doing a read/write to and from another same NVMe. I highly suspect either the TrueNAS configuration (maybe i need to tune something) or maybe its the Switch or the NIC itself. So far i have only tried to set the jumbo frame to 9000 but its the same. Long time back, i tried the same thing on UNRAID and it had the same situation as well.

Guys help me, I'm not sure what is the bottleneck here. You can find the specs above, in the first post.

Also, @Etorix and @danb35 please guide.

 

[Davvo]

What is RAIDZ0? I only understand mirrors and RAIDZ while on this forum.

You can't just look at the declared speeds of a commercial NVMe... also ZFS dataflow is not the same as other file systems.

The issue with the Aquantias should be the same as with the Realteks: driver issues.

 

[Fastline]

What is RAIDZ0? I only understand mirrors and RAIDZ while on this forum.

Mistake, like i mentioned, still new to the NAS thingy. Edited. I meant configured as Stripe.

The issue with the Aquantias should be the same as with the Realteks: driver issues.

Umm, but i guess TrueNAS supports the AQC officially?

 

[Davvo]

Umm, but i guess TrueNAS supports the AQC officially?

Define officially? Iirc they only ship Chelsio cards with their systems. @jgreco should be able to confirm this.

 

[Fastline]

Define officially? Iirc they only ship Chelsio cards with their systems. @jgreco should be able to confirm this.

I meant it has drivers for it. Not saying that iXSystems use AQC in their units they manufacture/assemble/build.