Getting faster speed /read / write with 10Gbps Ethernet

[BRIO]

NAS specs:
Motherboard: ASUS Prime X570-Pro
CPU: AMD Ryzen 5 5600G
RAM: T-Create DDR4 64GB 128GB (4X32GB)
10Gbps PCIe Card: Intel X540-T2 Dual 10GB port ( https://www.ebay.com/itm/143592050461?var=442552216136 )
4X 16TB Ultrastar HDD (Raid-Z2)
1TB M.2 Cache Vdev

Some small files (around 1GB to 3GB) transfer pretty fast around 1GBps from my NAS to my computer M.2 OS drive. Though when it comes to larger files it can peak at 1GBps but quickly go down around 200MBps / 300MBps. My first reflex was to purchase more M.2 drives to have a bigger Vdev cache storage but when I started to dig down on the subject, everything seems to point toward the RAM. I'm currently maxed out with 128GB of RAM using my actual Motherboard.

My questions are:
1. How can I get a constant 1GBps of transfer speed with my current setup (Motherboard / CPU)?
2. Is it possible to create a cache of any sort with a couple of M.2 drives that will do: FILES ---> Cache M.2 ----> HDD so I can speed up my transfers / read / write?

 

[NickF]

Short answer? More disks.
You’re system is plenty fast and has plenty of RAM for ARC caching (reads), the issue you have now is simply that your disks are holding your performance bsck. A single RAID Z2 VDEV is never going to be fast.

 

[BRIO]

Ok so more Vdev disks or more HDD disks?

 

[NickF]

4 more drives (one new raid z2 VDEV) will roughly double your performance in most use cases

 

[BRIO]

Ok! So I would have 2 different pool in one? Sorry I'm pretty new to this so if you just want to point me where I can find the info that would really help

 

[NickF]

Ok! So I would have 2 different pool in one? Sorry I'm pretty new to this so if you just want to point me where I can find the info that would really help

Nope, a pool is built from 1 or more VDEVs. A VDEV is a collection of disks. In your case you chose RAID Z2, which is one of the safest option but also one of the slowest. Your pool is 1 VDEV of a 4 wide raidz2. What I suggested was a pool of 2 vdevs of a 4 wide z2.

An Introduction to ZFS A Place to Start

In this article, Nick gives an introduction to ZFS which is a good place to start for the novice user who is contemplating ZFS on Linux or TrueNAS

www.servethehome.com

 

[BRIO]

Nope, a pool is built from 1 or more VDEVs. A VDEV is a collection of disks. In your case you chose RAID Z2, which is one of the safest option but also one of the slowest. Your pool is 1 VDEV of a 4 wide raidz2. What I suggested was a pool of 2 vdevs of a 4 wide z2.

An Introduction to ZFS A Place to Start

In this article, Nick gives an introduction to ZFS which is a good place to start for the novice user who is contemplating ZFS on Linux or TrueNAS

www.servethehome.com

Ok I just finished to read the article, I can't say I 100% understood everything but it helped me to clear some things up. Overall for my personal case, how much drive do you suggest by VDEV? You mentioned 4 more drive, should I put 2 drives by VDEV? Also, how much storage should I get by VDEV drives to not overkill the thing but still get enough to get maximum speed?

 

[NickF]

Mirrors, two drives per VDEV, are always going to be fastest

 

[BRIO]

Mirrors, two drives per VDEV, are always going to be fastest

Thanks a lot! And one last question, how much GB/TB per VDEV drive should I get?

 

[sretalla]

Just to interrupt here a little...

"Speed" isn't just one thing in ZFS.

It's IOPS and Throughput.

IOPS are certainly a limit for RAIDZx and you're limited to more-or-less the IOPS of one single disk per VDEV for the whole pool.

Throughput (which is what you're measuring in large part with that single large file copy) on the other hand can be very good for RAIDZx and you may find that a relatively wide RAIDZ2 pool can perform as well as the sum of the individual disks' capabilities (minus a couple for parity and perhaps a bit extra in overhead), so a RAIDZ2 of 6 disks (each being able to read at maybe 100MB/s... for the sake of the exercise) can probably put out something like 350-400MB/s. A RAIDZ2 of 8 disks can probably do 550-600MB/s.

For a 4 disk RAIDZ2, you're going to see about the same throughput (but half the IOPS) of a pool of 2 Mirrors.

Maybe this will help:

Six Metrics for Measuring ZFS Pool Performance Part 2

Learn more about why OpenZFS (ZFS) storage pool layouts have such a significant impact on system performance under various types of workloads.

www.ixsystems.com