ZFS write performance issues for a 4 drive raidz1 pool

[mcclown]

So I'm running TrueNAS-SCALE-22.12.1 and I have the following setup

Qnap TS-453Be
16GB RAM
2x SSD boot pool - both over USB.

Single raidz1 pool
- 2x WD 16TB (WUH721816ALE6L4)
- 2x Seagate 8TB (ST8000VN004-3CP101)
(I intend on upgrading the two 8TB over the coming months, to bring the whole pool on to 16TB drives)

I upgraded the networking to 2.5Gbe using one of these, there was a bit of messing around to get it to fit but it's working fine now.

This NAS is used as a backup for another Qnap that I have, which is still running the QTS. The other Qnap has built in 2.5Gbe and they're wired together directly, via the 2.5Gbe ports. They're also both connects to my standard 1Gbe network via their other port.

• So I've tested with iperf3 between the two NAS devices and get ~2.3Gbps reported, both directions.
• When I started using rsync to bring over the full backup files, I'm seeing speeds ~1Gbps.
• I did some testing with transferring files directly between the devices using wget and I saw a slight improvement of 100-120MB/s but still a long way off 2.5Gbe speeds.
• I decided to eliminate the disks as an issue and fetch files directly to RAM, using /dev/shm and when I do that I get over 220Mb/s, so roughly what I'm looking for.
• When I look at my disk IO reports in the TrueNAS dashboard, they are all averaging about 35MiB/s each, which seems very low to me.

Is there anything I can do to improve on this? Or is this to be expected with ZFS in this type of configuration? It looks like getting the 2.5Gbe may have been a bit of a waste, as I see the same performance over the 1Gbe network that I have on the 2.5Gbe network currently for rsync.

 

[sretalla]

They're also both connects to my standard 1Gbe network via their other port.

I've tested with iperf3 between the two NAS devices and get ~2.3Gbps reported, both directions.

Sounds like you need to be a lot clearer about what you're running and which IP addresses are used by what.

Also, that's a realtek card, so don't expect great reliability in general.

 

[WI_Hedgehog]

If you're transferring a lot of small files the speed decreases. Try transferring a batch of large files for comparison.