7x 8TB IronWolf only seeing ~100MB/s read/write

[mrsir]

Hi team,

I'm in the process of jumping ship from Unraid, because I'm ready for a bigger setup. I've got my disks loaded up and am in the process of transferring files into my datasets, but I'm not getting the performance I was expecting.

OS Version: TrueNAS-SCALE-22.12.0
Product: SYS-6028R-T
Model: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
Memory: 31 GiB (4x 2133 registered 8GB) - have more RAM on the way, but it's stuck in holiday shipping
HDD: 7x8TB IronWolf
Config: RAID-Z2
HBA: LSI 9207-8i (IT Mode)
Motherboard: X10DRi

What I am trying to do
rsync media files from USB3 external hard drives via a debian VM running on TrueNAS to a dataset via samba.

What I am expecting
I'm expecting dataset writes to be clocking >200MB/s (i.e. - I believe I should be bottlenecked by the performance of the external hard drives).

What I am seeing
rsync from external hard drive to the samba share seem to float around 110MB/s

What I have tried

1. iperf VM -> TrueNAS (22 Gbits/s)
2. iperf VM -> TrueNAS -R (28 Gbit/s)
3. Changing dataset config (compression off, record size to 1 MiB, sync off, dedupe off)
4. SFTP transfer from desktop machine to TrueNAS (1 Gigabit link) (93MB/s)
5. `dd if=/dev/random of=test bs=1G count=5` from TrueNAS shell (270MB/s sync off, 220MB/s sync on)
6. `dd if=/dev/random of=test bs=1G count=5` from VM to samba share (140MB/s sync off)
7. rsync file from VM to TrueNAS via SSH (110MB/s)

Despite my (relatively low) RAM, there always seems to be a decent chunk that is freeable, and neither the VM vCPU or TrueNAS seem to be working beyond 20% load. Everything seems to point to a network issue between the VM and TrueNAS (loopback), but iperf seems to discount this.

I'm running out of ideas a little bit and was hoping someone with TrueNAS experience might be able to nudge me in the right direction - especially if there is a glaring issue in my hardware setup!

 

[jgreco]

Perhaps a little resetting of expectations is in order. I'll concede it feels like this is somewhat slow-ish, BUT:

1) RAIDZ2 performance is typically fairly limited. Using multiple RAIDZ2 vdevs or using mirror vdevs is faster. A single RAIDZ2 vdev will be constrained in the number of IOPS it is capable of; it will be proportional to the speed of the slowest member device in the RAIDZ2 vdev.

2) The use of /dev/random for dd performance testing is strongly disrecommended because this is really just testing how good the random device driver's performance is. Turn off compression on the ZFS dataset in question and use /dev/zero instead. This way you will not be puzzling over whether random is acting as an artificial limit.

3) I'm going to go out on a limb here and cause haters to hate and angry Linux fans by saying that KVM may not doing you any favors here. I don't use SCALE personally but I have gotten the distinct impression a bunch of times when discussing performance testing jigs like yours that something may not be quite right, possibly because maybe you only have one or two 2.0GHz CPU cores assigned in the VM (you didn't indicate) but also possibly just because you're expecting quite a bit of context switching to be going on, and I never got the impression that KVM was as good at this as ESXi.

4) Heaped on top of that, the use of USB in this manner is probably also adding to the stress, since presumably the TrueNAS host is having to proxy this data on behalf of the VM.

It might be interesting to see if you can set up the data transfer directly on the TrueNAS host by mounting the external USB HDD's at the shell prompt. This would eliminate so many weak links in this pipeline you've set up.

 

[mrsir]

Perhaps a little resetting of expectations is in order.

and that's a perfectly valid point. I'm highly reliant on annecdotes I've read online, so I'll be the first to say I don't actually know what to expect here.

Turn off compression on the ZFS dataset in question and use /dev/zero instead.

Thanks for this - it was looking me right in the face but I didn't think of it (hence why I used random).

possibly because maybe you only have one or two 2.0GHz CPU cores assigned in the VM (you didn't indicate)

I have 8 cores assigned, but your point still stands - I don't trust KVM yet either, but I was trying to get a stick in the ground and work my way back a bit.

Heaped on top of that, the use of USB in this manner is probably also adding to the stress,

True - I hadn't really considered this in it's entirety. I have no clue how efficient the XHCI drivers/USB passthroug his under SCALE, and I could definitely just eliminate that as you have suggested.

Thanks for your input - some real concrete stuff for me to chase down. Appreciate it. I'll circle back once I've had another crack.

 

[Davvo]

Just to be sure... Your pool used space is under 80% right?
Also iirc switching off deduplication doesn't make it disappear from the dataset.

 

[jgreco]

I have 8 cores assigned,

Ow. I think that might be hurting. Try two, first, then four, and see if four is faster or slower (or the same).

Also iirc switching off deduplication doesn't make it disappear from the dataset.

@Davvo clearly read a bit more deeply into implications than I did. If your message was intended to imply that you had enabled dedup at some point, yes, this can permanently damage the pool's ability to perform well. Drop the pool and start fresh.

 

[mrsir]

Just to be sure... Your pool used space is under 80% right?
Also iirc switching off deduplication doesn't make it disappear from the dataset.

This is my first time filling the pool, and I've only just crossed 1% utilisation. Deduplication was disabled from the beginning (the conclusion I came to after reading up was that it's an "enable if you're sure you need it" type of thing).

Here's where I'm at.

1. I've turned off the VM
2. I've mounted the external drives directly onto a mountpoint in TrueNAS itself
3. I disabled compression and wrote a 128G file from /dev/zero to the dataset - got 957MB/s
4. I copied from an external drive to my Z2 pool and got 40MB/s
5. I copied from an external drive to my 1x NVME stripe (just scratch) and got 40MB/s
6. I copied a file from the external drive to the NVME stripe (40MB/s) and then I copied that file into my array and got 461MB/s

Minor clarification: the 100MB/s I was seeing prior was when copying from all external drives at once.

All signs now seem to point to USB itself being the problem! If I'm getting 40MB/s that tucks nicely into the 480Mb/s limit of USB 2. In theory my USB ports are USB 3.0... but it sounds like I need to do some digging in the BIOS to make sure they're properly configured.

This gives me some comfort that my array is fine but USB is not, which is thankfully something that is only a problem during the initial load of the data.

 

[Morris]

In my experience on Core, RAID Z2 is very slow for writes. I agree with jgreco
"1) RAIDZ2 performance is typically fairly limited. Using multiple RAIDZ2 vdevs or using mirror vdevs is faster. A single RAIDZ2 vdev will be constrained in the number of IOPS it is capable of; it will be proportional to the speed of the slowest member device in the RAIDZ2 vdev."

RAID Z1 will be faster yet still slow. Use multiple vdevs or dirror vdevs. 2- RAID Z1 3 drives each and a hot spare or 3- mirrors with hot spare. Mirrors will get you faster write and much faster reads and lower CPU. 2-RAID Z1 will get you very fast writes and reads yet not as fast as mirrors. You will need to chose speed v drive overhead.