Slow transfer speeds with compression on dataset

[jperham-ct]

Hello,

I'm having speed transfer issues with my TrueNAS system when compression is enabled on a dataset.

The issue is the transfer takes a good 30 seconds to start and then the speed drops down to 5.54MB/s, then down to 0 bytes, maybe speeds up for a bit and then stalls out again and repeats the cycle. The problem presents itself when highly compressed file is copied from a compression enabled dataset to any other dataset. It doesn't matter whether the destination dataset has compression enabled or not.

The specs of the system are:

Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz
128GB ECC RAM
53 WDC WUH721818AL5201 drives in a RAIDZ pool
TrueNAS-13.0-U5.3

I've recorded some video of what I'm seeing.

The video below is copying a highly compressible file from a non-compression enabled dataset to gzip enabled dataset. No speed issues with this transfer are observed.

copyfromnocomptocomp.mp4

drive.google.com

The video below is copying that same highly compressible file within a compression enabled dataset to another folder within the compression enabled dataset. This is the issue behavior:

compressioncopy.mp4

drive.google.com

The highly compressible file stats look like this:

I've tried lz4 and gzip compression and it's the same behavior with both. I've also confirmed it's not networking related by completely wiping a Windows 11 Pro machine, connecting it directly to the TrueNAS system via ethernet and attempting the same transfers as above. The issue is the same on that test device.

Any ideas?

 

[Arwen]

First off, many performance issues can be tracked to the pool layout. You list;
53 WDC WUH721818AL5201 drives in a RAIDZ pool
But, you don't list HOW they are setup.

For example, 5 vDevs of about 10-11 disks each, would be one way to configure 53 disks.

So, what is your pool's layout?

 

[jperham-ct]

First off, many performance issues can be tracked to the pool layout. You list;
53 WDC WUH721818AL5201 drives in a RAIDZ pool
But, you don't list HOW they are setup.

For example, 5 vDevs of about 10-11 disks each, would be one way to configure 53 disks.

So, what is your pool's layout?

Apologies, I'm new to knowing what info to provide when talking about a TrueNAS setup.

Here's some additional info:

Code:

root@fs3[~]# zpool status -P
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:09 with 0 errors on Tue Nov 14 03:45:09 2023
config:


        NAME           STATE     READ WRITE CKSUM
        boot-pool      ONLINE       0     0     0
          /dev/nvd0p2  ONLINE       0     0     0


errors: No known data errors


  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 13:56:11 with 0 errors on Sun Nov 12 13:56:12 2023
config:


        NAME                                                 STATE     READ WRITE CKSUM
        tank                                                 ONLINE       0     0     0
          raidz3-0                                           ONLINE       0     0     0
            /dev/gptid/cea314c8-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/cec979a0-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/cfe61047-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/cfc57389-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/cfcebbf2-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/cfda7668-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d0045e97-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d0254333-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d2e6fde7-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d303759f-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
          raidz3-1                                           ONLINE       0     0     0
            /dev/gptid/ceb08c66-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/cebba123-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d26eba01-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d2784ffb-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d2b187b0-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d2c42c4a-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d3ea8377-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d40944c9-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d41f8186-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d4316e4d-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
          raidz3-2                                           ONLINE       0     0     0
            /dev/gptid/d3f512a8-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d4fc2d90-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d529acc1-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d53e3d0d-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d533ad65-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d55426fd-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d580afdc-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d68587e7-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d6bc547d-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d682e960-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
          raidz3-3                                           ONLINE       0     0     0
            /dev/gptid/d6ac47b4-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d6c869c6-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d6d33f1b-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d7d23f3f-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d7f99d3d-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d7f018a9-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d8130f77-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d81e7a3e-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d836d83b-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d97bde8d-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
          raidz3-4                                           ONLINE       0     0     0
            /dev/gptid/d91ca0d9-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d92737ed-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d95130af-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d95f099a-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/d96dc0a2-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/da803142-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/da8aef60-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/da7190f9-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/dafa6342-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
            /dev/gptid/db5e1d69-1b49-11ed-8f1e-3cecef62b8ce  ONLINE       0     0     0
        logs
          /dev/gptid/da468208-1b49-11ed-8f1e-3cecef62b8ce    ONLINE       0     0     0
        spares
          /dev/gptid/db088c23-1b49-11ed-8f1e-3cecef62b8ce    AVAIL
          /dev/gptid/db4bf40e-1b49-11ed-8f1e-3cecef62b8ce    AVAIL
          /dev/gptid/db6818f2-1b49-11ed-8f1e-3cecef62b8ce    AVAIL


errors: No known data errors

Code:

root@fs3[~]# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool   216G  7.49G   209G        -         -     0%     3%  1.00x    ONLINE  -
tank        819T   325T   494T        -         -     1%    39%  1.00x    ONLINE  /mnt

 

[Arwen]

Using RAID-Z3 takes more time for writes than RAID-Z2, And with 3 spares, RAID-Z3 could be considered over-kill.

Their are other things that affect writes. I am no expert on performance problems.

Perhaps someone else can assist further.

 

[HoneyBadger]

Hey @jperham-ct

I believe this is a compression (or rather, decompression) related issue. OpenZFS 2.2's block cloning feature should circumvent this (as the records are cloned in their compressed format) but that's currently only in SCALE 23.10 - CORE 13.1 will likely get this as well, but I know that doesn't correct the issue you're dealing with now.

 

[winnielinnie]

OpenZFS 2.2's block cloning feature should circumvent this (as the records are cloned in their compressed format)

Across datasets? (That traverses different filesystems.) @jperham-ct seems to imply there's a massive slowdown when copying a file from one dataset to a different dataset.

Unless I'm conflating two different things here.

 

[HoneyBadger]

Across datasets?

Yes, block cloning works across datasets - just not across pools.

 

[winnielinnie]

That's hard for my mind to grasp. I might start a new topic on it, since I don't want to go off topic here. But I can't imagine how something like replications would work. A dataset contains a file with pointers to blocks... on another dataset?

 

[jperham-ct]

Across datasets? (That traverses different filesystems.) @jperham-ct seems to imply there's a massive slowdown when copying a file from one dataset to a different dataset.

Unless I'm conflating two different things here.

The issue I'm seeing occurs with a copy of a highly compressed file within the same dataset or to another dataset but for the issue to occur, the file has to have originated from a dataset where compression was enabled and the file was compressed.

Hey @jperham-ct

I believe this is a compression (or rather, decompression) related issue. OpenZFS 2.2's block cloning feature should circumvent this (as the records are cloned in their compressed format) but that's currently only in SCALE 23.10 - CORE 13.1 will likely get this as well, but I know that doesn't correct the issue you're dealing with now.

Thank you for the info. Thankfully, it appears that update isn't too far out according to the release timeline and hope the update includes block cloning.

I've been going back and forth for a while now with support trying to get to the bottom of this issue.

 

[HoneyBadger]

Thank you for the info. Thankfully, it appears that update isn't too far out according to the release timeline and hope the update includes block cloning.

13.1-MASTER right now has the same ZFS 2.2 as SCALE 23.10 - so that should be good.

I've been going back and forth for a while now with support trying to get to the bottom of this issue.

Can you DM me your support case number?

 

[jperham-ct]

13.1-MASTER right now has the same ZFS 2.2 as SCALE 23.10 - so that should be good.



Can you DM me your support case number?

Just DM'd it.

Thank you!

 

[winnielinnie]

@jperham-ct, curious: Did you try this out directly on the TrueNAS server (i.e, an SSH terminal) to rule out Windows and SMB?

So for example, you would use the "time" command with "cp" or "rsync".

Something like this:

Code:

time cp -v /mnt/mypool/alpha/test.bin /mnt/mypool/beta/test.bin

And then what about sending it to /dev/null ?

Code:

time cp -v /mnt/mypool/alpha/test.bin /dev/null

 

[HoneyBadger]

Local cp would work around this issue, but ssh'ing into the machine and manually shuttling files around every time you want to copy something isn't exactly an ideal workaround.

 

[winnielinnie]

I tried to recreate the issue myself. But no matter what, I got consistent speeds of about 115 MB/s when using Windows Explorer.

These are spinning WD Red Pluses, mirror vdevs only, 1-GbE Intel NIC, on TrueNAS Core 13.0-U5.3.

It made no difference if the dataset uses ZSTD (default, level 3) or LZ4.

Unless it's because I only use mirrors (no RAIDZ) that I don't experience this?

Local cp would work around this issue, but ssh'ing into the machine and manually shuttling files around every time you want to copy something isn't exactly an ideal workaround.

Of course. But it was meant to side-step around Windows Explorer, SMB, and the network, since the OP mentioned the consistent factor was "compression on the source dataset". (Because using Windows Explorer + SMB breezed by if the source dataset was not using compression.)

Was curious if there would be a notable slowdown if they tried it locally, with the source dataset using compression. (To see if Windows Explorer and/or SMB was a culprit in this, and that compression simply "added" to the problem or was the "needed ingredient" to expose the problem.



EDIT: Doh! I changed one crucial factor. I tested with a 12 GB video file. (Which is obviously not compressible.) I have yet to test this out on a large non-compressible file.

 

[HoneyBadger]

EDIT: Doh! I changed one crucial factor. I tested with a 12 GB video file. (Which is obviously not compressible.) I have yet to test this out on a large non-compressible file.

Do you mean "you have yet to test this out on a large compressible file"?

 

[winnielinnie]

Do you mean "you have yet to test this out on a large compressible file"?

Yes.

(I just didn't want to disappoint you with my rapid-fire edits. I'm trying my best here!)

(Okay, that's a cop-out. I actually did write the wrong word, but didn't actually catch it. Quite surprising, knowing myself.)

 

[gyfer]

Using RAID-Z3 takes more time for writes than RAID-Z2, And with 3 spares, RAID-Z3 could be considered over-kill.

Their are other things that affect writes. I am no expert on performance problems.

Perhaps someone else can assist further.

It's not necessary for RaidZ3 to be slower than RaidZ2. The read and write performance of a RAID pool, especially for streaming large files, depends on the number of drives in a vdev across the pool. For example, in a 12-disk pool with a single vdev using RaidZ3 (12 disks - 3 parity), there are 9 data disks. In contrast, a pool with 2 vdevs, each consisting of 6 disks and using RaidZ2 (12 disks - 4 parity), results in 8 data disks. In this setup, RaidZ2 might be slower in throughput speed for read and write operations involving large files but faster for numerous smaller files.

The choice between RaidZ3 and RaidZ2 ultimately depends on personal preference and specific needs for fault tolerance in a pool. Rebuilding an 18GB drive can take days or even weeks. If another drive fails during the resilvering process, there won't be any additional room for another disk failure in the same vdev. In comparison, a 2xRaidZ2 setup is more likely to fail than a 1xRaidZ3. Similarly, a 6xmirror pool is less prone to failure than a 4x3-disk RaidZ1 setup.

 

[gyfer]

View attachment 72467

Any ideas?

When copying files using Windows, the operating system scans the files for viruses or malware. This means that you have to decompress your entire 13.4GB file, read and analyze it with the antivirus, and then write it to the destination. Turning off your real-time protection can help improve the overall speed and throughput during this process. I hope I am right.

 

[winnielinnie]

Can't reproduce this. With a "highly compressible" 1.5 GB file on a ZSTD dataset:

• Nearly instant (only 1.5 seconds) to copy the file to the same SMB share
• Nearly instant (only 1.5 seconds) to copy the file to another SMB share on a different dataset

This demonstrates server-side copy (not block-cloning, since I'm on TrueNAS Core 13.0-U5.3.)

Same exact result, whether using Linux or Windows 10 as the client PC connecting to the network share.

NVMe drives in a mirror vdev. ZSTD compression on all datasets involved. 1-GbE network.

Same thing even with a 10 GB file created out of /dev/zero.

 

[winnielinnie]

Just wanted to add, I've tried every combination I could to recreate this on Windows 10 and Arch Linux as SMB clients. No matter what I tried, server-side copy worked as expected. No slowdowns whatsoever. The speed at which the large file copied was congruent with the speeds of a local copy; even though it was over the 1-GbE network via an SMB share.

So I'm sort of out of ideas here. The main difference is that my vdevs are mirrors, and you are using RAIDZ3.

I would be curious to know if you witness the same behavior on a different Windows PC or even on a Linux client?


@gyfer could be onto something, in regards to antimalware software causing this issue. On my Windows 10 PC, I do have Microsoft Defender's real-time protection. (But obviously, it's not an issue on my end.)