From LZ4 to Zstd. Best way?

[mikael]

Hi!

Short version:
See screenshot below. I have data02 (mirror of 2*12 TB). What would be the best way to move the data on the data02 dataset (arrow 2 in screenshot) to Zstd? What I'm thinking:

• Change compression of dataset data02 (arrow 2 in screenshot below) to lz4 (not inherit).
• Change compression of base dataset data02 to Zstd (arrow 3 in screenshot below).
• "Add dataset" by clicking dots (arrow 1 in screenshot below). It will use Zstd and have the name data03.
• Copy files from data02 to data03.
• Delete data02.
• Rename data03 to data02.

Does this sound like a good way of doing it? Does the compression of the base dataset really matter if the dataset below it has another compression?

Longer version:
Installed Zstd on my workstation and noticed that Zstd at the default compression level (3) compresses my uncompressed .arw Sony raw files very well. A 50 MB raw file goes to 25 MB. On the TrueNAS, after running the below line, I notice that the same 50 MB files goes up to 40 MB with LZ4.

Code:

find . -type f -iname "*.arw" -exec sh -c "echo -n ' de-compressed: '; du -Ah '{}'; echo -n 'compressed (used): '; du -h '{}'; echo ' '" \;

The following TBs I will upload will mostly be uncompressed .ARW.

 

[Arwen]

Changing the base dataset is not required. If you purposefully change your dataset "data03" to use ZSTD compression, then that automatically means it won't inherit compression from the base dataset.

If you have enough space, it may be possible to use ZFS Send & Receive feature to copy "data02" to "data03" as one single copy. I've not tried it with changing compression, but a quick test on some temporary datasets & data might be worth while.

 

[Ericloewe]

Send/recv works fine, but will only recompress if the send does not use the compressed option.

 

[mikael]

Send/recv works fine, but will only recompress if the send does not use the compressed option.

Awesome thanks!

Changing the base dataset is not required. If you purposefully change your dataset "data03" to use ZSTD compression, then that automatically means it won't inherit compression from the base dataset.