Migrating from FreeNAS 11.3U5 to TrueNAS Scale 22.12 - I lost any compression on videos with Scale

[ohboi]

I apologize and I don't want to sound ungrateful as you have really taken a lot of time to answer a bunch of my questions, but I am not getting this, unfortunately.
It isn't even about the difference anymore, just about what it is (I don't get the VDEV configuration argument, I don't see it mattering in anything else but speed, redundancy and risk factor) and it must be right - everything reports this seemingly impossible compression and I can't ignore it, for my storage use case, that 5% is massive.

I mean, I can try to test FreeNAS 11.3U5 on this new NAS and/or even back up the old server's configuration and taking that drive and putting it in the new one and try it over, but I am not sure if that isn't quite a witch hunt (which seems I am onto already).

 

[winnielinnie]

Why do you believe that your FreeNAS system/pool is reporting that a large uncompressible video file is supposedly taking up less space compared to all other reports on your TrueNAS SCALE system/pool?

Conversion? Metrics? Vdev layout/members/width?

Do you truly believe that your FreeNAS server is using magic to squeeze your uncompressible video file to such a degree where the file remains the same size (as it should) yet somehow requires fewer bytes to store it on the physical medium?

There could be something throwing you off in the way that FreeNAS reports this value compared to TrueNAS SCALE.

Try creating a 4 GiB random file (using "dd" and "/dev/urandom"). Compression will play no role. Does this 4 GiB file supposedly show that it takes up less than 4 GiB on the physical medium?

 

[ohboi]

I tested this by generating a 1GB file using dd if=/dev/urandom of=1GB.bin bs=64M count=16 iflag=fullblock. There is indeed a "compression" on that file 1GiB vs. 972 MiB.

If this is the VDEV structure, is there any read on the effects of VDEV structure on the efficiency of stored files?

 

[winnielinnie]

With what are you measuring the file sizes and sizes on disk?

You intuitively know that there's no way a file comprised of random non-compressible data is being "compressed" by LZ4 or ZSTD by a factor of around 3% shrinkage.

If this is the VDEV structure, is there any read on the effects of VDEV structure on the efficiency of stored files?

It was partially explained in the earlier link. But I don't think it applies in this case, since you're supposedly seeing the inverse (of data on disk less than the actual non-compressible file size.) I'm thinking either there's an error in how the outputs are being presented or some bug from an older version of FreeNAS / ZFS.

 

[ohboi]

In this case, only Windows Explorer. It is accurate, or at least as accurate as the FreeNAS server itself when doing the same using du -s, du -As or zfs list (I confirmed it with du).

In any case, I have also just tried to pack the testing file into an archive and used LZMA-9 compression and I can confirm, no compression. With that being said, even the archived file still ended up being "shrank" to 972 MiB.

At this point I am interested in just one question - what exactly is causing this? Is it a bug or is it being contributed by something else? If so, wwhat is it and how does it scale so that I can replicate those results?

 

[faviann]

In this case, only Windows Explorer. It is accurate, or at least as accurate as the FreeNAS server itself when doing the same using du -s, du -As or zfs list (I confirmed it with du).

In any case, I have also just tried to pack the testing file into an archive and used LZMA-9 compression and I can confirm, no compression. With that being said, even the archived file still ended up being "shrank" to 972 MiB.

At this point I am interested in just one question - what exactly is causing this? Is it a bug or is it being contributed by something else? If so, wwhat is it and how does it scale so that I can replicate those results?

I encountered the same issue and I'm wondering if you ever found the root cause of that discrepancy. I only get this problem with files that have been compressed or somewhat uncompressable. In that case, it seems compression is aborted or avoided.

One interesting element is that I'm using the same VDEV structure so that is definitely not the cause you guys were checking at previously.

My suspicion is that I remember reading somewhere that somekind of early abort had been put in place in zfs in case it can't compress but I can't seem to find how to disable it that feature to test out my theory. The reason I'm suspecting it is because I used to run zfs 0.8.* and now I'm running zfs 2.1.* on the new install.
Still hoping you've found a solution because in the end it leads to ~9% compression missing (more space being used/read)

 

[ohboi]

I encountered the same issue and I'm wondering if you ever found the root cause of that discrepancy. I only get this problem with files that have been compressed or somewhat uncompressable. In that case, it seems compression is aborted or avoided.

One interesting element is that I'm using the same VDEV structure so that is definitely not the cause you guys were checking at previously.

My suspicion is that I remember reading somewhere that somekind of early abort had been put in place in zfs in case it can't compress but I can't seem to find how to disable it that feature to test out my theory. The reason I'm suspecting it is because I used to run zfs 0.8.* and now I'm running zfs 2.1.* on the new install.
Still hoping you've found a solution because in the end it leads to ~9% compression missing (more space being used/read)

I have continued, as I was still intrigued and craved for some more information, on the ZFS subreddit

Reddit - Dive into anything

www.reddit.com

There was some good information there.

 

[faviann]

I have continued, as I was still intrigued and craved for some more information, on the ZFS subreddit

Reddit - Dive into anything

www.reddit.com

There was some good information there.

Thank you for sharing. That reddit post is truly great for information.

 

[RetroG]

my personal experience:

on ZFS, using LZ4 with 1MB recordsize results in a ~5% savings with MP4 and MKV files (across 300+TB worth of files, but is true at any scale from file to dataset), regardless of the topology (raidz and stripe on my experience)

not using compression and using 128k is a bad idea, I learned this as I expanded my pool and eventually 128k was no longer wide enough and resulted in wasted space! however turning on compression and changing the recordsize to 1MB, when rewriting the files in-place I was able to reclaim about 15TB at that scale.


just a bit of speculation, there is header data and subtitle data that could still be compressed? either way along the lines of "a video file is already compressed therefor it can't compress further" is a bit misleading. there is more nuance here.


a friend who also uses TrueNAS Scale, upon telling this to went through the effort of testing several TBs of his media under ZSTD and even at level 19 it was never better than LZ4.