I've left the default lz4 compression on for all of my datasets except for a few that I've migrated to zstd. I was looking to get a bit more fine grained info on the amount of compression than what Compression Ratio can tell me. I found this post talking about the difference between du and du -A. https://www.truenas.com/community/t...n-of-a-single-file-has-answer-is-not-a.90145/
This led me to discover that some of my files are actually bigger on disk than they're reporting. I thought at first it might just be small files taking up a whole block but I found large files doing the same thing. I thought the compression was supposed to abort if it wasn't able to compress a file? So how did I end up with files bigger than the original? Am I missing something?
This is a dataset with 1M block size and lz4 compression.
truenas% du -Aha | sort -h | tail
3.6G ./S00E1075 - In_The_Next_Life_FOREVERVERSE_WEREWOLF [HHWVCyXJDUA].mp4
4.0G ./S00E1937 - Biomutant_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [uJJEkGhEhOc].mp4
4.2G ./S00E1926 - Minecraft_Part_3_-_Save_Point_with_Becca_Scott [eXTje8a8DsA].mp4
4.2G ./S00E1941 - Biomutant_Pt._2_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [QgUr08nAF34].mp4
4.4G ./S00E1916 - Jackbox_Games_Pt._2_-_Save_Point_with_Becca_Scott [L4haddIQBeo].mkv
4.5G ./S00E1923 - Fall_Guys_-_Ultimate_Knockout_-_Save_Point_with_Becca_Scott [JDEvGZjO14U].mp4
4.7G ./S00E1942 - Biomutant_Pt._3_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [0u0EwgmvAco].mp4
4.9G ./S00E1929 - Subnautica_pt._3_-_Save_Point_with_Becca_Scott [UxL8HYUrgig].mkv
5.7G ./S00E1919 - Subnautica_pt._2_-_Save_Point_with_Becca_Scott [I-ckh6CzuiQ].mkv
506G .
truenas% du -ha | sort -h | tail
3.6G ./S00E1075 - In_The_Next_Life_FOREVERVERSE_WEREWOLF [HHWVCyXJDUA].mp4
4.0G ./S00E1937 - Biomutant_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [uJJEkGhEhOc].mp4
4.2G ./S00E1941 - Biomutant_Pt._2_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [QgUr08nAF34].mp4
4.7G ./S00E1926 - Minecraft_Part_3_-_Save_Point_with_Becca_Scott [eXTje8a8DsA].mp4
4.7G ./S00E1942 - Biomutant_Pt._3_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [0u0EwgmvAco].mp4
4.9G ./S00E1916 - Jackbox_Games_Pt._2_-_Save_Point_with_Becca_Scott [L4haddIQBeo].mkv
5.0G ./S00E1923 - Fall_Guys_-_Ultimate_Knockout_-_Save_Point_with_Becca_Scott [JDEvGZjO14U].mp4
5.3G ./S00E1929 - Subnautica_pt._3_-_Save_Point_with_Becca_Scott [UxL8HYUrgig].mkv
6.2G ./S00E1919 - Subnautica_pt._2_-_Save_Point_with_Becca_Scott [I-ckh6CzuiQ].mkv
511G .
This led me to discover that some of my files are actually bigger on disk than they're reporting. I thought at first it might just be small files taking up a whole block but I found large files doing the same thing. I thought the compression was supposed to abort if it wasn't able to compress a file? So how did I end up with files bigger than the original? Am I missing something?
This is a dataset with 1M block size and lz4 compression.
truenas% du -Aha | sort -h | tail
3.6G ./S00E1075 - In_The_Next_Life_FOREVERVERSE_WEREWOLF [HHWVCyXJDUA].mp4
4.0G ./S00E1937 - Biomutant_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [uJJEkGhEhOc].mp4
4.2G ./S00E1926 - Minecraft_Part_3_-_Save_Point_with_Becca_Scott [eXTje8a8DsA].mp4
4.2G ./S00E1941 - Biomutant_Pt._2_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [QgUr08nAF34].mp4
4.4G ./S00E1916 - Jackbox_Games_Pt._2_-_Save_Point_with_Becca_Scott [L4haddIQBeo].mkv
4.5G ./S00E1923 - Fall_Guys_-_Ultimate_Knockout_-_Save_Point_with_Becca_Scott [JDEvGZjO14U].mp4
4.7G ./S00E1942 - Biomutant_Pt._3_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [0u0EwgmvAco].mp4
4.9G ./S00E1929 - Subnautica_pt._3_-_Save_Point_with_Becca_Scott [UxL8HYUrgig].mkv
5.7G ./S00E1919 - Subnautica_pt._2_-_Save_Point_with_Becca_Scott [I-ckh6CzuiQ].mkv
506G .
truenas% du -ha | sort -h | tail
3.6G ./S00E1075 - In_The_Next_Life_FOREVERVERSE_WEREWOLF [HHWVCyXJDUA].mp4
4.0G ./S00E1937 - Biomutant_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [uJJEkGhEhOc].mp4
4.2G ./S00E1941 - Biomutant_Pt._2_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [QgUr08nAF34].mp4
4.7G ./S00E1926 - Minecraft_Part_3_-_Save_Point_with_Becca_Scott [eXTje8a8DsA].mp4
4.7G ./S00E1942 - Biomutant_Pt._3_-_Save_Point_w_Becca_Scott_Gameplay_and_Funny_Moments [0u0EwgmvAco].mp4
4.9G ./S00E1916 - Jackbox_Games_Pt._2_-_Save_Point_with_Becca_Scott [L4haddIQBeo].mkv
5.0G ./S00E1923 - Fall_Guys_-_Ultimate_Knockout_-_Save_Point_with_Becca_Scott [JDEvGZjO14U].mp4
5.3G ./S00E1929 - Subnautica_pt._3_-_Save_Point_with_Becca_Scott [UxL8HYUrgig].mkv
6.2G ./S00E1919 - Subnautica_pt._2_-_Save_Point_with_Becca_Scott [I-ckh6CzuiQ].mkv
511G .