Correct storage operation

[Stratagem]

Dear TrueNas community,

I have been lurking around the forums for a while and I have learned a lot from the users on this forum, I hope to be able to provide help to others as time passes.

TLDR ; How do you store and operate your data correctly, both short- and long-term?

I am currently building a TrueNas CORE machine to act as my main storage server, with a backup machine coming up very soon. This will be my first experience using TrueNas CORE and ZFS.

I have always stored my data in ZIP folders as to move the data as large files, rather than many small files, since the data transfers faster. With ZFS being able to provide compression, I am wondering whether or not I should store the data I don't access frequently in ZIP folders. The data I often access is placed in normal folders.

My understanding is that ZFS' LZ4 algorithm does the compression for you, hence I only transfer the files in their uncompressed formats. I am afraid by storing my data as ZIP, I am negating the effects and benefits of LZ4 compression ; as well as not getting bit-rot and error detection protection.
On the other hand, I have folders in excess of 150GB containing both small and large files ; and I want the fastest file transfers possible on the hardware I possess.

How would you recommend I store and operate my data? And how do you professionals do it, both for yourself and for clients?

 

[Samuel Tai]

It's much safer for your data to have ZFS handle the compression and metadata management for your individual files, instead of aggregating them into a large ZIP with only a single set of metadata.

 

[Etorix]

LZ4, and now Zstd, are designed to detect data which can actually be compressed; there's no harm in leaving compression enabled in ZFS, even if the data being fed is already compressed (zip, mp3, h.26x).
ZFS would detect, and repair whenever possible, bit rot in a ZIP archive. It's just a file. But it's just easier to feed individual files ("normal folders") and let ZFS do the compression. What's the point of saving some file transfer time on data which is accessed infrequently?

 

[Stratagem]

It's much safer for your data to have ZFS handle the compression and metadata management for your individual files, instead of aggregating them into a large ZIP with only a single set of metadata.

LZ4, and now Zstd, are designed to detect data which can actually be compressed; there's no harm in leaving compression enabled in ZFS, even if the data being fed is already compressed (zip, mp3, h.26x).
ZFS would detect, and repair whenever possible, bit rot in a ZIP archive. It's just a file. But it's just easier to feed individual files ("normal folders") and let ZFS do the compression. What's the point of saving some file transfer time on data which is accessed infrequently?

Thank you for taking the time to reply.

If I understand correctly, keeping the files in non-ZIP folders for ZFS to handle compression and metadata is the safest option.

What's the point of saving some file transfer time on data which is accessed infrequently?

Would a good example of handling frequent vs infrequent data be:
Main pool
VDEV #1 using LZ4 for frequently accessed data (hot storage)
VDEV #2 using ZSTD (or for infrequently accessed data (cold storage)

You are absolutely correct, I might be overthinking the actual transfer time on ZIP vs non-ZIP folders when they are accessed infrequently.

 

[Arwen]

@Stratagem - Read up on ZFS. Your use of VDEV should be replaced by "dataset", (aka file system). ZFS can have many datasets, with different properties, like different compression settings.

ZFS vDevs make up a pool, and you normally can't direct specific data to be written to one vDev and other data to another vDev. (ZFS did introduce special vDevs, which CAN have have special data handling, but that really is an advanced option that the beginner would not be using.)


Back to the original questions:

As others have said, ZFS will protect any file, ZIP or individual, as normal.

Using individual files does have the advantage of easy access, plus it's own metadata to describe it. Another reason can be re-organization into different folders / directories as time goes on, (without having to remove from one ZIP and put in another ZIP).

For different compression methods, choose which you want based on how fast it writes, and how much compression you can get. In my case, write times are less important, so I would choose the compression option with the highest amount.