Mount non-zfs drive at boot (ext4, xfs, ntfs)

[makko]

Hello!

I have a truenas scale system with a few ZFS pools and some apps running.

I want to add some disks formatted in ext4 (or xfs). Those disk are currently formatted in ntfs, but on what I saw I cannot mount ntfs on truenas.
I'm fine in reformatting those drives in ext4 or xfs.

I don't want to use ZFS for those drives: they will only be used by one container and I do not care about some data being corrupted. If a file gets corrupted I will delete it and recreate it. On what I know ZFS will make all the data unavailable if a corruption is found.

What I want is a a way to mount those drives at boot (like I manually do). So when the app starts the drives are mounted and accessible by the app.

Please focus on the question: since the PC running truenas is already always on, I want to also use it for purposes different that just doing NAS, media player.
The reasons to have ext4 (or similar) are:
- If I change my mind I can just move the disks to another system (linux, mac, windows).
- As I already said the data ara not important: If one files is corrupted (the app will tell you) I will just delete it and recreate it. We are talking about 20 TB each drive.
- As far as I know ZFS will make invalid the whole disk even if just few bytes are corrupted, while ext4 will only invalidate the single file(s). And of course I do not want to make any redundant ZFS array: I do not want to trade disk space for data integrity. To make clear again: I can loose files and just recreate them.

Would be so nice if from the GUI you could just choose other filesystems!

Thanks!

 

[Ericloewe]

No, filesystems other than ZFS are not supported and likely will never be.

- If I change my mind I can just move the disks to another system (linux, mac, windows).

With ext4 or XFS? That's not going to work. ZFS, on the other hand, would allow you to do that.

As far as I know ZFS will make invalid the whole disk even if just few bytes are corrupte

No, that would be very hostile to the user, I don't know where you got that impression from, but it's wrong.

ext4 will only invalidate the single file(s

No, ext4 and other legacy filesystems will happily read back junk, since they have no way of knowing if what they're storing is intact.

 

[makko]

No, that would be very hostile to the user, I don't know where you got that impression from, but it's wrong.

So what will happen in a real case ?
Let's say I have a 20 TB drive with 200 files (about 100 GB per file). One bytes gets corrupted. One or two of those files will be corrupted...
What truenas will show me ? I could still use the other 199 files ? The corrupted file(s) will be deleted automatically or what ?
What I would like is some kind of error just telling me that file is corrupted, with the possibility to just delete it and copy another one as replacement.

As additional question, I tried for few disks to format them using ZFS (single disk) with all the default parameters (like the compression level). Based on the number of files I can store, it seems I'm loosing something like 200 GB... maybe is just wort to disable compression ? There is no actual compression you can do (like there will never be a block of zeros).

Since I do not care about loosing single files, but loosing all the files at once will be a problem since it takes a lot to create 20 TB of data (on just a drive), would make sense to create a stripped array ? What will happens if one drive is broken ? Will I be able to just replace the drive with a new one ? Again, ideally I would agree on loosing a file which was stored half on a disk and half on the broken one, but what about the files only stored on working disks ?
On many sources from forums and youtube it seems that loosing one disk on a stripped array will make you loose the whole array.

Right now I have all those disk on a linux pc and if one drive fails, I will just unplug it and buy a new drive (and recreate about 200 files). If on a disk I see one file marked as corrupted (the app will do some checks to tell if is valid or not) I will just delete it and recreate it (I have to say it never happened).
Will I be able to have the same degree of freedom with truenas ZFS ?
The plan is simply to reuse an already running pc instead of having 2...

 

[makko]

And in addition to my previous message: if I have a stripped array, the data will probably divided into the 2 disks. Meaning that If I loose a drive, I will loose all data (like raid 0). I guess there is no way to use the disks in a sort of sequential way (you fill one, then the next in the array). So probably only doable option is to have a vdev for each drive. Still remains unclear to me what happens when in a single disk vdev a corruption is found.

 

[Arwen]

To second @Ericloewe's comment, ZFS will return error only for corrupt blocks. Any good blocks or files will be returned properly.

Now it should be clear on another issue. ZFS by default keeps 2 copies of standard metadata, like directory entries. And 3 copies of critical metadata. Even on a single disk pool. The reason for this redundancy is that in the event of a bad block in a directory entry for example, another file system might loose the entire file. But, ZFS would not. It will automatically read the redundant metadata and both use it for the current need, AND fix the bad block, if a spare block is available.

This default redundancy for metadata & critical metadata should not take up too much extra space, when compared to actual data. (Plus, actual data can use automatic compression...) So it is a small price to pay for not loosing access to data.

Next, ZFS will notify the user if a bad block is detected, and can not be automatically fixed. Thus, the user can erase that file and restore from backups. But, all the other files remain available and intact.

The one rare condition where a non-redundant pool is more or less gone, generally occurs with multiple failures. For example, using hardware RAID controllers with ZFS, (highly discouraged). Or, ignoring the early warning signs of a disk failing and not doing anything. Or, a whole disk failure.


On occasion, someone does not need redundancy with ZFS.

I personally run a miniature computer as my media server. I take a small piece from each storage device and mirror it for the ZFS OS pool. Then stripe, (aka NO redundancy), for the ZFS media pool. Over the years I have lost blocks and had to restore video files, (videos are larger files, so more likely to have had bad blocks). Easy enough to restore on-line and return the media server to full function.

 

[Arwen]

To answer your question;

So what will happen in a real case ?
Let's say I have a 20 TB drive with 200 files (about 100 GB per file). One bytes gets corrupted. One or two of those files will be corrupted...
What truenas will show me ? I could still use the other 199 files ? The corrupted file(s) will be deleted automatically or what ?
What I would like is some kind of error just telling me that file is corrupted, with the possibility to just delete it and copy another one as replacement.

Yes, you can use the 199 other files. The corrupted file(s) would NOT be deleted automatically. You manually do so as part of either restoring the file. Or simply getting rid of the error.

The output of zpool status would tell you which file was affected.

It is important to schedule monthly, (or more often), ZFS pool scrubs to detect bit rot, (aka unknown), bad blocks. Normal reads of a file can also detect bad blocks. But, how likely is it that you would read every file block, every month? (And you can never manual read redundant metadata. Only a chance read, or actual scrub would detect such problems.)


As for;

And in addition to my previous message: if I have a stripped array, the data will probably divided into the 2 disks. Meaning that If I loose a drive, I will loose all data (like raid 0). I guess there is no way to use the disks in a sort of sequential way (you fill one, then the next in the array). So probably only doable option is to have a vdev for each drive. Still remains unclear to me what happens when in a single disk vdev a corruption is found.

Yes, if you have 2 disks in a striped pool, (similar to RAID-0), files are split between the disks. So total loss of 1 disk, means loss of all data and the entire pool.

No, their is no way to have 2 disks in a pool, but have files prefer a single disk for writing. That would be a nice feature for exactly this use case, but does not exist.

A vDev is a Virtual Device in a POOL. So, a 2 disk striped pool automatically has 2 vDevs, 1 per disk. What I think you mean is 2 ZFS pools, each with 1 disk / vDev. But, it then requires manual balancing of the files.

As for a single disk pool that has corruption, my prior response should be clear. Only the affected file has a piece missing. The rest of the files should be file.


ZFS and TrueNAS are not the end all for NAS. They both have limitations that some people can't work with or around. So we will not be offended if you decide TrueNAS is not for you.

 

[Ericloewe]

I could still use the other 199 files ?

Yes.

The corrupted file(s) will be deleted automatically or what ?

No, ZFS will error out instead of returning the bad data, but it will still be present (unless you delete the file or something like that). You could probably manually retrieve the bad data with zdb, but we're talking pretty extreme measures. For your use case, delete the file and move on with life.

As additional question, I tried for few disks to format them using ZFS (single disk) with all the default parameters (like the compression level). Based on the number of files I can store, it seems I'm loosing something like 200 GB... maybe is just wort to disable compression ? There is no actual compression you can do (like there will never be a block of zeros).

Are we talking about large media files (pictures, videos, audio, etc.)?

Since I do not care about loosing single files, but loosing all the files at once will be a problem since it takes a lot to create 20 TB of data (on just a drive), would make sense to create a stripped array ? What will happens if one drive is broken ? Will I be able to just replace the drive with a new one ? Again, ideally I would agree on loosing a file which was stored half on a disk and half on the broken one, but what about the files only stored on working disks ?
On many sources from forums and youtube it seems that loosing one disk on a stripped array will make you loose the whole array.

Yes and no. In theory, you can recover anything that's entirely present on the disk that's still alive (either because it happened to all end up there, you used copies=2/3, or whatever). In practical terms, this is such an unlikely scenario that you can safely ignore it.
Of course, if a disk doesn't fail completely, you can replace it and deal with whatever errors remain.

Will I be able to have the same degree of freedom with truenas ZFS ?

Strictly speaking, yes, though I question the use of the term "freedom". To me, "freedom" implies not having to worry about any of this and knowing that the data is safe. ZFS isn't going to be any worse than $LEGACY_FS in that regard, but the workflow screams "pain" and is not something I would tolerate much less recommend to anyone.

 

[makko]

Thanks all for your feedbacks.

Based on what you said the best thing to do seems to have each disk assigned on a ZFS pool (1 disk, 1 vdev, 1 pool). Still not sure about the compression.... I may try 2 identical disks where in one I enable compression and then compare how many files I can store (all files have nearly identical dimension). If one files will be detected corrupted I will just replace it without anything to do for the other files.

To reply to @Ericloewe, the files i'm storing are about 100 GB each and are binary files (nothing to do with multimedia files). You may think as a sort of database (even if are not) with some proprietary format. Because of the format any general purpose algorithm to compress them cannot do anything to reduce size (this why I'm thinking about disabling compression).

Anyway, thanks for the useful explanations !

 

[Ericloewe]

Give LZ4 a try and just disable it if it doesn't compress.

 

[winnielinnie]

(this why I'm thinking about disabling compression).

I would leave any type of inline compression enabled, even for datasets with non-compressible files.

Compression (of any kind) will remove the extraneous "padding" on the last block of a file. This is especially important for datasets with a larger recordsize.


Example:
You have a dataset with a recordsize of 1 MiB. You save a non-compressible file that is 1.1 MiB in size.

Without ZFS compression, this will be stored on two blocks, each one physically consuming 1 MiB on the disk, for a total of 2 MiB consumed.

With ZFS compression, this will be stored on two blocks. The first block consumes 1 MiB, and the second block consumes 0.1 MiB, for a total of 1.1 MiB consumed.


EDIT: Combined with the fact that LZ4 is blazing fast, and has "early abort", and now even ZSTD has some sort of early abort implemented (with OpenZFS 2.2+), the "costs" of inline compression is negligible, and I'd even argue that you won't notice any difference in performance.


EDIT 2: In your case of the "large binary files", you still might be able to squeeze some blocks somewhere in the file, and you might later start saving different files on the dataset that will benefit from inline compression (for multiple reasons). LZ4 + early abort maintains high performance, especially if your storage medium is a spinning HDD.

 

[makko]

On a similar topic, for storing 100 GB files does it make sense to choose block size of 16 MB in the dataset ? The rational would be that having such huge files it is more more performing to choose a big block size, while wasting 16 MB for the last block of the file is nothing when compared to 100 GB file.
Maybe the only problem could be that I'm not fully sure when accessing the file how big it will be the data access... I guess truenas will read the whole block containing the byte(s) you ask to read.

 

[winnielinnie]

On a similar topic, for storing 100 GB files does it make sense to choose block size of 16 MB in the dataset ?

That seems like *overkill. Besides, I don't believe the TrueNAS GUI even allows setting a recordsize larger than 1 MiB.

* Overkill, because as far as I understand with ZFS: The smallest read into RAM is a "block" itself, which means that simply reading a megabyte here, some kilobytes there (from random spots within the entirety of the file) will each require loading 16 MiB from the storage medium. If these files happen to only ever be read sequentially (or in massive chunks at a time) then perhaps the issue is moot.

I wouldn't do it (yet), until you read of others who report pragmatic success and positive experiences with it. This is one of those "wait and see" options. What little gains a 16 MiB recordsize might offer may not be worth it for the unforeseen costs and potential complications.

 

[Ericloewe]

The question really is "how will this file be read from and written to" and if the answer is anything other than "extremely sequentially" you will not want to use giant blocks.

 

[winnielinnie]

you will not want to use giant blocks.

What @Ericloewe is basically saying is that you should consider the saner 8 MiB recordsize instead of going extreme with 16.

* runs away *