My FIRST TIME IN TrueNAS (HELP!!)

[Samuel Tai]

@Samuel Tai Maybe you're right.
But in reality I'm not looking for very high performance (also because my network currently doesn't allow me... gigabit). Most of all I wanted a good compromise between security/throughput/speed (more throughput) and since I move very often small files, I wanted to somehow speed them up...

As you've heard, the best bang for the buck in speeding things up is to throw more RAM at TrueNAS, as ZFS needs RAM to really stretch its legs.

 

[jgreco]

@jgreco so, is it useless to add this SSD as cache?

Well, if you only have 16GB of RAM and you add a 250GB L2ARC, you have about a 16:1 ratio of L2ARC:ARC, which is far beyond the 5:1 advised starting point, and even a good bit past the upper advisable limit of 10:1. As I noted previously, the ARC really needs to be able to classify cache entries, which means that it needs to be able to count the number of times a cached block is accessed, in order to understand which blocks are valuable from a caching perspective. If this doesn't properly happen, then ZFS evicts effectively random blocks out to the L2ARC, which is a waste of resources, and can actually be detrimental on performance.

On the other hand, an L2ARC device can be safely removed from a pool if it turns out to be useless.

 

[jgreco]

In my use case, I do see a small benefit to L2ARC on 16 GB RAM when enumerating directories containing >= 1000 photos over SMB. Without L2ARC, thumbnail generation is noticeably slower than with L2ARC.

Also, @Constantin has a nice summary of how to configure your system for persistent L2ARC for metadata only.

Use cases for L2ARC

@Constantin Thanks for that. The 10Gb connection may be key to your use case here. The video that @Chris Moore linked also speaks of maxing out a 10Gb link. In my experience, the benefit of L2ARC is use-specific and the connection speed is just one factor. For example, when I look at rsync...

www.truenas.com

This can make small file accesses or traversal operations substantially faster.

 

[ChrisRJ]

To rephrase what has been said a couple of times: ZFS works differently than other file systems. It was developed in the early 2000s (SSDs not even in sight yet) for high-end enterprise storage systems. Such systems at the time had (and still have today) a typical starting price beyond 100k USD, and quite often would reach 7 digits. In such an environment different rules apply, relative to what people are doing now for a homelab or personal NAS. That is basically the reason why extending a RAIDZ? vdev with a single disk is a non-issue there. If you need more capacity you simply add e.g. another 8 or 16 or 32 disks in one go.

Another such aspect is that you cannot simply throw in a consumer-grade SSD and expect higher speed. As the saying goes "With great power comes great responsibility". So you need to understand your workload and how to configure the NAS for it.

As to performance: I am running a plain RAIDZ2 of HDDs for archive and backup purposes (no SSDs) and have been doing so for more than 10 years. I do not remember a single situation where performance was a real issue. My strong recommendation to @vincentj is the same that @Samuel Tai already pointed out: Do not make things too complicated at the beginning, but start with something solid. 16 GB of RAM and two mirrors or RAIDZ1 are a good beginning.

Rather than spending too much time on premature optimization (because that is what we are talking about here), look at how you want to back up your NAS. Yes, you do need backups external to the NAS! Otherwise you would be working with a technology, that you have no experience with, and have an increased risk of loosing data. After all, the biggest risk for data loss is still human error.

 

[Arwen]

All you folks are mistyping "16GB" instead of "64GB" in this thread. As a general rule of thumb, the ARC needs a substantial amount of elbow room to categorize cache entries by MFU/MRU, and the general consensus of those who have experimented with it is that this usually doesn't become meaningful on a real workload until about 64GB of ARC. Additionally, since L2ARC consumes some ARC for the pointer records, if you want to add a 250GB L2ARC device, you really do want 64GB of ARC. There are some special cases where this might not be true, such as highly cacheable workloads or metadata-only L2ARC.

I was quoting the size of memory that the user stated. But, your point about more memory is valid. (I feel picked on, I tell you, abused, I call fowl! Ah, taking a drink from my wild turkey calmed me down... back to your regularly scheduled programming.)

 

[gianfelicevincenzo]

@ChrisRJ well, of course I know.. I have "recycled" hardware that I already had... so I'm far from something done "properly" (in the future, work permitting, I intend to buy suitable hardware...but for the now I have to make do with what I have). As I said before, I'm not really looking for stratospheric performance! I just use SAMBA a lot (both reading and writing with thousands of mostly small/medium sized files) and was looking for a way to speed up the read/write performance.

@jgreco I also set L2ARC and of course there is a much smaller cache filling and a further "lightning" in the ram when I read thousands of files from a folder. arc_summary I've heard of it, but personally I can't read it...I see many percentages and values...but what are the reference points? In any case, i attach the dump of the command in a file so maybe you can take a look at it too (I did a lot of tests by copying and reading many GB of files):​

One last question that comes to mind... these days I couldn't help but notice that during each "large" copy of folders, the ZFS cache value inexorably rose hand in hand. Now, even if I added 64GB of RAM, sooner or later I'd fill those too... so, what's the point of increasing the RAM so much if, after each copy of several GB, you're still back to square one again?
Example: if I had to move 500GB of data (having 128GB of ram in the NAS) after a few minutes I would fill them all! So I still have performance drops... or am I wrong?

 

[ChrisRJ]

One last question that comes to mind... these days I couldn't help but notice that during each "large" copy of folders, the ZFS cache value inexorably rose hand in hand. Now, even if I added 64GB of RAM, sooner or later I'd fill those too... so, what's the point of increasing the RAM so much if, after each copy of several GB, you're still back to square one again?​

That means that the available RAM is being used as cache. Think about it by mentally turning things around: If the value for "ZFS cache used" did not rise, it would mean that you had added the RAM in vain. If another process needs RAM, the ZFS cache will be reduced dynamically.

 

[jgreco]

personally I can't read it...I see many percentages and values...but what are the reference points?

Take it slowly.


ARC size (current): 81.5 % 12.1 GiB
Target size (adaptive): 82.2 % 12.2 GiB

So you have a 12GB ARC.

ARC total accesses (hits + misses): 93.9M
Cache hit ratio: 99.8 % 93.8M
Cache miss ratio: 0.2 % 144.5k

And a 99.8% hit rate inside that.

Cache hits by data type:
Demand data: 0.4 % 383.4k
Demand prefetch data: < 0.1 % 8.5k
Demand metadata: 99.5 % 93.3M

99,5% of it appears to be cached metadata

L2ARC breakdown: 136.9k
Hit ratio: 2.9 % 4.0k
Miss ratio: 97.1 % 132.9k
Feeds: 35.0k

L2ARC is 97% misses, which isn't quite as terrible as one might think. This would probably warm up further under regular usage.

 

[Arwen]

...​

One last question that comes to mind... these days I couldn't help but notice that during each "large" copy of folders, the ZFS cache value inexorably rose hand in hand. Now, even if I added 64GB of RAM, sooner or later I'd fill those too... so, what's the point of increasing the RAM so much if, after each copy of several GB, you're still back to square one again?
Example: if I had to move 500GB of data (having 128GB of ram in the NAS) after a few minutes I would fill them all! So I still have performance drops... or am I wrong?

Sun Microsystems, (inventor of ZFS), took the approach that unused RAM was a wasted resource. Thus, using it as a file system cache to speed up reads made perfect sense.

As @ChrisRJ said, any need by a program for memory will take priority over ZFS. Meaning ZFS will take the least used ARC entries and free them up until enough free memory exists to satisfy the program's memory request.

It is somewhat like having a multi-core CPU, where you get panicky if all the cores are used. It only maters if they are all used AND you can't get any work done. But, if they are doing back ground tasks, like a ZFS pool scrub, and the normal use of the computer is still functional, (NAS in our case), then it does not matter as much. (Of course reduced power consumption and having a bit of unused capacity is always nice.)

 

[gianfelicevincenzo]

@jgreco, I have new data recorded in these 3/4 days. I also tested by restarting the NAS and opening a few random files (with empty cache). These are the data:

What value should I check to see if the L2ARC cache is actually useful? And what is the optimal value?

Thank you

@Arwen, thanks... all very clear.

 

[gianfelicevincenzo]

I have a 2TB Toshiba HDD (external) in pretty good condition. How could I use it? My idea was to use this external hdd as disk parity and then switch to Raidz-2. It's a good idea? because I don't like to see stuff lying around unused. And if I can't use it for this purpose, how could I use it?

 

[Davvo]

It's a good idea?

USB is generally not a good idea.

 

[Arwen]

The assumption we would make on "external" is USB attached. That is not a good idea for pool member disks:
Why you should avoid USB attached drives for data pool disks
If you can shuck the drive out of it's enclosure, and install it via SATA, then that would be okay. But that means you need another internal disk bay and SATA port. (Plus, the power from the power supply to power the disk...)


On the other hand, using an external / USB attached for backup of critical files is another possibility.

 

[gianfelicevincenzo]

The assumption we would make on "external" is USB attached. That is not a good idea for pool member disks:
Why you should avoid USB attached drives for data pool disks
If you can shuck the drive out of it's enclosure, and install it via SATA, then that would be okay. But that means you need another internal disk bay and SATA port. (Plus, the power from the power supply to power the disk...)


On the other hand, using an external / USB attached for backup of critical files is another possibility.

I ask you a question, since LUCKYLY it didn't happen to me. What happens to the pool if a parity disk (in this case an external HDD connected via USB in a raidz-2) is suddenly disconnected?

So what could I do with this 2TB HDD? Right now it's gathering "dust" and I honestly don't like this thing

 

[jgreco]

I ask you a question, since LUCKYLY it didn't happen to me. What happens to the pool if a parity disk (in this case an external HDD connected via USB in a raidz-2) is suddenly disconnected?

There is no such thing as a "parity disk." Parity is stored for each block in whatever sectors happen to be adjacent. This will end up being more or less evenly distributed amongst all the disks in the vdev. RAID levels are sometimes incorrectly explained to newcomers in a way that includes a "parity disk", but even in classic RAID5, parity is not stored on a single disk. This generates an I/O hotspot that is undesirable, so parity is never stored on a single disk these days.

If you disconnect a component of a RAIDZ2 vdev, the vdev will become degraded and the vdev may need to pull additional data from the other disks in order to fulfill block read requests. If you disconnect a second component, you lose redundancy and can no longer tolerate disk errors on the remaining disks. If you disconnect a third, you're toast.

 

[gianfelicevincenzo]

so, in my case if i disconnect a second HDD (having a raidz-1) i lose all the data?
(Note: by "disconnection" I mean if I accidentally unpower a disk and start only 3 instead of 4)

 

[Arwen]

ZFS RAID-Zx levels tell you how much redundancy you have for data.

1. RAID-Z1 means you can loose 1 disk without any data loss
2. RAID-Z2 means you can loose 2 disks without any data loss
3. RAID-Z3 means you can loose 3 disks without any data loss

Any disks lost beyond that, and your pool is toast, as @jgreco said. If you just loose blocks of used data on additional disks, then only the affected files will need to be restored.

ZFS does have some tricks, like "copies=2", which can help even beyond RAID-Zx levels. But, if you have to ask about "copies=2", (or "copies=3"), then you probably need to learn more about ZFS first, before asking.

 

[jgreco]

ZFS does have some tricks, like "copies=2", which can help even beyond RAID-Zx levels. But, if you have to ask about "copies=2", (or "copies=3"), then you probably need to learn more about ZFS first, before asking.

copies=N is not really relevant. It refers to storing entire additional copies of ZFS blocks of data, which can help recover certain kinds of faults. However, this is not all that helpful to RAIDZ. If you had, for example, a RAIDZ1 12-device vdev, and stored a 1MB block of data, ZFS takes the parity of the first stripe (9 4K sectors), writes that, and then takes the data of the stripe and writes that. I don't have a custom picture of this, but for reference, here's how RAIDZ data gets stored:

So when you lose an entire drive, you effectively lose a column out of this table, which results in a loss of sectors in every large stripe. Multiple copies would be similiar to the light purple block starting at 4/A and blue starting at 9/D, if we were to assume these were the same size. Now if you lose disk A, you lose sectors 4-7/A and 10-12/A, meaning that you have lost P0-P3 in the first copy and D3-D? in the second copy. In theory, you could "merge" the contents of the block copies as a form of data protection, but I don't believe that this is done. This prevents copies= from being helpful when you lose two disks in a RAIDZ1 vdev. I expect this wasn't done in part because it would make things really weird for smaller blocks (such as a 4KB block) because you can't reliably protect those with copies= and relying upon a "merge the copies" strategy. The allocator would need to make sure that extra copies= started at different columns.

 

[Ericloewe]

(Note: by "disconnection" I mean if I accidentally unpower a disk and start only 3 instead of 4)

That's actually very recoverable. Plug in the missing disks and you should be good to go with a minimum of fuss. Of course, no pool for you until you do so, but it's just temporary in your scenario.
This is the sort of thing best left as an academic discussion, though.

 

[Arwen]

@jgreco - Good point, "copies=2", (or 3), would only help on RAID-Z1, if their were 2 or more RAID-Z1 vDevs. ZFS is supposed to write the second copy to another vDev, if possible. Thus, you loose a disk in vDev 1 and a block on another disk in vDev 1, "copies=2" should then be able to read vDev 2 for the failed block.

Sometimes I get the 2 x Mirror vDev feature with "copies=2" confused when working with RAID-Zx vDevs.