Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!

[cyberjock]

Slideshow explaining VDev, zpool, ZIL and L2ARC and other newbie mistakes!

I've put together a Powerpoint presentation(and PDF) that gives some useful info for newbies to FreeNAS. I decided to create this slideshow because in the last 5 months I've been on this forum I've seen a lot of people confused about vdevs, zpools, zils, l2arcs, etc. Hopefully we can put to rest alot of the confusion once and for all.

We get a large amount of duplicate threads with the same questions being asked every other day. Personally, we get them so often I decided to stop answering them and decided a better use of my time would be to create this presentation. I literally read every thread and every post that goes on the forum. So if I don't answer either the answer is in the thread or the answer is found in this presentation or FreeNAS manual. Answering every 3rd thread with "Consult the manual" gets a little old after a while and I have better uses for my time.

This presentation also contains a lot of information that is explained in a little more detail for new users. It includes many common errors newbies make and can save you some heartache. If you are brand new to FreeBSD, I recommend reading the manual cover to cover. There are a lot of recommendations throughout the manual, and they are typically there because they are an error trap for many people.

I've saved this in Powerpoint because I have some animations in the slideshow. I'm not sure what other formats would work. If you would like this in another format that supports animations please let me know and I'll see what I can do. Currently I provide this in a powerpoint presentation and PDF. The PDF has no animations therefore the Powerpoint is preferred.

I'll try to keep it updated as necessary. If this presentation helps you please "like" it so others can see that it really is worth the time to read!​

A newer document, focused on ZFS, is also available in the Resources section:
https://www.ixsystems.com/community/resources/introduction-to-zfs.111/


Below are links to the PowerPoint and PDF versions of the presentation as of: May 2, 2016

 

[James]

Is it possible to generate a PDF for the non-animated portions for those of us who don't use Windows?

 

[cyberjock]

Is it possible to generate a PDF for the non-animated portions for those of us who don't use Windows?

Link has been added for PDF.

 

[bfishernc]

nicely done - confirmed everything I've learned so far (as a noob) :)

 

[paleoN]

First, let me say good job. I would also like to officially request a web-based version of this. You can even use animated gifs and no external program needed to view it.

Now, on to the critiques.

• VDevs are allocated in RAID formats such as RAID0(striped), RAIDZ and RAIDZ2

You left out mirrors.

• You cannot add more hard drives to a VDev once it is created.

Keep it. It's a very important point. Also, it's inaccurate in the one exception that you can turn a stripe into a mirror. Perhaps asterisk it?

• Failure of a ZIL will result in a loss of data.

I'd change this to a loss of the entire pool. Which is what will happen with ZFS v15, what we use now.

I'd add to the L2ARC section that using an L2ARC will decrease the amount of RAM available to the ARC cache. This is particularly noticeable with a large L2ARC.

Perhaps add the caveats that most people do not need a ZIL, L2ARC, or see a benefit from them and the single best increase in performance will come from maxing out their RAM. If they even need to do that.

 

[cyberjock]

I didn't leave out mirrors, at least not intentionally. I was simply providing a few examples.

I didn't know that you can turn a stripe into a mirror. I didn't know you could turn a VDev stripe into a mirror. I'll have to look into that. Can I find more information in the manual?

As for the failure of a ZIL, will it's failure cause the pool to go offline and be unrecoverable? Or is the data that is in the ZIL and not committed to the pool simply lost? I've never played with ZILs myself because I don't see a need for them in any of my circumstances.

Can you explain the cause/effect of the L2ARC changing the amount of RAM allocated to the ARC? I believe you, I've just never heard of this before and I'm very curious to know more about this phenomenon and the though process behind it.

I am trying to shy away from comments about what to recommend people. Each setup can potentially have different situations. I'd prefer to show what things you shouldn't do for data security than things you shouldn't do for performance reasons. Of course, I'm arguing this idea with myself because far too many people have first posts with comments like "I have 2GB of RAM and 30TB of storage and I get kernel panics all the time". My concern is that the presentation could suddenly include situations such as "recommend using an Intel NIC because they are so cheap, they work, and they have excellent performance". Maybe I should add a slide of other random tidbits of information for noobs at the end with comments like these.

 

[paleoN]

I didn't leave out mirrors, at least not intentionally. I was simply providing a few examples.

Just all the others. ;)

I didn't know that you can turn a stripe into a mirror. I didn't know you could turn a VDev stripe into a mirror. I'll have to look into that. Can I find more information in the manual?

From the zpool man page:

zpool attach [-f] pool device new_device

Attaches new_device to an existing zpool device. The existing
device cannot be part of a raidz configuration. If device is not
currently part of a mirrored configuration, device automatically
transforms into a two-way mirror of device and new_device. If
device is part of a two-way mirror, attaching new_device creates a
three-way mirror, and so on. In either case, new_device begins to
resilver immediately

And from the ZFS Administration Guide see Example 4–6.

As for the failure of a ZIL, will it's failure cause the pool to go offline and be unrecoverable?

The entire pool unrecoverable:

• There is a bug/feature with this version of ZFS that will cause you to lose your pool if your ZIL goes offline or gets detached.

Or is the data that is in the ZIL and not committed to the pool simply lost? I've never played with ZILs myself because I don't see a need for them in any of my circumstances.

This will be the expected behavior with ZFS v28 which will be in FreeNAS 8.3 as it's in FreeBSD 8.3.

Also, the ZIL is for sync writes only.

Can you explain the cause/effect of the L2ARC changing the amount of RAM allocated to the ARC? I believe you, I've just never heard of this before and I'm very curious to know more about this phenomenon and the though process behind it.

ZFS has to keep track of what's in the L2ARC tables which of necessity requires RAM. I believe it's all or almost all metadata which also means you may hit the metadata cache limit for ZFS depending on how much RAM & how large the L2ARC. I've read that it uses 25MB of RAM per 1GB of L2ARC, but I remember thinking that was high & the source wasn't the best. I will have to see if I can track down some decent references, as I'm no expert on the L2ARC, and maybe even bookmark them this time.

Also, the L2ARC is for random reads mostly static data, i.e. databases. You will see little to no benefit for streaming workloads.

I am trying to shy away from comments about what to recommend people. Each setup can potentially have different situations. I'd prefer to show what things you shouldn't do for data security than things you shouldn't do for performance reasons.

Understandable and you're right each setup is different. It's just the ZIL and L2ARC are specifically for performance. Most noobs seem to think, "I have a SSD. I'll just add it to the pool and it will be faster." Where in truth it depends on their workload and even then it's usually much cheaper to max the RAM on their motherboard than get an SSD. Which will also get them superior performance.

 

[paleoN]

Found a good source.

> Is there a guideline in which relation L2ARC size should be to RAM?

Approximately 200 bytes per record. I use the following example:
Suppose we use a Seagate LP 2 TByte disk for the L2ARC
+ Disk has 3,907,029,168 512 byte sectors, guaranteed
+ Workload uses 8 kByte fixed record size
RAM needed for arc_buf_hdr entries
+ Need = ~(3,907,029,168 - 9,232) * 200 / 16 = ~48 GBytes

Don't underestimate the RAM needed for large L2ARCs
-- richard

Now it's not nearly as bad as that. Most of us won't be using 2TB L2ARC and have only zvols or tuned the zfs record size to 8K.

A much more typical scenario using a 300 GB L2ARC:

So, depending on the data, you need somewhere between 440 MBytes and 111 GBytes to hold the L2ARC headers.

Once again you would never actually see the 111GB. For an 8K record size like above the L2ARC would use about 6.8GB of RAM. A 32K record size would drop it down to 1.7GB.

It's been my understanding/assumption that ZFS auto-tunes the L2ARC. Which would mean if you had a SSD that was "too big" or you had some crazy record sizes you would just never fully use it. The impression I have now is the "auto-tuning" is done when arc_meta_limit is hit. By default arc_meta_limit is 25% of arc_max. Using a L2ARC would fill this. What's more I believe there will be memory contention between ARC & L2ARC metadata. While the ARC would win most of it, the L2ARC would still use some.

 

[calgarychris]

Thanks for putting together the presentation, very helpful from this noobie's prespective!

 

[cyberjock]

Ok, I've added all of your recommendations Paleon and the files have been updated accordingly :).

Edit: I actually added lots of small things. I also made the comment that emulation using VMWare works great for experimenting if you don't want to dedicate hardware. Anyone know if Openbox works with FreeNAS for experimenting?

 

[Joshua Parker Ruehlig]

Thanks for doing this noobsauce. You might want to specifiy that the zil will cause pool failure now, but in the future it will not. Once FreeNAS is on ZFS Pool Version 19 or up the zil can be removed, or crash without loss of data. (Only 95% sure on this though)

 

[cyberjock]

You are correct. My intention is to keep it accurate. Right now it is accurate. When ZFS is upgraded, then I'll revise it accordingly. :)

 

[paleoN]

Ok, I've added all of your recommendations Paleon and the files have been updated accordingly :).

Looking good noobsauce80. :) It's paleoN :D by the way.

I quite like the Additional thoughts for noobs... section.

Now I don't mean to harp on this and I know I got this changed but...

• [size=-2]
  You cannot add more hard drives to a VDev once it is created. The only exception is converting a zpool from RAID0 to RAID1. Consult the FREENAS manual for more information.
  
  [/size]

It's lost some of it's impact. "You cannot add more hard drives to a VDev once it is created", is a vital and poorly understood point. Perhaps have the exception on a line below and a smaller font or something?

This didn't make the cut: ;)

Also, the L2ARC is for random reads mostly static data, i.e. databases. You will see little to no benefit for streaming workloads.

The L2ARC was specifically designed for small random reads. The 2nd part is also true without any tuning.

 

[cyberjock]

To be honest.. I actually considered not including the exception. Who really will use that one exception? It's hard to believe someone would create a RAID0, then convert it to a RAID1. For noobs, do we REALLY want to be giving out a bunch of exceptions? I mean.. I dunno. I'm very torn on it. It almost seems better to say you can't add more drives than to try to give this exception. I can see it now.. the forum filling up with people that thought that "while they converted it to a RAID1 they'd add a few more hard drives to make it bigger".

Part of me prefers 100% accuracy, but part of me wants to leave out this exception because it's kind of odd and I don't know who would actually use it. I could see maybe ZFS masters using this "feature", but noobs? No way. Personally, I just accept that you can't add more drives to a vdev and leave it at that. Although it might work, I'd prefer to not deal with this exception just in case it backfires. ;)

Edit: I'll incorporate the latest changes tomorrow. Thanks for all of the feedback so far paleo*N*. :) My darn spell checker/grammar checker keeps trying to fix it as a typo when I type the upper case. LOL.

 

[BryanOB]

noobsauce80, thank you for posting the slideshow. It did help me and I particularly liked the animations.

However, I bet when you started this project you had no idea that it might mushroom into something larger, because of requests like the following :)

Define the VDEV - e.g. Virtual Device.

There are two other important aspects that could help others (me too).

1) The relationship with hard drives having 4KB sectors, the ZFS 128KB stripe size and the number of drives in a VDEV.
re: http://forums.freenas.org/showthread.php?158-What-number-of-drives-are-allowed-in-a-RAIDZ-config

2) Page (18) of slideshow: The new VDEV can use a different size than the old VDEV. Please clairfy.
To me, it implied that an unbalanced Zpool was OK. E.g. A Zpool with......
VDEV1 with RAIDZ2 and 6 drives
VDEV2 with RAIDZ1 and 3 drives

Thanks,
Bryan

 

[cyberjock]

Ok. I've added ZFS striping info and updated Page 18 of the slideshow based on BryanOB and paleoN's feedback.

 

[cyberjock]

Updated slideshow again. Now states it is up to date with 8.2-RELEASE-p1.

Also added the following to slide 26:

-Always keep a backup of your FreeNAS configuration(section 2.5 of the FreeNAS manual). Especially before you update FreeNAS.

 

[BryanOB]

Thanks for the update. I will review the PPT file , and reply if I can contribute again.

WOW! version 8.2 is released? I didn't know that. Just a couple days ago I visited the download page - didn't notice it. OK! It was just released 21 hours ago.

Anyway I've just built my server and I'm in the process of configuring it. I certainly hope that they fixed the broken UPS service.
Anyway, I will try the upgrade process - thanks for reminding me to backup the configuration.
I'm not concerned about trashing any data, because I haven't reached that far yet. :)

Regards,
Bryan

 

[Stephens]

Good job. I found it surprising that in example 3 a HDD crash in VDEV1 would cause a loss of the whole ZSPOOL (including VDEV2), so that was nice to learn. I'm staying away from multiple VDEV's, though and going with multiple NAS's. Among other reasons, it gives me some hardware redundancy (like if a motherboard or power supply goes out on NAS1, I still have NAS2).

Reading the PPT, I found "Adding VDEVs" confusing. Specifically... "It is not recommended that you [...] NOT add..." I could be wrong, but I think this means to say "It's recommended that you [...] NOT add..." But the lesson pointed to is really important.

 

[cyberjock]

At Stephens request I have fixed the slide. Although I think it was grammatically correct, it could be confusing(especially to users that speak english as a second language). I also changed the virtual machines to VMWare ESXi and VirtualBox. VMWare Workstation 8.0.4 seems to not play very well with FreeNAS(at least for me). If someone else confirms it works fine I'll readd it and consider it a fluke on my end.

Edit: If an Admin could change the thread title to "Slideshow explaining VDev, zpool, ZIL and L2ARC and other newbie mistakes!" I'd appreciate it. I tried to change it, but either I can't find it or I don't have the rights to. Thanks!