Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!

[cyberjock]

The Google Docs link is now fixed.

 

[liukuohao]

@Cyberjock

I hate answering the same questions 3-5 times a week

Unfortunately that is fact of life when you are a Super Moderator.

To mitigate this problem, perhaps someone can start a sticky for
the noobs, like me:), to educate themselves the fundamentals of
FreeNAS. Fundamentals for those who don't have any background
knowledge of IT and computer science.

If someone, is asking this question that is well-answered in the
sticky, you can just request he/she to refer the sticky.

 

[cyberjock]

@Cyberjock



Unfortunately that is fact of life when you are a Super Moderator.

To mitigate this problem, perhaps someone can start a sticky for
the noobs, like me:), to educate themselves the fundamentals of
FreeNAS. Fundamentals for those who don't have any background
knowledge of IT and computer science.

If someone, is asking this question that is well-answered in the
sticky, you can just request he/she to refer the sticky.

We do have stickies... and a FAQ... and the manual... and my presentation. But the problem is many people don't read them, listen to them and take away most of the information. They read it and see stuff like "you should consider buying server grade parts" and they dismiss it as "yep.. I don't have time/money for that"(It's often cheaper and more energy efficient to buy server parts than desktop parts because of incompatible hardware). They have preconceived opinions of many of the recommendations and justify ignoring them for reasons that are often Windows-centric. The truth of the matter is that some of the more experienced moderators have made those stickies not because we want to see people spend as much money as possible but because the vast majority of users will get one or more of those talking points wrong(hence we created the sticky to save them time, money, and another forum thread explaining that Realtek NICs still suck as much as last week. And they'll get plenty upset when their shiny new mini-ITX $350 motherboard/CPU combo won't work with the onboard Realtek NIC and they have no PCIe slots to add an Intel NIC. I've been cursed at plenty of times because they spent money on hardware that wouldn't work sufficiently enough for FreeNAS and I told them that their NIC wasn't supported and that they spent $300+ on hardware that doesn't appear to work with FreeNAS.

Most people have years and years of experience with Windows. They have no idea that most of that knowledge is completely useless as soon as you "walk across the street" to FreeBSD. Most people just aren't prepared for the humbling that takes place when you go from "top dog" Window MCSE, A++, super-badass-awards-galore-on-my-cubicle-wall in Windows to being a can-someone-change-my-diaper "toddler" in FreeBSD. Most people just aren't willing to make that realization right from the start and end up realizing it after the fact and in the most painful way. Windows admins need to swallow their pride and realize they are clueless. The faster they realize this and stop ignoring the wealth of information available to them the faster they'll be on the right foot and doing things right. I've provided so much one-on-one support for data recovery because of ill-conceived FreeNAS builds that went horribly wrong that if I charged money I'd be rich. One of my friends told me that I should start charging $250 per incidence or something since it typically takes me more than one day over Skype and/or Teamviewer. Being a disabled vet I have lots of spare time on my hands. ;)

I actually put my guide together because I was one of those people that went from Windows/DOS only for more than 20 years to FreeBSD. I literally spent about a month doing lots and lots of reading and experimenting in a VM. I wasn't going to let FreeNAS/FreeBSD beat me and I wasn't going to trust it with my data until I was convinced that the FreeNAS developers were good at their job and didn't make decisions that put my data at risk and that I knew exactly what I was "buying into" when I switched from a Windows Server to a FreeNAS server. After all, if I suddenly can't mount my zpool I want to be able to fix the issue myself if its fixable. Expecting forum people to help me recover my data is somewhat unrealistic(not to mention nobody values my data as much as me).

One thing I'm good at is writing lectures and training presentations, so I put my guide together hoping to help out some people. I was really hoping there would be a guide like mine when I first started dabbling with FreeBSD and ZFS. I was completely clueless but I was determined to figure all this stuff out. I had so many questions and I didn't want to inundate the forum with more questions to validate the knowledge I had gotten. But at the same time I had no friends to ask for help either. I was on the island and I was very much alone at that point. I couldn't find a guide quite like mine that, so I made one.

My guide has helped a lot of people, and some of the repetitive questions have decreased in numbers. But now when I tell people to consult the manual, my guide, etc. they get all upset because they don't want to hear that. They've sometimes heard that answer at least twice and they want validation that they can do what they want and it'll still work out for the best. They also want the answer handed to them on a silver platter. And they want all future questions answered for them in the same fashion, regardless of how many they may want to ask. Not all users are this way, but plenty of them are.

So now the reality of it is now the forum is so full of repetitive questions that have the exact same question and answer over and over that when someone has a question and they try to search they end up with 500 questions that are all the exact same(I get emails from people because that is exactly their problem). They could spend hours trying to find their answer... or... they could just start a new thread(yipee!). Guess which one they do? So now the forum is more or less a Q&A for the noobs and moderators. Even I've had problems finding my own posts when I know exactly what keywords to use. There's just so many posts now on any given topic.

So for the most part I don't answer questions if I think there is a chance they could have found the answer on their own. Lately I read the questions carefully and try to determine how much effort the poster put into his question. If it looks like they are trying really hard I'll help them out. If I get the impression they've installed FreeNAS on a 2TB hard drive because they heard about FreeNAS 2 hours ago and can't figure out why the mouse won't work and then why they can't use the 1.9TB of free space on their boot drive, I ignore the thread(and sometimes the user). I do keep a list of questions that keep being asked in the forums and I do try to add those topics to my guide with each update. Unfortunately its a losing battle and many moderators have come and gone from the forum just in the 18 months I've been using FreeNAS. Even I've been at that point where I just gave up and left more than once. But I've decided that next time its for good. I spend quite a bit of time in the forums every day(probably 2+ hours or so). Since its all voluntary its a matter of how I choose to spend my time. Diablo 3 doesn't play itself you know? :P

I do have fun maintaining the guide though. It keeps my memory sharp on all of this stuff and makes me more useful to potential employers when I can put a bullet on my resume that I'm somewhat familar with FreeBSD/FreeNAS (and now linux thanks to the last few months). I'm just glad the guide has proved its usefulness. Through all the revisions I've put well over 100 hours into the guide and I'd hate to think that the time I've spent on the guide was for nothing. I get plenty of emails from people that have no interest in FreeNAS but love the ZFS information.

 

[liukuohao]

@Cyberjock

Ok noted, I understand you have put a lot effort in helping out
noobs in the forum and somehow the forum is asking you to
do more and more of "spoon feeding". And that is where you
can start to get frustrated. Hey! I understand, support a forum
user is no easy job when they ask a lot of silly questions (include me).

Also I did learned from the from the former FreeNAS forum (now is calle NAS4Free)
that Realtek on board network really sucks! Intel NICs are highly
recommended in running FreeNAS, that is why my motherboard
has a dual Intel NICs

 

[Richard Elling]

Good preso

Two things:
1. Top-level vdevs are striped. So when you have a single-disk vdev, it is at the same level as a raidz or mirror vdev. You can attach another disk to mirror it (a good thing), but there is no real concept of a striped or raid-0 vdev.

2. The lack of recovery tools is because if we can automate ZFS recovery, we will. The existence of fsck, ckdsk, and similar tools is because the developers of those file systems knew that they would be inconsistent and lose data, but also knew how to try and reconstruct the metadata -- data is often lost, but metadata can sometimes be recreated from backup copies on the disk or other heuristics. ZFS is a transactional object store, more like a database, where the integrity of the transaction commit can be checked. All metadata is redundant with copies distributed across vdevs or spatially within a vdev (for single-disk pools). So if a pool cannot recover itself with high confidence of correctness, it might be recoverable by an expert or by rolling back the transactions. Bottom line: if we can figure out a way to recover with an algorithm, we would code it into ZFS.
-- richard

 

[cyberjock]

1. You are correct. I recently changed the wording to more clearly explain that. The new 9.1 release will include my new presentation with the changes.

2. As for the lack of tools, there's lots of theories. I agree that your comment is a very solid reason to justify there being no tools. Of course, a bunch of noobies will get lost the second I say "metadata" and "transactional object store". So for simplicity I'm leaving it the way it is. The bottom line is it really doesn't matter why we have no tools. The point I was trying to make was to not expect to use some tools to recover your data after your zpool has failed.

Thanks for the feedback. I'm really surprised nobody had commented on the raid0-vdev thing before you. LOL.

 

[cyberjock]

Updated to support FreeNAS 9.1.0 that was released this morning!

 

[Callipygous]

Thank you for the time and energy you put into this presentation. An amazing resource.

 

[SmilingInSeattle]

Great presentation ... and some great comments, particularly, to me, those within 20 posts above this one. You did a great job of explaining vdevs to me. I'd started Googling vdevs just before you answered me in another forum. While the author of the first post was correct, he didn't do the job of making it appear"common sense" as you have done. coming from a career of training others how to get the most out of sensory devices (more than the manufacturer ever thought wheh the device was built by using techniques) kudos to your ability to present! I had to smile when I went through the "animations" as, being a linux user, I viewed the presentation with Libre Office's Impress Version 3.5.7.2 which still does not support animations. Fortunately, due to your presentation style, it was not hard to figure what the animations probably did (and I admit to having been impatient with some author's animations viewed with PowerPoint. :(

From my perspective, while I've built and maintain "lesser quality" hardware than you recommend, I take the "old fashioned" route of lots of offline backup sources, so if my hardware goes down my data is not lost, just time is lost. But I can afford that approach as, while I still work because I'm bored with too much time on my hands if I don't, I'm retired and the data I have is not "mission critical." So I think your presentation does what it is intended to do if the readers will regard themselves as risk takers with differing levels of risk tolerance and more importantly, take personal responsibility for their actions. What you've written does a great job of pointing out risks and readers should always remember that there is no "perfect" solution. Only babies understand "black or white," as we mature everything becomes more and more shades of grey. In my senior years now, I think my personal greyscale color chart is up to 2^2048 level of grey!

Thanks, once again for a really good presentation!

 

[cyberjock]

Yeah, I wish I could have made the Powerpoint into an executable or something that could be universally used across at least Linux and Windows. Of course, if I had gone with an executable then you have people that would swear its a virus, etc.

I thought about making the presentation into a youtube video, but I'm not sure of any tools that easily convert it.

 

[xxJeevesxx]

@cyberjock

Thank you for all your work in putting this together! It is greatly appreciated :). I am new to FreeNAS and FreeBSD and am trying to suck up as much information as fast as possible. I was able to successfully get a box up and running with FreeNAS 9.1.0 and have started to copy all of my data over. Little things here and there have come up but (as you have made it so clear) the forums have done a good job providing answers. My next task...jails lol. Again, I just thought I would say thank you!

 

[Colin Tisdall]

Hello cyberjock thanks for all your hard work.

We are about to embark on a new setup and I have a question about the amount of RAM for a combination Production file server and archive server. We are looking to create a 45 drive machine with 5 vdevs of 8 disks each(3TB). Each vdev will be RAID-Z2 so about 16.4 TB of usable space. 1 vdev will be for the production file server that will be used everyday and the other 4 vdevs will be for older archives and used rarely. Based on the basic 2GB of RAM for this we would need around 128GB of RAM. My question is because most of the server will be rarely used can we get away with less RAM?

Colin

 

[cyberjock]

You can. The thumbrule is a recommendation for a good place you are likely to end up. If you are building the server I'd buy hardware that is capable of using 128GB of RAM(or potentially more). And perhaps buy 16GB DIMMs. You will almost certainly need 64GB of RAM at that kind of size, with 96 or 128 being the point you will end up using. With bigger servers RAM quantity is most often the limiting factor for people trying to build a big server while not buying lots of expensive RAM.

 

[alpha]

Hey cyberjock, I read the PDF for noobs in addition to some of my previous knowledge, but I just have a few questions, that honestly probably have been discussed. I am planning on deploying a home NAS system with somewhere between 8 and 10 3 TB drives in RAIDZ2. They will be in a single vdev. I also plan on using encryption. Problem is, the system I plan on using is a core i3 with only 8 gigs of RAM. This nas will only have 2-3 users at any time accessing it, usually no more than one. The files are all large files that I will be streaming to my TV, or other places around my network. I don't necessarily need lightning fast reads or writes, but I also don't want this system to be to slow to even use.

Should 8 gigs of RAM be enough for such a use case? I know it isn't "recommended" but it should suffice, should it not? The i3 worries me a little as it doesn't have AES-NI, but worst case I can swap my htpc i5 which does have AES-NI into this system.

 

[cyberjock]

Doubtful with a pool of that size. You "might" be able to get by with "just" 16GB of RAM. I know from personal experience, if you don't have enough RAM you might not even be able to stream a single video! I couldn't. Upgraded from 12GB to 20GB, and back to rocket speeds.

Also, if you aren't going to use a CPU with AES-NI, expecting ANY kind of performance from an encrypted pool is asking for miracles.

But it sounds like you aren't going with ECC RAM if you are going to use an i5, which is another potential pitfall.

 

[alpha]

Doubtful with a pool of that size. You "might" be able to get by with "just" 16GB of RAM. I know from personal experience, if you don't have enough RAM you might not even be able to stream a single video! I couldn't. Upgraded from 12GB to 20GB, and back to rocket speeds.

Also, if you aren't going to use a CPU with AES-NI, expecting ANY kind of performance from an encrypted pool is asking for miracles.

But it sounds like you aren't going with ECC RAM if you are going to use an i5, which is another potential pitfall.

Yea, no ECC RAM. Just a normal PC. Maybe this isn't a good idea then :(. Do you think nas4free would perform fine with these specs? Or would that even be an issue. Right now I am using openmediavault but it has some issues and I am not a huge fan. But it works just fine speed wise, but it is also using ext4, not ZFS. Should I maybe consider freenas and then not use ZFS? I would like to keep the encryption, and if that means swapping some CPU's around that is fine by me. But I would rather not have to buy more RAM... I know RAM is "cheap", but I would rather not have to spend money if I don't have too.

Also, this isn't mission critical data. So ECC I don't think will be a problem. I don't want to loose my data, thus the RAID Z2 idea, but unless using normal RAM is actually a bad idea, I would stick with that. (I don't have other options anyways, don't have server components that will support ECC).

Any other suggestions? And thanks so much for your input!

 

[cyberjock]

NAS4Free still should use ECC RAM since you are using ZFS. If you aren't using ECC RAM and your RAM goes bad a scrub could destroy all the data on your pool. Everyone that has used non-ECC RAM and the RAM went bad suffered a loss of their entire pool. Unfortunately, because of how everything works, it'll trash your original copy of the data, which will trash the backups too. So if you want to gamble, go with non-ECC. It's rare for it to fail. But when it fails, you can expect serious and devastating consequences.

 

[alpha]

NAS4Free still should use ECC RAM since you are using ZFS. If you aren't using ECC RAM and your RAM goes bad a scrub could destroy all the data on your pool. Everyone that has used non-ECC RAM and the RAM went bad suffered a loss of their entire pool. Unfortunately, because of how everything works, it'll trash your original copy of the data, which will trash the backups too. So if you want to gamble, go with non-ECC. It's rare for it to fail. But when it fails, you can expect serious and devastating consequences.

Hmm. Guess I have some thinking too do. Would you expect nas4free to perform better at least, considering only having 8 gigs of ram? Also, should I be looking into other file systems? Maybe ZFS is not in my budget quite yet.

ZFS* not NFS.

 

[cyberjock]

Hmm. Guess I have some thinking too do. Would you expect nas4free to perform better at least, considering only having 8 gigs of ram? Also, should I be looking into other file systems? Maybe ZFS is not in my budget quite yet.

ZFS* not NFS.

NAS4Free doesn't cache the USB stick the same as FreeNAS. It also doesn't have the overhead of the GUI architecture like FreeNAS does. I can't make any guesses because I'm not a fan of NAS4Free. In my opinion, its less polished and I'm not about to start guessing. But I still wouldn't expect 8GB to work with a pool of that size. I'd still think that 16GB is about the lowest you could conceivably go. You're just asking for too much for too little RAM. Remember that the same ZFS code in FreeNAS is in NAS4Free, so you shouldn't expect significant differences in ZFS. Of course, alignment and other non-ZFS things can have a significant effect too.

 

[alpha]

Thanks for the help and info man! I think I am going to try and stick with openmediavault and use XFS as it seems it will not have the same overhead as ZFS (hopefully this is true...). One day when I have the money, I WILL be switching to freenas, with some stupid amount of harddrive space, and ALL OF THE RAM. Seriously though. I would like to build a box with at least 50 TB and 64 gigs of ECC RAM, but that will be easily a few years away. I will definitely be back looking for advice!