Newbie Seeking Drive Setup/Pool Layout Advice

[mikestecker]

Sorry for the double post but I'm not sure if I posted in the wrong forum earlier, so re-posting here to maybe make some more traction.

I'm in the process of setting up my new FreeNAS build, here are the current specs for reference:

• Supermicro 4U Server Chassis BPN-SAS2-846EL1
• SuperMicro X9DRL-iF Motherboard
• Dual Intel Xeon 2620's
• 96gb ram (6x 16gb ECC RDIMM)
• LSI SAS9201-8i 6Gbps Raid Controller Card
• 47C8210 Quad-Port GbE Ethernet Adapter
• 2x 250gb WD Blue SSD (Boot + Boot Mirror)
• 1x ADATA SU800 256GB SSD
• 1x Samsung 860 QVO 1TB SSD
• 12x 4TB HGST HDD's
• 2x 4TB WD Red HDD's
• 4x 8TB WD Red HDD's (to be added later)

At the moment of writing this, I have everything setup and installed on the 2x 250gb WD SSD's listed above, but nothing else. All other SSD's and HDD's (except the 4x 8TB's listed above) are installed in the system, but no pools have been setup yet.

I'm seeking some advice on how to structure the pools to balance speed/efficiency and safety so that to minimize the chance of losing data in the event of failure. I've skimmed over many of the articles/URL's for beginners and while I understand how to actually set up a pool and all that, I haven't been able to get a took takeaway as to what the "right" way to do it is. How many pools, datasets, zvols or whatnot I need to setup. One thing I do understand however is that I should probably use RAIDZ2 for parity, but I could be wrong.

Here's what I'll be using this for:

• Plex -- I'm upgrading from a QNAP 4-bay with 21TB of data on it. The data is currently on the 4x 8TB's listed above. Once the 21TB of data transfers to this new FreeNAS build, I'll remove these drives from the QNAP and repurpose them in the FreeNAS build. I have a mix of 1080p content (10-25gb movies), 4K (for direct stream on local network only), and TV Shows, music, etc.
• Mac Time Machine Backups
• Docker VM's (linux) for web development/testing (NO live websites, just for in-house development/testing)
• Windows VM's (one or two) to use for browser testing
• Video editing storage -- I'm thinking about adding a 10GBe card to direct connect to my iMac Pro with 10GBe
• Photography storage (raw files, Lightroom library)
• NextCloud storage (to offset Dropbox costs)
• GitLab CE (want to setup internal CI process for above Docker VM's used in web dev
• Sonarr, Radarr, Lidarr, NZBGet, Hydra2, Home Assistant, Traefik, etc -- all in Docker.

Given the above mentioned, I'd appreciate any tips/tricks/suggestions! Thanks in advance!

 

[JaimieV]

Normal file storage, VMs running on-FreeNAS, 10gigE ethernet. Ok. Fairly similar to my own home setup so here's my thoughts, ready for anyone else to rip into :)

With the 14 4TB drives you can get up to 10gigE speeds - see my spec for Sisyphus on the button below, I get 860meg/sec write and 680meg/sec read from my iMac Pro from three RAIDZ1 vdevs in a pool, makes it quite snappy. So group yours into at least three vdevs, maybe four - play around, use RAIDZ2s or RAIDZ1s with a hot spare may be sensible.

If you're not going 10gigE you don't need to worry so much - one fat RAIDZ3 or a pair of RAIDZ2 vdevs in one pool will be entirely sufficient.

You won't need a ZIL with that workload, nor an L2ARC with that RAM (unless your VMs/Dockers eat 60+gig). Plus with the 10gigE pool config above your pool will be quicker than a single SATA SSD anyway!

You probably should put your VMs/Dockers on an SSD mirror pair pool for the IOPS, or it may not be needed if your HDD pool works out quick enough.

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files...

www.ixsystems.com

If you go SSD pool, you would do well to pick up a pair of cheapo ebay Intel 60gig SSDs for your boot pool and repurpose that pair of WD Blues for a VM pool - 250gig is overkill for a boot pool, you'll use ~20gig max and can't re-use the spare gigs for anything except stats history.

Once you move the 8TBs over, you can choose to either hive them off as a separate pool RAIDZ1 or Z2 and copy all of that back to them, or add them to the main pool as a RAIDZ1 or Z2. Or, if you don't have any other solution, you might want to consider setting them up as a RAIDZ1 or non-redundant pool of their own and using them to backup the important parts of everything else...?

 

[mikestecker]

With the 14 4TB drives you can get up to 10gigE speeds - see my spec for Sisyphus on the button below, I get 860meg/sec write and 680meg/sec read from my iMac Pro from three RAIDZ1 vdevs in a pool, makes it quite snappy. So group yours into at least three vdevs, maybe four - play around, use RAIDZ2s or RAIDZ1s with a hot spare may be sensible.

Thanks for the reply!! So what's the secret to balancing speed and redundancy? RaidZ1 and multiple Vdevs?

If you go SSD pool, you would do well to pick up a pair of cheapo ebay Intel 60gig SSDs for your boot pool and repurpose that pair of WD Blues for a VM pool - 250gig is overkill for a boot pool, you'll use ~20gig max and can't re-use the spare gigs for anything except stats history.

I know the 250gb is overkill, they were the smallest I could find on Amazon for the price. I'll try looking around before "locking" this setup down to see if I can't find smaller drives and repurpose these 250gb drives for VMs.

 

[Heracles]

So what's the secret to balancing speed and redundancy? RaidZ1 and multiple Vdevs?

Consider that a pool will give you the performance of as many drives as you have vDevs. A pool with a single vDev will give you the performance of a single drive. A pool with 4 vDevs will give you the performance of 4 drives.

Your plan mentions that you expect to start with 14 drives and go up to 18. 14 and 4 only share a single factor : 2. And with 2 drives, you can do only one thing : mirrors.
It is really not recommended to mix and match different vDev types in a single pool (don't make a pool out of a mirror and a Raid-Z2).
Should you start your pool with 7 drives Raid-Z2 vDevs, you will not be able to add 4 drives using the same logic. Only mirror will keep your vDev on the same format. Also, that is the config that will give you the more vDevs, so the highest speed for most cases including VMs.

I would not do Raid-Z1 because whenever a disk fails, all of the remaining drives must be perfect up to the end of the re-silvering process. Should a single error happens, it may corrupt the entire pool and you may loose it all. With a degraded 4 drives Raid-Z1 vDev, you need 3 drives to be perfect for you to survive. With mirrors, you need only 1. Math is easy to do : 4 drives Raid-Z1 vDev is 3 time higher risk than mirror. That is significant!

Another thing : you can not convert a mirror to a Raid-Z2. You would need to empty the mirror, destroy it and create a brand new vDev. So should you wish to end up with a specific vDev format, you must do it from the beginning.

Mirrors are also easier to auto-expand because you need to replace only 2 drives.

At the end, I would really go with mirrors here.

 

[mikestecker]

Mirrors are also easier to auto-expand because you need to replace only 2 drives.
At the end, I would really go with mirrors here.

But with mirrors, I lose half my usable space right? I'm leaning towards doing 2x RaidZ2 vdev's with 6 drives each and then use 2 spares.

vdev (4+2) = 16tb usable
vdev (4+2) = 16tb usable
= 32tb usable

What are thoughts on going this route? Then maybe I could maybe purchase 4 more 4tb drives, remove the 2 spares mentioned earlier to create a 3rd vdev with 6 more drives. From everything I've been reading, it seems like this would greatly improve the speed, correct?

I'm probably understanding everything wrong, to be honest this is hurting my brain. I just want to maximize my speed and also usable space while keeping some parity in mind. I still have a good (cloud-based) backup strategy, but I'd like to avoid those headaches if possible.

 

[Heracles]

Hi again Mike,

But with mirrors, I lose half my usable space right?

Indeed, mirrors will give you the usable space of half your drives.

2x Raid-Z2 vDev + 2 spares = 6 drives dedicated to redundancy out of 14 drives.
Mirrors with 14 drives = 7 drives dedicated to redundancy out of 14 drives.

So only 1 drive difference between the two options, out of 14 drives. Is that such a big difference ?

 

[mikestecker]

Indeed, mirrors will give you the usable space of half your drives.

2x Raid-Z2 vDev + 2 spares = 6 drives dedicated to redundancy out of 14 drives.
Mirrors with 14 drives = 7 drives dedicated to redundancy out of 14 drives.

So only 1 drive difference between the two options, out of 14 drives. Is that such a big difference ?

How does expandability work for this down the road? Say I were to order the 4 other drives I had previously mentioned, bringing me to a total of 18 drives. Does the pool easily expand to a mirror of 9+9? As I understood with vdevs, it seems you can only add a new vdev to a pool and not actually expand the existing vdev. Is that correct or no? Apologies for all the questions, I'm still learning

 

[JaimieV]

What Heracles is recommending is not (7 disks) mirrored (which would be just one vdev) but seven separate mirrored-pair vdevs in a pool. You would add more capacity in additional mirror-pair vdevs. And it's a *very* good point that drive usage is almost the same, I think he's actually just persuaded me to go to mirror pairs on my primary - I was considering doing a rebuild anyway.

Straight talk: If you start with a pair of 7-disk(4TB) RAIDZ2's there's actually nothing stopping you just adding a 2-disk(anyTB) mirror pair to the pool, but it's considered a bit messy and makes throughput, IOPS and resiliency more unpredictable. It will work though, ZFS won't stop you.

 

[mikestecker]

Ahh now I understand, thanks for explaining it in that way since I wasn't understanding before. Next question, if I go this route, do I do multiple pools in case a pool might potentially fail? Or are the chances of this pretty low? Let's say I take all 14 drives and put them in a big pool, then after I transfer my data from my old NAS, and move over those 4 more drives, should I go ahead and add those 4 new drives to the same pool or is it wise to create a new pool instead? I'm just curious what you think "best practices" are

 

[Heracles]

Hi again Mike,

The only reason for doing multiple pool is when you need different pool options in them. Something like encryption (which I highly discourage you to do) is done at pool level. Should you wish to have one encrypted and one not encrypted, you need two pools.


Other than for different pool level option, you usually do not create multiple pools. You create multiple datasets / zvol instead.

Also, be aware that no matter how strong and robust your main server will be, you will always need backups. See the 3 copies rules in my signature about that...

 

[mikestecker]

Other than for different pool level option, you usually do not create multiple pools. You create multiple datasets / zvol instead.

Also, be aware that no matter how strong and robust your main server will be, you will always need backups. See the 3 copies rules in my signature about that...

Great, thanks! When I was reading about if a pool were to go bad then everything in the pool is lost, I started to get worried.

As for backups, my main plan at this point is using CrashPlan's NAS server plan. $5/unlimited data. I know it's not the best plan and that I should have another physical copy somewhere, but I'm just not able to quite yet. Eventually I'll get around to building another server.

Thanks for all the information and help you've provided! I think I have a good plan now. Just waiting on my two 60gb SSD's to replace the 250's I'm currently using as my boot. Once those arrive, I'll re-install and setup FreeNAS again with the plan for mirroring like you suggested. Then I can use the 250's for jails and/or VM's.

 

[Jessep]

Please read the thread stickies about the various pool vdev options, striped mirrors is significantly more risky during resilver after a failed drive.

When a single drive in a mirror fails you have no redundancy on that remaining drive, if the second drive in that mirror fails during resilver you lose the entire pool.

If your data matters think about how you want it to perform vs redundancy. Personally expansion is way low on my priority list vs. data security. I tend to expand my pool once every few years and replace the entire pool. Drives have warranty and lifespan, you can't keep them forever.

My suggestion is split your data into multiple pools, long term important data RaidZ2 or RaidZ3, other data at whatever is your risk level.

Performance pool go with SSD. Things like VM storage.

 

[Heracles]

Hey Jessep,

You talk like your FreeNAS is holding your only copy of your data... If that is the case, know that no FreeNAS server can protect you efficiently against all scenarios. You always need backups for everything you consider important.

You can do 3-way mirrors if you wish. You can do 4 drives Raid-Z2 to have the same redundancy level of 50% while ensuring redundancy to every vDev after a single failure. But everything will be destroyed by a fire, a flood, etc.

Here, I consider that backups is a hard requirement for everything. For that, 2 drives mirrors are good enough for me because I do have all the backups required. Should my main server fails because of 2 failed drives in the same vDev, I will survive. I will re-build the pool with good drives, then do a local sync from Thanatos for most of the data to be transferred locally. Once that sync is done, I will re-sync from Hades for the latest data up to the last 15 minutes.

If you are afraid of standard mirrors, it is because you do not trust your backups.
If you do not trust your backups, it is as good as not having backups at all.
If you do not have backups at all, do some ASAP as a higher priority then Raid-Z3 or other increased local redundancy.

 

[Jessep]

Hey Jessep,

You talk like your FreeNAS is holding your only copy of your data... If that is the case, know that no FreeNAS server can protect you efficiently against all scenarios. You always need backups for everything you consider important.

I have backups of important data (3-2-1), that doesn't change the numbers that you are much more likely to lose a pool using mirrors than R2/R3.

Mirrors have their uses, however new folks should be informed of the downsides as well as the benefits so they can make their own judgement based on their own risk stance.

This isn't a "one true way" forum, it's about informing and letting folks makes a choice.

Please don't "evangelise" solutions, that's how we got into the "1GB RAM per 1TB storage" silliness among others, we don't need a new one of striped mirrors is the "one true way".

Personally I'm risk adverse, I use server/enterprise gear, ECC, and conservative pool layouts as well as backups. I have yet to require restore from backup, though I have recovered a fair few snapshots over the years (VMware level mostly, usually due to crappy software updates). Others will have a different risk stance.

Now do you want to hold hands and sing "kum bah yah"? (grin)

 

[Heracles]

we don't need a new one of striped mirrors is the "one true way".

When one is asking for a pool configuration announcing that he will run VMs and do so many different things all at once, he is clearly on the side where mirrors will serve him better as a matter of performance. That is a matter of fact.

With a pool made of 7 mirrors, the probability to survive the loss of a second drive is 12/13, so higher than 92%. Not that far of Raid-Z2 that is 100%. That is also a matter of fact and math.

The remaining 8% is to be mitigated by backups which in all cases are required for everyone, in every case, for everything valuable. That is a third fact.

Not to mix-n-match vdev design is a clear recommendation that is valid for everyone. To add a total of 4 drives in that 14 drives config while respecting that reality again pushes for the only vDev size that can do the math : 2.

So with that many drives, planning for that expansion, for such a use case and once backups are confirmed to exist, mirrors are a clear winner.

I am using Raid-Z2 myself on my backup server. That one is meant to hold data and nothing more. I even have a Raid-Z1 pool for my offline server. That Raid-Z1 risk is mitigated by the fact that cold spare are available on site and that it can fail without consequence as long as any of the 2 other servers survive (one with mirrors and the other as Raid-Z2).

So there are cases for Raid-Z, but this one is just clearly not one of them...

 

[blueether]

that doesn't change the numbers that you are much more likely to lose a pool using mirrors than R2

I actually challange you to do the maths on that for a 6 disk array (mirrors vs raidz2) and then for 8 disks (and more)

take into account the time it takes to resilver for a loss of two disks

new do both of these loosing 3 disks before you nottice and get the resilvering complete...

Are mirrors that bad?