Pool Configuration

[brando56894]

Since I'll probably be jumping back into FreeNAS in the coming months and I'll have the money to upgrade, I'd like to flesh out my future optimal pool setup and get some input before I drop a grand or two on drives.

My current hardware won't change, I just be adding in either 4, 6, 8, or 10 TB drives and an NVMe drive. Here's my current hardware

Server
Asus X99-WS/IPMI
Liquid cooled Xeon E5-1650 (6 cores/12 Threads @ 4.2 GHz)
2x 32 GB Samsung DDR4 ECC RAM
Liquid cooled Nvidia GeForce GTX 1070
EVGA 1KW PSU
NZXT H440 Case
7x 4 TB HDDs

I'm currently using unRAID and have two VMs setup: Windows 10 with the 1070 passed through which I use to run Kodi for my HTPC (server is in my living room and connected to my TV), which is also used for occasional (heavy) gaming, and then an Arch Linux VM which runs Nginx as a reverse proxy for the following apps: Radarr, Sonarr, Deluge, NZBget, NZB Hydra, and HTPC Manager. I also have a few Docker containers setup in unRAID: Plex, PlexPy, NextCloud, and MuxiMux (dashboard combining multiple sites under one domain), some of which also run through Nginx.

Plex isn't used that often, but I do have it shared out to about 8 people. We mostly use Kodi inside the network, and that's max two clients.

I will have a separate pool for the NVMe drive which will be mostly used for block storage (ZVOLs) for the VMs, that's an easy decision. All drives will be connected via a 4 port HBA (minus the NVMe, which will be directly connected to the motherboard).

I was previously using striped mirrors for my pool and loved the performance benefits and ease of upgradability, but the halving of storage space and limited redundancy sucks. I figured I'd go with RAIDZ2 this time around, but since I know it takes a bit of pre-planning, I don't know what the optimal configuration for my storage pool should be. I currently have 20 TB with only about 11.5 TB used, so I don't really need much more than that, the question is more is it better to have wider vdevs (I was thinking 2 of 6 drives each) or multiple vdevs so that they can be striped together (3 of 4 drives each)? I was thinking 3 vdevs of 4x 4 TB WD Red Pro drives so that they wouldn't hurt my wallet too much and smaller vdevs would be easier to upgrade incrementally. Which would benefit me more?

I would be using my existing drives in the vdevs, but would first need to create a pool with the empty drives so that I could copy the data from unRAID's JBOD array to the ZFS pool (I don't feel like downloading multiple TBs again for the umpteenth time, even though I do have a gigabit connection hahaha), luckily unRAID (unofficially) supports ZFS via a plugin so that would be a breeze.

 

[BigDave]

I was previously using striped mirrors for my pool and loved the performance benefits and ease of upgradability, but the halving of storage space and limited redundancy sucks. I figured I'd go with RAIDZ2 this time around, but since I know it takes a bit of pre-planning, I don't know what the optimal configuration for my storage pool should be. I currently have 20 TB with only about 11.5 TB used, so I don't really need much more than that, the question is more is it better to have wider vdevs (I was thinking 2 of 6 drives each) or multiple vdevs so that they can be striped together (3 of 4 drives each)? I was thinking 3 vdevs of 4x 4 TB WD Red Pro drives so that they wouldn't hurt my wallet too much and smaller vdevs would be easier to upgrade incrementally. Which would benefit me more?

With today's larger capacity drives, RAIDz2 is best for a reasonable amount of parity/redundancy.

When using (for simplicity of discussion) two drives for parity, it makes no sense to make the Vdevs
with only 4 drives, unless your only concern is redundancy. The capacity of the 4 drive RAIDz2 Vdev is basically the same as two RAIDz mirrors as far as usable space is concerned. For your use case my choice would be two Vdevs
of 6 drives each as you stated above, this configuration can be upgraded by replacing with larger drives or adding
another Vdev later.
I was told years ago that the sweet spot for RAIDz2 was 6 drives per Vdev and I still think that advice holds true today.

 

[brando56894]

That works for me, thanks.

 

[Chris Moore]

I was told years ago that the sweet spot for RAIDz2 was 6 drives per Vdev and I still think that advice holds true today.

@brando56894 , another vote for the 6 drive vdev. I have a 24 bay chassis and six goes into 24 perfectly.
The other option would be 8 drives and 3 vdevs, but more vdevs equals more performance.

 

[joeschmuck]

While it sounds like running two vdevs of 6 drives is better (and I agree), do you have any idea how much throughput you need? Right now it sounds like you are using the rust spinning drives just for storage, not for anything requiring speed. If you go with the two vdevs you might be able to drop your Arch Linux VM to the pool and free up the NVMe all for Win10.

 

[Arwen]

It's not clear if you would want to dual, (or tri), boot this computer. If so, I suggest a dedicated NAS box, even if it has to be lower in performance and capacity. (Because some of the capacity is used by the clients.)

Linux dual boots fine, and generally MS-Windows does too. But, server OSes like FreeBSD & Solaris tend to not have a configuration that other OSes recognize. Thus, the boot drive for FreeNAS, (which uses FreeBSD), could be over-writen by an over zealous MS-Windows program, (or user). And possibly the ZFS data disks. However, if you are the only user, then maybe you would be okay.

Ignore me if you were not going to multi-boot this computer..

 

[brando56894]

@Arwen No multi-booting going on, just FreeNAS as the "host" OS and then two VMs running Arch and Windows 10 on top of it :)

While it sounds like running two vdevs of 6 drives is better (and I agree), do you have any idea how much throughput you need? Right now it sounds like you are using the rust spinning drives just for storage, not for anything requiring speed. If you go with the two vdevs you might be able to drop your Arch Linux VM to the pool and free up the NVMe all for Win10.

As much as I can get! :D It doesn't need to be ridiculously fast (although I did like reading and writing around 1 GB/sec with striped mirrors), but I'd like it to be able to handle multiple simultaneous reads and writes without a hitch. The speed of one drive, be it SSD or HDD is my current choke point. I'll probably just buy the NVMe drive now and use that as my cache drive in unRAID and put the VMs on the SSD while I'm waiting for the kinks to be ironed out in FreeNAS.

If you go with the two vdevs you might be able to drop your Arch Linux VM to the pool and free up the NVMe all for Win10.

I'd still probably keep the Arch VM on the NVMe drive because I'll probably get the 1 TB model and I definitely don't need that much space for Windows since I don't do much with it (128 GB boot drive, and maybe an iSCSI drive). Most gaming happens on my PS4 Pro. What I probably end up doing is put my download directory for Usenet on there so that it can do all the post processing on flash media and then it will just have to dump the resulting file from flash to mechanical storage. Some of these files can be 60 GB a piece, so like 7 of them completely overwhelms my current SSD. Ideally I'd like to have everything in the best quality available (UHD w/HDR if available).

 

[brando56894]

...And of course NewEgg has to be evil and tempt me with their damn Pre Black Friday sales hahaha I just saw that the 4 TB WD Reds are on sale for $134 for 3 days, but the Red Pros are $170.

 

[joeschmuck]

...And of course NewEgg has to be evil and tempt me with their damn Pre Black Friday sales hahaha I just saw that the 4 TB WD Reds are on sale for $134 for 3 days, but the Red Pros are $170.

I myself would not spend the extra money on the Pro version unless you desire 7200 RPM or the price was very good. My Reds are a few days away from the 5 year point and are plugging along great. I have the server in my computer room right now with the case sides off and I can't hear them unless I put my ear to the drives. They are just working great which is torture to me because as soon as I see the 6TB drives on sale again then I'm going to buy them. I want to replace my drives but these damn things are not failing yet and I'm not out of capacity either. My luck would be Feb 1st 2018 all my drives start to fail and I miss all the sales. But if I buy the new drives then the current drives will keep running for another 2 years. If I were made of money then I'd have new drives already but like most people I need to figure out what makes more sense.

 

[Jailer]

They are just working great which is torture to me because as soon as I see the 6TB drives on sale again then I'm going to buy them.

Paging @joeschmuck. @joeschmuck to the white courtesy phone.....

https://www.newegg.com/Product/Prod...ternalHardDrives-_-22236737-S1A1A&ignorebbr=1

 

[Chris Moore]

You could go for this one: https://www.newegg.com/Product/Product.aspx?Item=N82E16822235157
It is only $146 and at that price it might allow you to buy a couple extra spares.

 

[joeschmuck]

Paging @joeschmuck. @joeschmuck to the white courtesy phone.....

https://www.newegg.com/Product/Prod...ternalHardDrives-_-22236737-S1A1A&ignorebbr=1

I saw the NewEgg 6TB Red sale just a few minutes ago and it needs to drop just a few more bucks to make it worth my while. I wish those 8TB external drives would come on sale again.

You could go for this one: https://www.newegg.com/Product/Product.aspx?Item=N82E16822235157
It is only $146 and at that price it might allow you to buy a couple extra spares.

LOL, buy spares! Remember one thing, old people live on limited income. While I'm not retired yet my mind does think about not having extra money just laying around in the near future, unfortunately not soon enough. Unfortunately it appears that these are WD Blue drives internally, nothing against these drives as in general they appear to work just fine but I could get the 4TB Ironwolf drives and buy 5 of these for about $50 USD more total price and I get a true NAS drive. The downside here is it's 5 drives vice 4 drives which is my goal.

I appreciate both of you paging me, wish I had a smart phone so I could have seen these sooner but I really like my cheap little flip phone, it calls the wife and work can call me if they need to.

I think I'll be waiting for Cyber Monday unless a super sweet deal comes around like those 8TB WD Easyshare drives for $129 each, then you can bet I'll drop some money on those immediately.

 

[Jailer]

I really like my cheap little flip phone,

When I retire, exactly 3 years from tomorrow, I will not have a phone period. The warden can keep hers but I'm done with em.

 

[joeschmuck]

When I retire, exactly 3 years from tomorrow, I will not have a phone period. The warden can keep hers but I'm done with em.

I wish it were 3 years for me but my daughter will not have a Masters degree by then, I have a few more to go after that. Of course if the wife hit the lottery then I could be done next week :)

It's amazing how the different generations of people are with respect to cell phones. While I think it's nice to have one available, I could live without it pretty easy.

P.S. Sorry for getting off topic in this thread.

 

[brando56894]

I myself would not spend the extra money on the Pro version unless you desire 7200 RPM or the price was very good. My Reds are a few days away from the 5 year point and are plugging along great. I have the server in my computer room right now with the case sides off and I can't hear them unless I put my ear to the drives. They are just working great which is torture to me because as soon as I see the 6TB drives on sale again then I'm going to buy them. I want to replace my drives but these damn things are not failing yet and I'm not out of capacity either. My luck would be Feb 1st 2018 all my drives start to fail and I miss all the sales. But if I buy the new drives then the current drives will keep running for another 2 years. If I were made of money then I'd have new drives already but like most people I need to figure out what makes more sense.

Thanks for the info, I have 2 of either the reds of red pros in my server right now and same as you, they're chugging right along, the only one with SMART issues (reallocated sectors) is one of my HGST drives which is only like a year or so old. I'm going to replace the HGSTs with the reds, not sure if the extra heat and increased platter rotation speed is worth it since they'll be crammed on top of each other and are already about 95F each.

It's amazing how the different generations of people are with respect to cell phones. While I think it's nice to have one available, I could live without it pretty easy.

My dad is 67 and only got a cell phone like 4 years ago and rarely ever uses it, he was finally forced to get a smart phone by TracFone since he needed a 4G phone and he still doesn't see its usefulness haha I'm 32 and just like most people my age, it's my life line haha

 

[HellQuaker]

Hey Guys, i got a 24Bay and i am considdering how to organise it...

4 pools with 6x8 TB each sounds great, but only 1 HDD from each pool can crash then, if I go for RAID Z1 (space nearly 7x8TB minus the check-sums, so i get around 49-51TB) ... if i make a complete pool of 24x8 TB, then 3 HDDs can crash simulaniosly without data-loss with the same space available or am I wrong?

 

[Chris Moore]

Hey Guys, i got a 24Bay and i am considdering how to organise it...

4 pools with 6x8 TB each sounds great, but only 1 HDD from each pool can crash then, if I go for RAID Z1 (space nearly 7x8TB minus the check-sums, so i get around 49-51TB) ... if i make a complete pool of 24x8 TB, then 3 HDDs can crash simulaniosly without data-loss with the same space available or am I wrong?

Based on what you are saying, I guess that you have not read any of the past guidance on this forum. We never suggest using RAIDz1 with drives larger than 2TB, and you would certainly not make four separate pools. I think you need to do some reading, here are the links that might give you answers but, after you read them, if you still have questions, please do ask.

Forum Guidelines
https://www.ixsystems.com/community/threads/forum-guidelines.45124/

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Introduction to ZFS:
https://www.ixsystems.com/community/resources/introduction-to-zfs.111/