Is this sane planning? (Larger home use FreeNAS, 40TB)

[Yoshimo]

I'm intending to built a FreeNAS-based storage server for a maximum of two simultaneous users in a 1GBit/s home network. Data will usually be written to the system only once and reading happens only occasionally. It shall store movies, music, audio books, etc; so it has to be huge. However, it also has to store important personal files like images, documents, scans, and work-related stuff; so there is need for parity. There won't be any encoding or similar stuff done on the FreeNAS; it's pure storage. Also there will be no jails.

The plan so far is to buy a 12-bay case and fill it with 12x4TB drives. Than I'll put them in a single RAIDZ2 vdev inside a single pool and be done. 40TB of useable storage space is perfect. RAIDZ feels a bit insecure, because when one drive fails the second drive follows hardon the heels of the first. And RAIDZ3 is a bit too paranoid.

Budget: 2,200.00€ (absolute maximum!)

With that budget everything has to be built on consumer-grade hardware, except for the ECC RAM and ECC-supporting components. So far I'm targeting the Seagate Desktop HDD 4 TB ST4000DM000 drives, as four of them run smoothly for many years now in my desktop system. And they are cheap! One bills at 125€, and when I buy more than 10 pieces I'll get a 10% discount. So 850 Euro will be left for the case, mainboard, CPU, RAM, SATA controller, etc.

So far I'm considering this setup:

MB: Supermicro X11SSM-F --- 232€
CPU: Intel Xeon E3-1230v5 --- 210€ (The Xeon E3-1240v5 would cost 379€!)
RAM: Crucial 16GB DDR4-2133 reg ECC --- ca. 150€
SATA: AMCC 9650SE-12ML PCIe SATA2 Raid Array Controller (used) --- 70€
PSU: be quiet! Straight Power 10 CM 500W --- 108€
HDD: 12x Seagate Desktop HDD 4 TB ST4000DM000 --- 1350€
Case: I'll use one of my old big tower cases.

---> TOTAL: 2,120.00€, leaving 80€ for shipping costs.

So the question is: Is this sane planning?

Btw: I already have a powerful(!) UPS for my router and my home server (runnig a Tor node, OSCam, JDownloader, CCTV), so there's no need to spent money on this.

 

[Spearfoot]

I'm intending to built a FreeNAS-based storage server for a maximum of two simultaneous users in a 1GBit/s home network. Data will usually be written to the system only once and reading happens only occasionally. It shall store movies, music, audio books, etc; so it has to be huge. However, it also has to store important personal files like images, documents, scans, and work-related stuff; so there is need for parity. There won't be any encoding or similar stuff done on the FreeNAS; it's pure storage. Also there will be no jails.

The plan so far is to buy a 12-bay case and fill it with 12x4TB drives. Than I'll put them in a single RAIDZ2 vdev inside a single pool and be done. 40TB of useable storage space is perfect. RAIDZ feels a bit insecure, because when one drive fails the second drive follows hardon the heels of the first. And RAIDZ3 is a bit too paranoid.

Budget: 2,200.00€ (absolute maximum!)

With that budget everything has to be built on consumer-grade hardware, except for the ECC RAM and ECC-supporting components. So far I'm targeting the Seagate Desktop HDD 4 TB ST4000DM000 drives, as four of them run smoothly for many years now in my desktop system. And they are cheap! One bills at 125€, and when I buy more than 10 pieces I'll get a 10% discount. So 850 Euro will be left for the case, mainboard, CPU, RAM, SATA controller, etc.

So far I'm considering this setup:

MB: Supermicro X11SSM-F --- 232€
CPU: Intel Xeon E3-1230v5 --- 210€ (The Xeon E3-1240v5 would cost 379€!)
RAM: Crucial 16GB DDR4-2133 reg ECC --- ca. 150€
SATA: AMCC 9650SE-12ML PCIe SATA2 Raid Array Controller (used) --- 70€
PSU: be quiet! Straight Power 10 CM 500W --- 108€
HDD: 12x Seagate Desktop HDD 4 TB ST4000DM000 --- 1350€
Case: I'll use one of my old big tower cases.

---> TOTAL: 2,120.00€, leaving 80€ for shipping costs.

So the question is: Is this sane planning?

Btw: I already have a powerful(!) UPS for my router and my home server (runnig a Tor node, OSCam, JDownloader, CCTV), so there's no need to spent money on this.

Welcome to the forums!

You're on the right track with your NAS design! I have a few observations and suggestions:

• Consider using a Pentium instead of a Xeon CPU. Many home users over-estimate their CPU requirements, and you've stated there will only be two users with no encoding. Spend the money you save on memory.
• Consider installing 32GB of memory. A dozen 4TB drives will provide ~32-40TB of storage depending on configuration and overhead. Our rough rule-of-thumb for specifying memory ("8GB + 1GB per TB of storage") suggests you'd need more, but I believe 32GB would be sufficient for your use-case.
• Save yourself some grief and dump the 9650SE RAID controller; it's utterly unsuited for use with FreeNAS. Get an LSI-based HBA (LSI 9211/IBM M1015/Dell H200/H310), which are well-supported and work great with FreeNAS.
• A 500W PSU won't be adequate for a 12-drive system; see "Proper Power Supply Sizing Guidance" for details. You'll probably end up needing 750W or thereabouts.

A 12-disk vdev is considered a little too 'wide' for RAIDZ2, so consider being a little more paranoid and using RAIDZ3. :D For better performance, consider using two vdevs made up of 6 disks in a RAIDZ2 configuration. The later approach would let you create a pool with a single six-disk RAIDZ2 vdev now, to get started, and you could then expand the pool by adding a second six-disk RAIDZ2 vdev later. This would let you spread your capital outlay out over time.

Before you make any purchases, do some study here on the forum. With the proper design, you can build a reasonably-priced FreeNAS server that will give you years of reliable service. We want you to be pleased and happy! Anyway... here's a reading list that may help:

https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/
https://forums.freenas.org/index.php?threads/zfs-primer.38927/
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
https://forums.freenas.org/index.php?threads/comprehensive-diagram-of-the-zfs-structure.38865/
https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/
https://forums.freenas.org/index.php?threads/how-to-hard-drive-burn-in-testing.21451/
https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/
http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance
https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz
http://doc.freenas.org/9.10/

Good luck!

 

[Ericloewe]

Save yourself some grief and dump the 9650SE RAID controller; it's utterly unsuited for use with FreeNAS. Get an LSI-based HBA (LSI 9211/IBM M1015/Dell H200/H310), which are well-supported and work great with FreeNAS.

+1

A 500W PSU won't be adequate for a 12-drive system; see "Proper Power Supply Sizing Guidance" for details. You'll probably end up needing 750W or thereabouts.

I'd say it's cutting it close, but doable. 750W leaves a nice margin for growth.

RAM: Crucial 16GB DDR4-2133 reg ECC --- ca. 150€

Nope, as is written everywhere, LGA115x processors do not take RDIMMs. You need UDIMMs.

 

[tvsjr]

I'd suggest looking at Seasonic for your PSU. Many of us use them and have been very satisfied.

The hard drives you've selected are intended for desktop PC use, not for a NAS. Consider going to one of the tried-and-true hard drives around these parts... WD Red or HGST NAS. If you're stuck on Seagate, they build a NAS drive as well.

 

[Yoshimo]

Hello, thank you for your inputs. I revised my list according to your thoughts:

(1) Changed MB from Supermicro X11SSM-F to Supermicro X11SSM, as I don't need the IPMI port.
(2) Changed CPU from Intel Xeon E3-1230v5 to Intel Pentium G4520 as advised. *
(3) Changed RAM from Crucial 16GB DDR4-2133 reg ECC to Crucial 32GB DDR4-2133 unbuffered ECC as advised.
(4) Changed SATA from AMCC 9650SE-12ML PCIe SATA2 Raid Array Controller (used) to LSI SAS 9211-4i Host Bus Adapter (used) as advised. **
(5) Changed PSU from be quiet! Straight Power 10 CM 500W to be quiet! Straight Power 10 CM 700W as advised. ***

* = A Xeon E3-1230v5 is still in consideration, as it's only 106€ more and maybe better in handling a potential RAIDZ3.
** = Will simply use the 6 onboard SATA3 connectors plus the controller's 4 SATA3 connectors.
*** = 700W really is enough. I've a 750W PSU in my workstation and it works just fine, despite a dual GPU setup (2x AMD Radeon R9 Fury X).

The new list:

MB: Supermicro X11SSM --- 212€
CPU: Intel Pentium G4520 --- 104€
RAM: Crucial 32GB DDR4-2133 unbuffered ECC --- 311€
SATA: LSI SAS 9211-4i Host Bus Adapter (used) --- 66€
PSU: be quiet! Straight Power 10 CM 700W --- 140€
HDD: 12x Seagate Desktop HDD 4 TB ST4000DM000 --- 1350€
Case: I'll use one of my old big tower cases.

---> TOTAL: 2,183.00€

Now for that RAIDZ3 thing: I once read that RAIDZ/RAIDZ2/RAIDZ3 works best when the number of total drives in the vdev minus the number of parity drives in the vdev is an even number. As a 12 drives RAIDZ3 setup would result in an odd numer of regular drives (9 pieces), isn't RAIDZ3 on a 12 drives setup a bad idea? Or is this rule of thumb just a myth? Of course I could reduce the number of drives from 12 to 11 and therefore get 8 instead of 9 regular drives. But this would bring down usable data space from 40TB to 32TB and also would increase workload on the CPU. (How bad is the performance penality of RAIDZ3 compared to RAIDZ2? I have seen many warnings about RAIDZ3!)

And yes, an alternative to RAIDZ3 would be 2xRAIDZ2 vdevs. However, this would bring down usable data space from 40TB to 32TB while still needing 12 instead of 11 drives. So same space as with 11 disc RAIDZ3, but more cost. Basically 2xRAIDZ2 is 1/3 of drives for parity, while 1xRAIDZ2 on 12 drives is 1/6 for parity and 1xRAIDZ3 is 1/4 for parity. The third one reads really good to me, and I would like to do it as long as there is no problem with having and odd number of regular drives inside that RAIDZ3 and as long as there isn't a too big performance penality. It would mean 36TB of useable space and that up to 3 arbitrary drives could fail without data loss. In 2xRAIDZ2 setups up to 4 drives but only 2 arbitrary drives are allowed to fail.

 

[tvsjr]

I would strongly suggest keeping the IPMI port. It's one of those things you don't need often, but it can be very useful when you do. And it doesn't exactly cost much.
Make sure the memory you selected is on the Supermicro QVL; otherwise, they'll suggest your memory is the problem if you call them with issues.
You can cable all of your drives to the HBA via a SAS expander. If you want to direct cable, you could go to an 9211-8i, giving you an extra 4 channels.

As far as the vdev sizing, the even number thing isn't valid any more with compression in play. Wider vdevs pose two issues... longer rebuild times and less aggregate performance (you only get the IOPS of the slowest drive in each vdev).

You still need to swap the drives for models that are rated for NAS use.

 

[Yoshimo]

I'd suggest looking at Seasonic for your PSU. Many of us use them and have been very satisfied.

Nope. I'm very satisfied with the be quiet! PSU lineup. You can push them to 100% for prolonged periods of time and they still work like a charm and are virtually inaudible. And it's not just wattage: Their output is perfectly flat and accurate even when fluctuations in demand occur. Also they are ultra-durable: I'm using one of their units in my homeserver and it's from 2009 - it worked nearly uninterrupted for the last 8 years and is still going. A newer one is in my workstation and it's great, too.

The hard drives you've selected are intended for desktop PC use, not for a NAS. Consider going to one of the tried-and-true hard drives around these parts... WD Red or HGST NAS. If you're stuck on Seagate, they build a NAS drive as well.

Wish I had enough money for WD Reds. But one 4TB WD Red costs 189€ here - too much. Moreover, I'm really satisfied with the Seagate ST4000DM000 drives. Using 4 of them in my PC (no spindown) and another one in my sat reciver for many years now and so far none of them died.

 

[Yoshimo]

I would strongly suggest keeping the IPMI port. It's one of those things you don't need often, but it can be very useful when you do. And it doesn't exactly cost much.

Can you specify a use case for IPMI on a home server?

Make sure the memory you selected is on the Supermicro QVL; otherwise, they'll suggest your memory is the problem if you call them with issues.

Okay, I just looked it up and found out my selected Crucial RAM is not on their list. Normally I wouldn't mind, as I always used uncertified RAM in my systems without any problems. (IMHO this "certification" is pure moneymaking by the mainboard vendors.) But this time it will be different, as I found out there is certified RAM available at a significantly lower price!

(1) Changed RAM from Crucial 32GB DDR4-2133 unbuffered ECC to Hynix 32GB DDR4-2133 unbuffered ECC (5AN4G8NAFR-TFC) as advised.

The new list:

MB: Supermicro X11SSM --- 212€
CPU: Intel Pentium G4520 --- 104€
RAM: Hynix 32GB DDR4-2133 unbuffered ECC (5AN4G8NAFR-TFC) --- 231€
SATA: LSI SAS 9211-4i Host Bus Adapter (used) --- 66€
PSU: be quiet! Straight Power 10 CM 700W --- 140€
HDD: 12x Seagate Desktop HDD 4 TB ST4000DM000 --- 1350€
Case: I'll use one of my old big tower cases.

---> TOTAL: 2,103.00€

You can cable all of your drives to the HBA via a SAS expander. If you want to direct cable, you could go to an 9211-8i, giving you an extra 4 channels.

I'll have to read about SAS expanders first. But it looks promising.

As far as the vdev sizing, the even number thing isn't valid any more with compression in play. Wider vdevs pose two issues... longer rebuild times and less aggregate performance (you only get the IOPS of the slowest drive in each vdev).

I wouldn't have activated compression, as most data can't be compressed anymore because it already is heavily compressed (e.g. video data). But when compression fixes the odd/even number issue, I'll switch it on and go with a single 12x4TB RAIDZ3.

You still need to swap the drives for models that are rated for NAS use.

No, I won't do this. I'm convinced it will be fine with the drives I selected. These ones or no NAS at all.

 

[Ericloewe]

Can you specify a use case for IPMI on a home server?

Not needing a monitor + keyboard, not needing to be next to the server...

 

[Spearfoot]

Can you specify a use case for IPMI on a home server?

Certainly. You can access the console of the server without having to attach it to a keyboard, mouse, or monitor. You can boot the server from USB drives or ISO images attached to your workstation. In my case, my servers are located in my shop, 20-30m from my desk -- I can access them both just as though I am sitting in front of them.

I couldn't image living life without IPMI... :)

 

[Kevin Horton]

The IPMI can be set up to email alerts for fan failure, CPU over temperature, etc. You can then log in to the IPMI to check all the sensor readings, and shut the whole system down if it is warranted. I've set up a VPN on my router, so I do this while I'm on the road, if need be.

 

[tvsjr]

I wouldn't have activated compression, as most data can't be compressed anymore because it already is heavily compressed (e.g. video data). But when compression fixes the odd/even number issue, I'll switch it on and go with a single 12x4TB RAIDZ3.

Actually, it's recommended to leave compression on. In fact, it defaults on, using lz4. Do not confuse this with deduplication, that's a totally different thing. Basically you won't even know compression is there, and it has several benefits. On my storage array, which is pretty much music (95% FLAC, quite a bit of 96/24), movies (primarily MKVs), and software, I'm currently showing a compression ratio of 1.15:1... so, it's actually somewhat beneficial.

 

[CraigD]

No, I won't do this. I'm convinced it will be fine with the drives I selected. These ones or no NAS at all.

For your needs a Pentium is all you need, my G3258 Pentium serves four users DATA fine, and never goes over 20% useage

I use to think like this, and put 8 desktop HDDs in my NAS

TWO have failed over the last six months and another is on its way

Have Fun
PS IPMI is no needed but is good to have

 

[Kevin Horton]

For your needs a Pentium is all you need, my G3258 Pentium serves four users DATA fine, and never goes over 20% useage

I use to think like this, and put 8 desktop HDDs in my NAS

TWO have failed over the last six months and another is on its way

Have Fun
PS IPMI is no needed but is good to have

Keep in mind that some of the items in Seagate drive SMART results are encoded in binary form, thus the raw integer results reported by smartctl for these values are meaningless. Specifically, the Seek Error Rate, Raw Read Error Rate, and Hardware ECC Recovered SMART attributes as reported by smartctl are essentially meaningless. You would need to use a proprietary Seagate tool to interpret these results. More info here.

 

[Yoshimo]

Actually, it's recommended to leave compression on. In fact, it defaults on, using lz4. Do not confuse this with deduplication, that's a totally different thing. Basically you won't even know compression is there, and it has several benefits. On my storage array, which is pretty much music (95% FLAC, quite a bit of 96/24), movies (primarily MKVs), and software, I'm currently showing a compression ratio of 1.15:1... so, it's actually somewhat beneficial.

I didn't confuse compression with deduplication. Deduplication is something I distrust because of possible hash collisions, and bit-for-bit validation really isn't worth the truble. Compression however is ok; but I seriously considered it to be "high pain low gain", i.e. it would bring down performance significantly. But a 1.15:1 compression ratio at a negligible performance hit is definitely worth it!

Btw: We have similar media preferences. My NAS will be full of FLAC files (I'm an audiophile and proud of it!) and Blu-Ray movies remuxed (= not reencoded) into MKV containers.

Not needing a monitor + keyboard, not needing to be next to the server...

That's why I asked about use cases on a home server that will either sit directly beside my desktop PC or directly beside the router, so plugging a monitor and peripheral equipment will be no problem.

However, maybe an IPMI port will increase the board's resale value. I'll buy the F variant of the mainboard with IPMI in case the budget allows it.

For your needs a Pentium is all you need, my G3258 Pentium serves four users DATA fine, and never goes over 20% useage

One thing I forgot to mention in my initial post was that encryption will be turned on. But thanks to AES-NI this won't be a problem. So the Intel Pentium G4520 it is.

I use to think like this, and put 8 desktop HDDs in my NAS

TWO have failed over the last six months and another is on its way

Backblaze has a drive reliability page with stats spanning many years. And know what? The ST4000DM000 is performing better than most WD Reds! Over 2 years the ST4000DM000 had a lower annualized failure rate than the WD40EFRX. That's why they have 34,729 ST4000DM000 drives in their systems - more than all of their other drives combined.

And once again: None of my 4TB Seagate Desktop HDD drives ever failed. And some of them are many, many years old. And should one of the new NAS unexpectedly drives fail, it can easily be replaced with one of the workstation's drives, as they won't be needed anymore after their data has been migrated onto the NAS.

 

[Victor0001]

Maybe a silly question but how much data do you have at the moment that you wish to store on your future NAS and how much growth do you expect per month/year or whatever you can guess.
If you have like a couple of TB's and expect growth over years to 10TB, then you have space wasted and rather have a smaller 'safer' vdev of 6 drives and expand later on with another vdev of 6 drivers.

 

[Yoshimo]

Maybe a silly question but how much data do you have at the moment that you wish to store on your future NAS and how much growth do you expect per month/year or whatever you can guess. :)
If you have like a couple of TB's and expect growth over years to 10TB, then you have space wasted and rather have a smaller 'safer' vdev of 6 drives and expand later on with another vdev of 6 drivers.

To answer your question with pure logic and dispassion: 22TB will be the absolute maximum as of now. And growth rate? Seriously: I don't know.

However, you can't just answer that question with pure logic. Humans aren't Vulcans. And to be honest: It feels good to have some free space! It makes me happy. It's like having more food on your plate than you actually will eat. It just feels so good that you can overeat and there is still something left.

So to make a long story short: I'll build a wall of drives to make my storage great again! It will be so yuge! (And I wish I could make Mexico pay for it...)

 

[Stux]

Btw, you are not backblaze.

I would recommend NAS drives. Either seagate or wd.

Anyway.

Remember you should be planning your next expansion when you hit 80% usage and you should be implenting it when you get to 90%.

The pentium certainly sounds like the right CPU for your projected requirements,beware though, some people end up using their FreeNAS systems for a lot more than they plan. The good news is you can simply upgrade your CPU if that happens.

Beware that a single vdev will have half the IOPs of two vdevs.

 

[Stux]

What is your backup plan?

 

[Yoshimo]

Btw, you are not backblaze.

You are neither.

But Backblaze ist Backblaze and their drive stats have an evident message when it comes to drive selection.

What is your backup plan?

May backup plan is switching from status quo with no backups at all to RAIDZ3.