BUILD Advice on hareware for new build configuration and scope for expansion.

[Lorin Zahra-Newman]

I've been using FreeNAS for over a year now on two different machine builds I jimmied together and cannibalized from some spare machines I had lying around. It's a really nice system and i've enjoyed poking around with it over the previous months. One of my builds has been working like a dream running a 9TB pool with only 6GB of RAM but the other build has been giving me headaches with its phantom shutdown and reboots. Anyways, im not surprised as I threw these machines together and virtually broke every recommendation in Cyberjock's hardware guide in the process.

I'm now ready to outlay some serious cash on a proper build and would like the communities advice on my selection of hardware and scope for expansion further down the track.

I currently have ~24TB of data to store and expect it to grow to 50-60TB in the next five years. This is primarily media files that have been either downloaded or digitized from our hard copies. My father is an audiofile so compressing media is a big sin when it comes to listening to it on his fancy HiFi System... I think there's about 3000 CDs still left to rip down lol

Backup is not a big issue but I do plan to build a few smaller scaled versions to this one and keep them offsite at my own and siblings place and use something like btsync to split the data amongst them kind of like a poor mans parity with related data we use. It's media so no big cry if it gets lost.

Intended configuration is as follows:

Lian Li PC-D8000 20 Bay Chassis
Thermaltake Toughpower XT 1275W 80Plus GOLD PSU
Supermicro X9SRH-7F ATX Motherboard
Intel® Xeon® Processor E5-1620 v2
Kingston 32GB Module - DDR3 1600MHz ValueRAM x2 =64GB
Intel® RAID SAS Expander RES2SV240

I haven't made a decision as to hard disks yet but am probably looking at an array of WD 6TB Red's. My main concern is to get the main backbone of the system up and transfer my existing HD's (4TB WD Reds) into it while I wait around and see what happens with these 10TB disks that are meant to be around the corner.

I'm pretty happy with my choices made but the only thing I wasn't sure about was the PSU and calculating my power consumption, so I elected to just put a whopper of a PSU in.

Thanks for your advice guys.

 

[marbus90]

Personally I'm not a big fan of those huge, custom-made chassis where each disk needs its own SATA data + power cable. It can get quite bothersome after a while.

You'll find that the 3TB HDDs have the lowest price per TB on tradidional HDDs (Seagate Archive 8TB are in the same ballpark, but SMR drives - heavily debated at this point). It might pay off to go with a smaller server-grade chassis which still holds 45 HDDs just fine. A good example with a rather quiet chassis is the 45drives.com model, available in 30/45/60bay variants. You can also upgrade in smaller increments here whilst achieving higher redundancy levels (4x 11disk raidz3 + 1 spare vs. 2x10disk z2), which are preferable when you upgrade to bigger HDDs, due to increasing resilvering times.

30 or 45bay chassis, no PSU, the 2-3 included backplanes, 6 of the fans plus optionally the vibration dampening foam. Power button? IPMI ;)
According to this article you won't need a huge PSU: http://45drives.blogspot.ca/2015/05/the-power-behind-large-data-storage.html
I'd add another 20A on top for the mobo/CPU/HBAs/overprovisioning and it seems that the Seasonic X-850 fulfils those requirements at 70A@12V + 25A@5V, altough the 1050W one isn't much more and is still in the range with high efficiency.
Supermicro X10SRL-F
Xeon E5-1620 v3 or E5-1650 v3
2-4x16GB Samsung M393A2G40DB0-CPB DIMMs
2x64GB Supermicro SSD-DM064-PHI SATADOMs for FreeNAS boot
2-3x LSI 9201-16i HBAs
30-45 drives - as this is the exact same chassis used for the Backblaze HDD reports, I'd stick to those with a low failure rate in their statistics: https://www.backblaze.com/blog/best-hard-drive/
I'd start on 12 (1 vdev + 1 spare) and add more drives in 11disk batches. They're easily swappable without risk to disconnect other HDDs and risk pool inconsistency.

You can also downsize a little bit and go for the 30bay chassis with 4TB drives and only 2 LSI 9201-16i HBAs, but there you'll be stuck again at 10disk z2 vdevs without spares. If you can re-rip the media at any time, that's not a big problem then. Both 30x4TB and 40-45x 3TB gives you around 76TiB usable storage in that box.
Also this one would allow you to downscale to a Xeon E3 system with 32GB RAM, this should be good for 100TB of storage according to jpaetzel. As this isn't an active VM datastore but just cold archival storage, it may be preferable. The 45bay system would need all 3 PCIe slots of such a board where the 30bay would only need 1-2 slots. For that I'd go with one of the Supermicro X10SL*-F series, a Xeon E3-1220 v3 or E3-1241 v3 plus 4x8GB ECC DIMMs from the tested memory list.

Last but not least the 30bay variant would allow to use only 1 HighPoint Rocket 750 HBA, which shaves a little bit off the TCO with lower energy usage and a slightly lower price. It seems that Highpoint did their homework on that controller regarding the FreeNAS community. Some are heavily against this card, but then - nobody actually used it. Same goes for the Seagate 8TB HDDs, many are claiming they're not fit for NAS usage, others pioneered and didn't find many complications. Each 11disk raidz3 vdev (I wouldn't go smaller and really add 1-2 spares there) would give you ~50TiB of usable storage.