Starting to look towards a build...researching hardware.

[wynnefraser]

I've been running a Synology RS812 for a number of years now, but hitting the limit of maximum volume size of 16TB and starting to look at my next option and where to go next for my storage needs. Mainly long term video file storage (I work in the television industry and have a large amount of ProRes HD footage that I was to keep) and also a collection of TV Show/Movies that I want to watch across my local network on my laptop and some small Intel Skeleton computers on TV's that I've got.

Starting to look towards the possibility of building a 2U case with 12 Hard Drive bays, to format with two volumes of 6 drives setup with Raidz2. Starting off with a single 6 drive volume to begin with, and eventually expand the storage with another set of 6 drives.

From my research so far, and having had Asus computers previously and a good history with their gear I'd like to use them where possible. This is the hardware list that I've come up with so-far...prices are from Scropion Technology in Australia (I'm in Melbourne)...

Silverstone RM212 2U Rackmount Case - $599
Asus P10S-M WS Motherboard - $370
HyperX Fury HX424C15FB2K2/16 16gb Ram - $259

Would like the thought of running the Freenas off 2x USB keys, if it could be setup to run redundantly too in-order to allow fall-over if one of them were to fail eventually. Looking for thoughts and feedback from those who have done it all before me...

 

[kdragon75]

I can't comment on the hardware side much but the HyperX Fury RAM is a bit silly. If you buying new hardware get ECC memory. For FreeNAS and video archival/streaming/transcoding ANY DDR4 will be fast enough. It make more sense to get the RAM that will ensure end to end file integrity.

As for running from two USB keys, yes this is a common setup. You need to use quality keys, no super cheap deals. That said my USB drive of choice is the Kingston Datatraveler SE9. I use these for FreeNAS, ESXi, SAN, and portable Linux booting and have never had one fail.

 

[LIGISTX]

Without being at a computer it will be hard to give you exact parts to use, but the hard drive array looks like, usually WD Reds are the drive of choice, but the layout is fine.

As far as “using asus” where possible, I have had great success with them in MANY desktops over the years, but when it comes to server hear, honestly, throw away your brand loyalties. Supermicro is the board manufacturer of choice, hands down. Not that I’m sure the asus board isn’t a great board, but there is a reason every production server on the planet uses supermicro. Maybe I am being a little extreme, but, trying to get my point across ;)

And same with RAM, you want to go the ECC route, and not hyperX or Corsair LPX or anything that is of that sort especially. While ECC is not 100% necessary, if you buying components might as well buy ECC. It’s not that much more expensive, and over the lifetime of the FreeNAS box it doesn’t cost all that much extra.


Sent from my iPhone using Tapatalk

 

[Redcoat]

Welcome to the forum.

First a suggestion - if you haven't already done so, go here in the Resource section (tab on the masthead) and get the Recommended Hardware Guide. Also, as you are in Oz, look here for a "local" build report by Stux, one of our prolific contributors, which I think may give you some pointers and potential links to further useful reading.

I took a quick look at your mobo specs - while it supports ECC memory, you don't seem to have chosen it. You may have decided that you don't mind the effect of bit-rot for video files, I don't know, but I believe that most here would urge you to start off with ECC memory. I can't comment on the mobo choice and I know I have seen comments here in the past about availability/pricing of some hardware in Oz. If Stux chips in here it's likely he'll make some comments on that for you.

While many here report good performance of USB sticks as boot devices, as many others do not, me included. Short life of even quality sticks is a frequent experience. So many of us use small SSD's for this duty and choose not to mirror them as their reliability is so high. If you decide to research it through the forum posts here you'll see it is still a passionately-argued topic.

Good luck with your project. I hope you'll share your progress with us as it unfolds.

 

[kdragon75]

You may have decided that you don't mind the effect of bit-rot for video files

Bit rot is still largely mitigated by ZFS without ECC but yes, EEC should be used on all new builds. ECC only prevents "bit-rot" in RAM. ZFS will still detect and correct* on platter, cable, and controller rot.

*ZFS uses checksums to detect and either parity or copies to correct data errors. Copies may be on mirrored disks or if copies=2 that copy may reside on the same disk providing limited ability to recover from bit rot in single disk systems i.e. laptops.

 

[Chris Moore]

Starting to look towards the possibility of building a 2U case with 12 Hard Drive bays

Even if you only want 12 drives, you should go with a 4U chassis so you don't need the small diameter high speed fans that are loud. There is really no reason to use 2U except in certain extreme circumstances in the datacenter. A 4U unit will allow larger, slower fans so it can be quieter and still have adequate cooling to keep your drives cool. If you keep the drives cool, the rest should be easy.

 

[Chris Moore]

prices are from Scropion Technology in Australia (I'm in Melbourne)...

You might want to add that detail to you "Location" because

Code:

Location:
Lilydale

comes up as being in the USA when I click on it. It makes a difference, because it is difficult to get good prices on hardware in Australia and many people have chosen to order equipment from eBay in the US and have it shipped to Australia because it is less expensive even with the cost of shipping using a company like this: https://www.shipito.com/en/

I would actually suggest using older generation equipment to save significant money on the build. There is no reason to use the latest gen hardware for FreeNAS.

 

[wynnefraser]

Thank you everyone for all the replies,

Ordered the 2U Silverstone RM212 case today, decided to go smaller as the rack space is more important to me than the noise. Also it should be relatively cool as it’ll end up in an office under my house.

Looking at ECC Ram, does anyone have any recommendations on the best place to look at this online? I’ve been using Scorptec as my kind of go-to for gear for a while now, at they don’t seem to have anything listed under ECC, the HyperX ram that I originally mentioned was one of their lower cost options in terms of 16gb Ram.

 

[Redcoat]

at they don’t seem to have anything listed under ECC

what about here - https://www.scorptec.com.au/product/memory/ECC-&-Registered ?

 

[Chris Moore]

Thank you everyone for all the replies,

This is the kind of hardware I would suggest. It is much less expensive because it is not new but it is more than capable and highly reliable.

Intel Xeon E5-2650 V2 2.6GHz 8 Core
https://www.ebay.com/itm/SR1A8-Inte...8-Core-CM8063501375101-Processor/192524131095

!NEW! Unopened Super Micro Motherboard LGA 2011 MBD-X9SRL-F
https://www.ebay.com/itm/NEW-Unopened-Super-Micro-Motherboard-LGA-2011-MBD-X9SRL-F-O/253300204890

Samsung 16GB 2Rx4 PC3 12800R-11-12-E2-D3 M393B2G70QH0-CK0Q8 ECC Server Memory
https://www.ebay.com/itm/Samsung-16...B2G70QH0-CK0Q8-ECC-Server-Memory/232759440666

Total of about $450 and it is all server gear that will last for ages.

 

[Redcoat]

what about here - https://www.scorptec.com.au/product/memory/ECC-&-Registered ?

Hmmm... I took a look and they don't have anything listed that's on the QVL for the motherboard you listed. In fact that board has a very limited QVL for ECC and the supplier options are few.
I'd be very tempted to follow Chris's suggestion above if the importation process is manageable. I know that I've seen Australian users discussing buying from the US before, as, obviously, has Chris.

 

[wynnefraser]

Right... I'm looking at the Super Micro motherboards. Think I've found a descent one.

Thinking this maybe?
Super Micro X10SRA (Maybe -F so that it would have an on-board VGA output?)
https://www.scorptec.com.au/product/Motherboards/Server-&-Workstation/58265-X10SRA

With a stick of this to begin with (and upgrade as I add a few more hard drives along the way).
Crucial 16gb 2400mhz ECC Ram
https://www.scorptec.com.au/product/Memory/ECC-&-Registered/67327-CT16G4RFD424A

Still looking at the Asus motherboards though, seems to be a few compatible with ECC and their available a bit more locally...

 

[Chris Moore]

Thinking this maybe?
Super Micro X10SRA

No. That is a workstation board, not a server board. I already told you exactly what you need to buy and the total for it is less than the cost of just that board alone.

 

[Inxsible]

Silverstone RM212 2U Rackmount Case - $599
Asus P10S-M WS Motherboard - $370
HyperX Fury HX424C15FB2K2/16 16gb Ram - $259

That seemed expensive !!

Then I noticed you are in Australia, so I guess it figures.

Try and follow @Chris Moore's advice and buy older gen hardware... will turn out to be much cheaper. $599 AUD = $450.88 USD seems way way way too expensive (even for Australia) for a chassis that gives you only 12 drive bays.

 

[silversword]

Chris, I didn't find that Samsung RAM on the MBD-X9SRL-F MB QVL list either. How did you select that Samsung RAM specifically?

I'm trying to build a box, and my initial pricing for a modern x10 board just for the DDR4 RAM alone is over $1000, with another $900 for MB and E5 v4 CPU and is starting to blow the budget. I'm shooting for a 64TB Z2 array, so another $1600-2000 for 8 x 8TB drive and I'm pushing $3500-4000 so I'm considering scaling back to older hardware after seeing this post :) I was thinking 64GB RAM. I'm hoping for 3-5 jails running: UrBackup, a plugin running Plex, and some testing jails. Then 3 VM's running a linux, Win2012, and Mikrotik CHR. Plex will be the biggest CPU hit, everything else should be pretty light.

Any thoughts?

 

[Chris Moore]

Chris, I didn't find that Samsung RAM on the MBD-X9SRL-F MB QVL list either. How did you select that Samsung RAM specifically?

I'm trying to build a box, and my initial pricing for a modern x10 board just for the DDR4 RAM alone is over $1000, with another $900 for MB and E5 v4 CPU and is starting to blow the budget. I'm shooting for a 64TB Z2 array, so another $1600-2000 for 8 x 8TB drive and I'm pushing $3500-4000 so I'm considering scaling back to older hardware after seeing this post :) I was thinking 64GB RAM. I'm hoping for 3-5 jails running: UrBackup, a plugin running Plex, and some testing jails. Then 3 VM's running a linux, Win2012, and Mikrotik CHR. Plex will be the biggest CPU hit, everything else should be pretty light.

Any thoughts?

The reason is that this memory is a lower clock speed than the intended memory for the board. It will slow down the system a little bit but it saves money and the loss of performance is a non-issue for a NAS because it is usually waiting for the hard drives to do the mechanics of accessing the data.
Because of the number of VMs, and you want one of those to be Windows, you are likely to need even more memory than you think. What were you planning?
I actually use the same system board, CPU, and similar memory modules, if Irecall correctly.
This is a value oriented selection, but your VMs would like faster (lower latency) RAM in a higher density module. It would be better, also because you will likely want to be able to bring the system closer to the max memory capacity. You will need 64GB just to run FreeNAS and 2GB for each VM, except for Windows, which needs 4GB minimum.
Is it for home use?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[silversword]

Is it for home use?

Home based business. Some home, some work :)

Shoot, I was trying to restrict my choices with MB's with 8 slots, and 8GB DIMM's. So you're saying I'll probably need to go with 16GB DIMM's. Could I do 4 x 16GB DIMM's, and 4 x 8GB DIMM's? Since we're off the QVL already with the DIMM modules I'm concerned about hardware working, I haven't had the best of luck with getting DIMM's working/compatibility in the past, and mixing DIMM sizes gives me greater concern.

The VM's I'm thinking will be ok running a bit slower because it's just running NMS server software of several varieties. Network is probably more important than RAM/HDD read/write.

Also trying to make sure the MB can support future HBA's for external enclosures for a ZFS replica to an external enclosure in the future so at least 2-3 PCI-e slots for cards

 

[Chris Moore]

Shoot, I was trying to restrict my choices with MB's with 8 slots, and 8GB DIMM's. So you're saying I'll probably need to go with 16GB DIMM's. Could I do 4 x 16GB DIMM's, and 4 x 8GB DIMM's? Since we're off the QVL already with the DIMM modules I'm concerned about hardware working, I haven't had the best of luck with getting DIMM's working/compatibility in the past, and mixing DIMM sizes gives me greater concern.

I like to keep all the DIMMs the same brand and model to reduce the possibilities of problems. The thing to know about memory is that they can't test every stick of memory out there. They test some in order to assure buyers that this combination will work but if you purchase a quality memory stick that is of the correct type, it should work, no matter who made it.
You probably want to go with these 16GB modules: https://www.ebay.com/itm/282744574825
You can always buy 4 and come back and get 4 more later. You could do two at a time if the budget is really tight. Here is what the manual says:

Code:

The X9SRL Motherboard Series  supports up to 256GB of 1600/1066/1333/1600
MHz ECC/Non-ECC DDR3 DIMMs in eight (8) memory slots (UDIMM/RDIMM). 
Populating  these  DIMM  modules  with  a  pair  of  memory  modules  of  the  same
type and same size will result in interleaved memory, which will improve memory
performance.

Although the manual doesn't say it, the Supermicro website says that 1866MHz memory is also supported.
http://www.supermicro.com/products/motherboard/Xeon/C600/X9SRL-F.cfm

The blue slots should be loaded first. Here is a link to the manual:
http://www.supermicro.com/manuals/motherboard/C606_602/MNL-1317.pdf
The other option would be to go with the 32GB LRDIMM modules, but you probably won't need that much memory.
I wouldn't suggest mixing 4 x 16GB modules and 4 x 8GB modules as you mentioned, because it says this in the manual:

Code:

Always use DDR3 DIMM modules of the same size, type and speed.

There is a table on page 2-11 that shows the configurations that Supermicro recommends.

Also trying to make sure the MB can support future HBA's for external enclosures for a ZFS replica to an external enclosure in the future so at least 2-3 PCI-e slots for cards

That is one of the reasons I chose this board. It has 4 PCIe 3.0 slots that are 8x electrically, 2 more PCIe 3.0 slots that are 4x electrically and a 1 more PCIe 2.0 slot that is 4x electrically. I think that is enough for me. I can use a couple for HBAs and one for a 10Gb Ethernet card and have some extra for later.

 

[silversword]

That is one of the reasons I chose this board.

Great minds think alike ;)

Many thanks for all that detailed breakdown...can't tell you the number of nights I've lost going back and forth between MB's/CPU's/DIMM's and pricing etc. trying to put together something that's going to work for the next several years and my needs.