Mini-ITX FreeNAS build

[shutch]

Hi all,

I am looking to put together a Mini-ITX build to act as a storage pool for my Proxmox VM nodes (i.e. share one data pool to each node, will be about 8 VMs across two nodes sharing the storage). I am thinking I will probably run this in raidz2. Your thoughts on the following build for this purpose would be much appreciated.

Motherboard - SuperMicro X11SDV-4C+-TLN2F (I know the model number is slightly different but I will buy from here as I am in Canada)
RAM - 2x8GB Kingston KVR24R17S8/8 (RAM is expensive at the moment?)
OS drive - Samsung 860 Evo 250GB
Storage drives - to start, 4x1TB WD Gold or 4x1TB WD Red or 4x1TB RE
Case - Fractal Node 804 or Silverstone DS380 (for hot swap)
PSU - Corsair SF600

I am not going to get an HBA SAS card at the moment as this won't fill the SATA ports on the motherboard yet.

Thanks

 

[HoneyBadger]

Mini-ITX will significantly increase your cost as I'm sure you've discovered; unless you absolutely require this form factor, I'd suggest building a physically larger system, even Micro-ATX. You could likely obtain a complete system (possibly even including drives) for the cost of just that Supermicro ITX board.

For your pool configuration I would strongly recommend mirrors for VM performance reasons, as well as using a smaller number of larger drives (2x4TB) which gives you better options for expansion down the road.

 

[shutch]

Mini-ITX will significantly increase your cost as I'm sure you've discovered; unless you absolutely require this form factor, I'd suggest building a physically larger system, even Micro-ATX. You could likely obtain a complete system (possibly even including drives) for the cost of just that Supermicro ITX board.

Thanks - I could go to Micro-ATX or ATX without an issue. I really only need an 8-12 bay so originally thought a small form factor would be better. But if it is a lot cheaper, I will look into Micro-ATX. Any board / processor recommendations for that?

Perhaps the Supermicro X11SSH-LN4F with Xeon E3-1225 v6? Although doesn't seem much cheaper than the ITX alternative.

For your pool configuration I would strongly recommend mirrors for VM performance reasons, as well as using a smaller number of larger drives (2x4TB) which gives you better options for expansion down the road.

Thanks. As in 2 vdevs, each with the 2x1TB mirrored and then stripe the 2 vdevs?

 

[HoneyBadger]

Thanks - I could go to Micro-ATX or ATX without an issue. I really only need an 8-12 bay so originally thought a small form factor would be better. But if it is a lot cheaper, I will look into Micro-ATX. Any board / processor recommendations for that?

In the word of Micro-ATX/ATX you can easily pick up a second-hand X9/X10 setup for much less money, which is where I'd lean, since you can at least find DDR3 ECC somewhat less expensively. (If you manage to luck into a setup that supports RDIMMs, you're rolling in cheap 8GB sticks.) @Chris Moore usually has good suggestions, although your location in Canada might mean you need to do a bit more hunting and poking around the bowels of eBay to match the prices. (Or have a trusted friend stateside do the buying, or use a cross-border parcel relay service?)

Thanks. As in 2 vdevs, each with the 2x1TB mirrored and then stripe the 2 vdevs?

Yes; ZFS spreads writes across multiple vdevs similar to a "stripe" so more vdevs = more IOPS generally.

I'd recommend sizing up to 2TB or larger drives at least, as the $/GB value should be better.

 

[Chris Moore]

@Chris Moore usually has good suggestions,

You called my name, here I am.
The biggest point I make in most of my hardware suggestions is that the difference in performance, generation to generation has been very minimal the last few years. It costs a lot of extra money to go with new, and does not gain you much at all. FreeNAS does not need the latest hardware and having the latest hardware can actually mean that you have hardware that is not properly supported. Another thing to think about is the staggering cost of new DDR4 memory. It is substantially less expensive to buy referb (used) DDR3 registered ECC memory as I have outlined below.
I am sure that this is not all to your taste, especially since I didn't take time to really look at your plan, but this might give you some ideas.

This is a buy list I put together for someone else recently. You might want to take a look at it and see if it is interesting to you:

Some assembly required...

(for hot swap)

CASE: Supermicro 24 Bay Chassis SAS846TQ Server 4U - - US $429.99 + $99.99 Standard Shipping
https://www.ebay.com/itm/202415880869

Code:

you won't need this with that SuperChassis, but the build plan was originally for...
CASE:  Fractal-Design-Define-R5-FD-CA-DEF-R5-BK-Black-Silent-ATX-Midtower-Computer-Case  -   -  US $89.24
https://www.ebay.com/itm/253026336681

POWER:  Corsair-Certified-CS-M-Series-CS650M-650W-80-Plus-Gold-Active-PFC-Modular-Power  -   -  US $64.99
https://www.ebay.com/itm/382130407495

These are older components, but still very powerful.
I use similar components in both my 48 bay primary NAS and in my 24 bay backup NAS that runs ESXi with FreeNAS in a VM.

This one comes with a CPU and fan, but it is really slow, so probably want to replace it:

System Board: Supermicro X9SRL-F Motherboard LGA2011 System Board w/Intel E5-2650L 0 @ 1.80GHz - - US $174.99
https://www.ebay.com/itm/132762499937

Nothing included with this one, so the price is lower, but an option:

Supermicro X9SRL-F Motherboard Socket LGA2011 System Board w/ I/O Shield - - US $161.49
https://www.ebay.com/itm/401593992194

Memory: SAMSUNG 16GB PC3L-12800R DDR3-1600 ECC Registered 1.35V RDIMM - - US $44.95
https://www.ebay.com/itm/302606459277

Note: You could go for the 32GB memory modules, but they are more than twice the price.

If the board above doesn't come with a cooler, or you want a better one, I use this model on two of my systems.
It is only slightly louder than the Noctua cooler I have on my wife's desktop PC.

CPU Fan: Dynatron R27 Side Fan CPU Cooler 3U for Intel Socket LGA2011 (Narrow ILM) - - US $39.59
https://www.ebay.com/itm/401284811045

The CPU that comes in the board above is pretty low speed, so you might find that you need a better one.
You have a lot of options for CPUs to go in this board, but I recently bought one of these for myself:

PassMark score of 13073... If you are wondering... This is the model I use for the NAS I run Plex in.

CPU: Intel Xeon E5-2650 V2 2.6GHz 8 Core 20MB 8GT/s SR1A8 LGA2011 ( Ivy Bridge ) Processor - - US $89.97
https://www.ebay.com/itm/142937685210

Only it was $30 more when I bought it... It works great. Plenty of resources for all the things I am doing.

If you want more, you can get a 10 core model like this:

Intel Xeon E5-2680 v2 Ten Core 2.8GHz 25MB CPU PROCESSOR LGA2011 SR1A6 - - Price: US $189.99
PassMark score of 15796... If you are wondering...
https://www.ebay.com/itm/192637399687

For the drive controller, I would suggest a SAS controller, just to get all the drives on a single controller. It works better that way. One SAS controller like this can run up to 256 drives by use of expander controllers. We can talk more about that when you need more drives but this will get you to eight drives to start.

Drive Controller: SAS PCI-E 3.0 HBA LSI 9207-8i P20 IT Mode for ZFS FreeNAS unRAID - - US $69.55
https://www.ebay.com/itm/162862201664

Drive Cables: Mini SAS to 4-SATA SFF-8087 Multi-Lane Forward Breakout Internal Cable - - US $12.99
https://www.ebay.com/itm/371681252206

I would suggest one of these SSDs for the boot drive.
These are used data-center drives, but as a boot drive in FreeNAS, it should last as long as the server, if not longer:

Boot drive: Intel SSD DC S3500 Series 2.5" SATA 6Gb/s, 20nm, MLC 80GB - - US $29.99
https://www.ebay.com/itm/273102509397

Thermal Compound: Noctua NT-H1 Thermal Paste Grease Conductive Compound for CPU/GPU - US $6.95
https://www.ebay.com/itm/302624513215

Just rough math in my head, I think that is all around $1100.... Still, you might need some odd bits, and hard drives, but it should be simple to get there from here and this should save you a buck or two vs buying new and still do the job for years to come.

PS. If you want to buy the expanders cards now, this is the product I am using:

SAS Expander: IBM (46m0997) ServeRAID Expansion Adapter 16-port SAS Expander - - US $15.88
https://www.ebay.com/itm/IBM-46m0997-ServeRAID-Expansion-Adapter-16-port-SAS-Expander/192326557905

I have two of those and run 12 drives from each. The cable to connect between the SAS expander and the SAS controller is this type:

SAS interconnect: Mini SAS SFF-8087 to Mini SAS SFF-8087 - - US $3.49
https://www.ebay.com/itm/183129085895

This gives you room to grow for the next three to six years.

 

[shutch]

Thanks very much @Chris Moore, very helpful!

After a lot of reading I think I am going to go with either an X10SLH-F or X10SL7-F as they seem to be the best and most recent DDR3, Xeon mATX boards.

Any thoughts on if better to go with X10SL7-F or get an HBA LSI card for the X10SLH-F?

Thanks

 

[Chris Moore]

The most significant price advantage is with Registered ECC. Both of those boards are NOT able to work with Registered ECC, which usually requires a Xeon E5 processor. They are also not mini-ITX. I picked the hardware in my list for very specific reasons. I prefer to not have the SAS controller integrated in the system board, that way it can be change out later when better technology comes along, without requiring a new system board.

 

[shutch]

The most significant price advantage is with Registered ECC. Both of those boards are NOT able to work with Registered ECC, which usually requires a Xeon E5 processor. They are also not mini-ITX. I picked the hardware in my list for very specific reasons. I prefer to not have the SAS controller integrated in the system board, that way it can be change out later when better technology comes along, without requiring a new system board.

Thanks again @Chris Moore. The X9SRL-F is full ATX. Anything in MicroATX with DDR3 registered supported?

 

[Stux]

E5 class processors generally call for ATX size boards I’m thinking. mATX has limited PCIe and DIMM slots and those are specifically one of the benefits of E5

On a different tack, take a look at my mini itx x10sdv build ;)
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[shutch]

E5 class processors generally call for ATX size boards I’m thinking. mATX has limited PCIe and DIMM slots and those are specifically one of the benefits of E5

On a different tack, take a look at my mini itx x10sdv build ;)
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

Very true! Thanks.

@Chris Moore i purchased most of the items you suggested on eBay. But what do you think of this instead? Thanks!

 

[Chris Moore]

@Chris Moore i purchased most of the items you suggested on eBay. But what do you think of this instead? Thanks!

It has a Dell H710 hardware RAID controller that you will have to rip out and discard, and Dell usually has some strange, proprietary tricks in their systems, but it should be able to do the job. I just don't like 1U and 2U systems in general, regardless of manufacturer, because the fan noise they make is usually louder and higher pitched which bugs me when I have to work around them. I have almost eliminated that form-factor from my facility at work and the last of them will hopefully be out next year. Then I will only have 3U and 4U systems, which are usually, but not always less painful to hear.
You need a proper SAS HBA in that system instead of the H710. Did you already get the "LSI 9207-8i P20 IT Mode" that I suggested above?