BUILD High Performance FreeNAS Build

[Zach Puls]

Hello everyone!

I have been reading through quite a few threads over the course of the past few months, and have slowly been piecing together a build, and just need a quick sanity check before I pull the trigger on these items.

The use case for this build is going to be somewhat of a SAN appliance, I am looking for a high-performance box on which to store VMs.

Here is what I have so far, with relevant links/questions added:

Chassis: SuperMicro SC846E16-R1200B

-From eBay, $319.99
​

Motherboard: SuperMicro X10SRL-F

-From Newegg, $269.99​

CPU: Intel Xeon E5-1620 v3

-From Newegg, $307.99
​

Memory: 4 x Samsung DDR4-2133MHz 16GB RDIMM

-From SuperBiiz, $179.99 ea ($719.96)
​

NIC: Intel X540T2

-From Newegg, $484.99
​

ZIL (SLOG): 1 x 200GB Intel DC S3710 SSD

-From Newegg, $327.00
​

L2ARC: 1 x Plextor M6e 128GB PCIe SSD

-From Newegg, $199.99​

Hard Drives: 4 x HGST DeskStar NAS 4TB

-From Newegg, $164.99 ea ($659.96)
-This is just a start, I will be buying more hard drives later on to stagger the hits to my bank account :) I can wipe and rebuild the array at any time until I get more drives, I keep all of my crucial data offsite.
​

Boot Device: Innodisk 32GB SATA DOM

-From Directron, $50.98
​

Thanks in advance for taking the time to look over my planned build and provide feedback, I look forward to seeing what everyone thinks. I am open to any suggestions.

 

[Mlovelace]

Your slog device needs to be power loss protected. Look at the Intel DC S3700 ssd.

Edit: also your l2arc shouldn't exceed ~5:1 ratio of your ram so shoot for around 200GB or add more ram. :D

 

[Zach Puls]

Thank you for the feedback! I will take a look at the Intel Taylorville series SSDs, and of course will be getting a UPS as well. On the L2ARC, are you saying it is too small? The L2ARC I have planned is 128GB, and I have 64GB of RAM listed there.

 

[SweetAndLow]

Did you price out the same build using newer hardware, X10SRL, 1620V3? It might turn out to be about the same price for new components.

 

[Mlovelace]

Thank you for the feedback! I will take a look at the Intel Taylorville series SSDs, and of course will be getting a UPS as well. On the L2ARC, are you saying it is too small? The L2ARC I have planned is 128GB, and I have 64GB of RAM listed there.

I am sorry I miss read your l2arc spec (my 3yr old was literally jumping on my head) the 128GB is fine size-wise. I did notice the sata Dom you linked is for a PCI Dom not a SATA Dom

 

[Zach Puls]

Did you price out the same build using newer hardware, X10SRL, 1620V3? It might turn out to be about the same price for new components.

I have priced it out for the newer hardware, and the motherboard/cpu works out to be the same. From what I have seen, though, the price for DDR4 is quite a bit higher than the DDR3 equivalents.

 

[Zach Puls]

I am sorry I miss read your l2arc spec (my 3yr old was literally jumping on my head) the 128GB is fine size-wise. I did notice the sata Dom you linked is for a PCI Dom not a SATA Dom

Ah, good catch! I have updated the original post with an acual SATA DOM this time :)

 

[SweetAndLow]

I have priced it out for the newer hardware, and the motherboard/cpu works out to be the same. From what I have seen, though, the price for DDR4 is quite a bit higher than the DDR3 equivalents.

About $40 less per 16gb stick. Not much if you ask me but still less in the end.

 

[Zach Puls]

About $40 less per 16gb stick. Not much if you ask me but still less in the end.

Oh, that is quite a bit less of a difference than I was thinking. For some reason, I was seeing a $150-200 difference per stick. I will update the build when I have a moment to re-spec those.

 

[SweetAndLow]

I think they go for 180-195 each, either Samsung or crucial.

 

[Mlovelace]

If you decide to go with the 2011-3 gen board make sure the HBA you use is SAS2. There have been issues reported in freeNAS when a SAS3 HBA is connected to a SAS2 backplane, which is what's in that SM chassis.

 

[marbus90]

"high performance" and WD Red doesn't mix. Go HGST NAS, they're 7200rpm. Or all the way with 15krpm drives.

SLOG either said S3700, the S3710 (may not be available yet) or ZeusRAM.

 

[Zach Puls]

Edited OP to include suggested changes.

 

[HoneyBadger]

Can you describe the client connectivity/workload you plan to put on this?

The low initial number of vdevs will be an issue under sustained write loads with 10Gbps in play. If you plan to expand to all 24 bays down the road this will be mitigated but with only four spindles in two mirror vdevs you're going to have to artificially limit the writes somehow.

Mirrored SLOG might be the only other thing to add. I might even say two 100GB S3700s as that will be roughly the cost of a single 200GB S3710 and won't hurt performance much as it's still a bloody fast drive.

 

[marbus90]

You don't need to wipe and restore. Striped mirrors are the way to go for performance -> adding 2 drives for another mirrorset is enough. The pool will automatically expand after adding the new vdev.

SLOGs don't need to be mirrored either since the data always exists in RAM as well. Only in case you have a power loss (unlikely with redundant PSUs and a UPS) _and_ lose the SSD at the same time (likely to happen with both SSDs) you will lose five seconds worth of data.

Btw, performance ballpark? For random I/O I wouldn't expect much more than Gbps speeds if you have all 24 bays full.

 

[Zach Puls]

Can you describe the client connectivity/workload you plan to put on this?

The low initial number of vdevs will be an issue under sustained write loads with 10Gbps in play. If you plan to expand to all 24 bays down the road this will be mitigated but with only four spindles in two mirror vdevs you're going to have to artificially limit the writes somehow.

Mirrored SLOG might be the only other thing to add. I might even say two 100GB S3700s as that will be roughly the cost of a single 200GB S3710 and won't hurt performance much as it's still a bloody fast drive.

I plan on getting quite a few more drives soon, just need to let my next huge two-week paycheck come in from working 80 hour weeks ;)

You don't need to wipe and restore. Striped mirrors are the way to go for performance -> adding 2 drives for another mirrorset is enough. The pool will automatically expand after adding the new vdev.

SLOGs don't need to be mirrored either since the data always exists in RAM as well. Only in case you have a power loss (unlikely with redundant PSUs and a UPS) _and_ lose the SSD at the same time (likely to happen with both SSDs) you will lose five seconds worth of data.

Btw, performance ballpark? For random I/O I wouldn't expect much more than Gbps speeds if you have all 24 bays full.

That makes sense, I do remember reading that in the noob guide (in the six or seven times I've read through it). As for performance, I am hoping to get close to saturating 10GbE. I know that is asking for way too much, and honestly I would be more than happy if I used 50% of the 10GbE capacity.

 

[anodos]

Don't forget to budget for backups. Racecars need roll cages. :)

If you need speed and redundancy, you can consider 3-way mirrors. :)

 

[HoneyBadger]

I plan on getting quite a few more drives soon, just need to let my next huge two-week paycheck come in from working 80 hour weeks ;)

Ah, I don't miss those days.

As for performance, I am hoping to get close to saturating 10GbE. I know that is asking for way too much, and honestly I would be more than happy if I used 50% of the 10GbE capacity.

What's the workload/protocol you plan to use here? Standard file sharing? Backups? VM hosting?

 

[Mlovelace]

As for performance, I am hoping to get close to saturating 10GbE. I know that is asking for way too much, and honestly I would be more than happy if I used 50% of the 10GbE capacity.

Have you had a look at the 10gbe primer? There is some good info in there.
https://forums.freenas.org/index.php?threads/10-gig-networking-primer.25749/

 

[Zach Puls]

Don't forget to budget for backups. Racecars need roll cages. :)

If you need speed and redundancy, you can consider 3-way mirrors. :)

I am planning on using Backblaze for off-site backups, so I've got my seatbelt and roll cage :) ahthis brings back memories of when I was younger driving go-karts. Not fun when they don't actually have seatbelts or a roll cage.
As for the 3-way mirror, that sounds interesting. I'll do some reading on that.

What's the workload/protocol you plan to use here? Standard file sharing? Backups? VM hosting?

I will be using iSCSI, for VM hosting. Without getting into too much detail, the VMs will be used by many clients under heavy load.

Have you had a look at the 10gbe primer? There is some good info in there.
https://forums.freenas.org/index.php?threads/10-gig-networking-primer.25749/

I've read that one, it is a pretty good read. I've read through most, if not all, of the stickies on the help/support forum. There is a ton of great information everywhere around here.