BUILD High Performance FreeNAS Build

Status
Not open for further replies.

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
"heavy load" equals a ZeusRAM as SLOG and many HDDs before bigger ones. and if you're making money off it, go for a TrueNAS. Z20 with single controller is good enough for a start, but the TrueNAS software already has patches without reboots integrated with business-level support.
 

Zach Puls

Dabbler
Joined
Mar 3, 2015
Messages
16
"heavy load" equals a ZeusRAM as SLOG and many HDDs before bigger ones. and if you're making money off it, go for a TrueNAS. Z20 with single controller is good enough for a start, but the TrueNAS software already has patches without reboots integrated with business-level support.

Oh I gave the general idea away, didn't I? :) I don't believe I am going to be making money off of it in the near future, I will want to stress test the heck out of it before I do that. I will need to beef up my network with some more Cisco Equipment beforehand, as well. Mainly this is going to be serving up VMs to friends who need development servers to start.

I did entertain the idea of getting a TrueNAS (iXSystems looks to be a great company with amazing products), but I just don't need the enterprise-level support, which is the main selling point of an appliance like that.

How much does ZeusRAM cost, on average? The only ones I could find were 8GB modules for around $1k.
 

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
No, enterprise support is not the only point of a TrueNAS vs. FreeNAS.

TrueNAS is a system where the "error" LEDs on the HDD bays work. They don't with FreeNAS. It's Dual Controller capable, which is as far as I know almost unique amongst ZFS based storage appliances. The software is guranteed to provide best experience without reboots. You have to reboot a FreeNAS completely for a software upgrade, you don't with a TrueNAS. That lowers the downtime even more.

There's only a big but there: you have to have all-SAS disks with a TrueNAS. No way around these. But for a VM workload I would go for 15krpm drives anyway... 300GB ones should be in the same price ballpark, but you get more than double IOPS out of the bay.
 

Zach Puls

Dabbler
Joined
Mar 3, 2015
Messages
16
No, enterprise support is not the only point of a TrueNAS vs. FreeNAS.

TrueNAS is a system where the "error" LEDs on the HDD bays work. They don't with FreeNAS. It's Dual Controller capable, which is as far as I know almost unique amongst ZFS based storage appliances. The software is guranteed to provide best experience without reboots. You have to reboot a FreeNAS completely for a software upgrade, you don't with a TrueNAS. That lowers the downtime even more.

There's only a big but there: you have to have all-SAS disks with a TrueNAS. No way around these. But for a VM workload I would go for 15krpm drives anyway... 300GB ones should be in the same price ballpark, but you get more than double IOPS out of the bay.

Wow, that is actually a pretty attractive option, then! I didn't know there were that many extra features of TrueNAS. I am getting in contact with an iXSystems sales rep, and will be exploring those options. While I do have quite a bit of money saved up to buy great hardware, I am not made of money here :D the TrueNAS option may end up being too expensive for me.
 

Zach Puls

Dabbler
Joined
Mar 3, 2015
Messages
16
Any recommendations on how to make sure I get close to saturating the 10GbE link? I know 15K RPM drives were recommended, but there are some 256GB SSDs that are a bit cheaper. Would those work to increase throughput?
 

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
SAS SSDs are quite expensive. I'd stick to 15krpms. You can saturate your 10Gbe link with sequential transfers, but random I/O will drop to around 1-2Gbps.
 

Zach Puls

Dabbler
Joined
Mar 3, 2015
Messages
16
Thanks to everyone who replied for your help, I really do appreciate it. I have ordered the parts, they should be here within a couple of weeks. Would anyone be interested in seeing pictures of the completed build/performance tests? I know I've read through a lot of the build threads, and have wanted to know how they turned out.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,543
SAS SSDs are quite expensive. I'd stick to 15krpms. You can saturate your 10Gbe link with sequential transfers, but random I/O will drop to around 1-2Gbps.
Assuming you have the following conditions:
A large case with lots of drive bays
A need for IOPS & large storage capacity
Couldn't you just add more mirrors with 7200RPM drives? For instance, you can get 2x 2TB WD Red Pros for the cost of a single 600GB 15k drive. If you use 2.5" 7200 rpm drives you can get some crazy densities in a 4U (with oodles of iops).

Just throwing it out there.
 
Joined
Oct 2, 2014
Messages
925
Thanks to everyone who replied for your help, I really do appreciate it. I have ordered the parts, they should be here within a couple of weeks. Would anyone be interested in seeing pictures of the completed build/performance tests? I know I've read through a lot of the build threads, and have wanted to know how they turned out.
I could use some photos, this sounds like a nice build :) right up my ally
 

Zach Puls

Dabbler
Joined
Mar 3, 2015
Messages
16
Due to an issue with SuperBiiz not liking my payment method, it looks like I'm going to end up with 96GB of RAM in the end :p
 

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
Assuming you have the following conditions:
A large case with lots of drive bays
A need for IOPS & large storage capacity
Couldn't you just add more mirrors with 7200RPM drives? For instance, you can get 2x 2TB WD Red Pros for the cost of a single 600GB 15k drive. If you use 2.5" 7200 rpm drives you can get some crazy densities in a 4U (with oodles of iops).

Just throwing it out there.
The IOPS with 15krpm disks are still near-triple than with Reds and more than double than 7200rpm disks. Yes, you can scale out beyond belief, but at some point you would prefer faster disks in less bays if you don't need much storage anyway.

Still: a full rack of 15krpm drives doesn't come close to a small all-flash array in IOPS.
 
Joined
Oct 2, 2014
Messages
925
I wouldnt mind some 15K drives....or some purty SSD's, my homelabs SAN gets some nice speed out of 10GB + x2 RAID 10's of 8 10K drives each. I'm trying to get work to give me a shelf of SSD's for testing and pure amusement :P
 

Zach Puls

Dabbler
Joined
Mar 3, 2015
Messages
16
Everything trickled in this week, and I finally got to start assembling this last night. Didn't get finished, as it got kinda late and I need a longer 8087-8087 cable that should be coming in today.

The first picture is of the first couple of shipments that came in, then there's a couple of the interior of the case, which is wonderful to work in. Last one is of the installed motherboard, CPU, and RAM.

I was a little rusty on my building (Haven't built a desktop in awhile), so it took me a bit to get back in the swing of things. This is by far the most fun I have had building a computer, just knowing what the results will be.
 

Attachments

  • IMG_20150407_144215544.jpg
    IMG_20150407_144215544.jpg
    264 KB · Views: 255
  • IMG_20150409_233318532.jpg
    IMG_20150409_233318532.jpg
    365.2 KB · Views: 251
  • IMG_20150409_233346292.jpg
    IMG_20150409_233346292.jpg
    327.2 KB · Views: 255
  • IMG_20150410_244149480.jpg
    IMG_20150410_244149480.jpg
    421.4 KB · Views: 241
Joined
Oct 2, 2014
Messages
925
mhmmmmmm server parts......
 

Attachments

  • homer-simpson-drooling-a.jpg
    homer-simpson-drooling-a.jpg
    85.8 KB · Views: 227

Zach Puls

Dabbler
Joined
Mar 3, 2015
Messages
16
A couple of differences from my original parts list, I ended up going with 96GB of RAM instead of 64, and I got an E5-1650v3 after seeing jgreco recommend it in quite a few different threads.

I got it booted up for the first time, and was ecstatic when it POSTed! That was surprising, as I almost always forget some minor detail and have to go back and check my work. One strange thing though, while it is booting up, the fans will speed up to full throttle, then slow down to no spinning. After a few seconds, they will speed up again, and repeat the cycle indefinitely. Looking around on Google, I am seeing that this is either indicative of a faulty PSU or a short somewhere on the motherboard, so I am going look around for loose screws or somewhere it has shorted. Any ideas on why this is happening, or is it normal during bootup? I haven't run it for more than a couple of minutes so far.
 
Joined
Oct 2, 2014
Messages
925
Are you using the fan headers on the motherboard? Is the BIOS up to date?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
A couple of differences from my original parts list, I ended up going with 96GB of RAM instead of 64, and I got an E5-1650v3 after seeing jgreco recommend it in quite a few different threads.

I got it booted up for the first time, and was ecstatic when it POSTed! That was surprising, as I almost always forget some minor detail and have to go back and check my work. One strange thing though, while it is booting up, the fans will speed up to full throttle, then slow down to no spinning. After a few seconds, they will speed up again, and repeat the cycle indefinitely. Looking around on Google, I am seeing that this is either indicative of a faulty PSU or a short somewhere on the motherboard, so I am going look around for loose screws or somewhere it has shorted. Any ideas on why this is happening, or is it normal during bootup? I haven't run it for more than a couple of minutes so far.

Most likely, you need to adjust the IPMI thresholds. Linky.
 

jwigg

Cadet
Joined
Nov 28, 2014
Messages
2
How has the final outcome of your build go? Any stats? I've been looking around for a while on putting a decent nas together. This build looks pretty good. From what you've experienced, would you have changed anything that isn't mentioned up to this point? Curious to know if this had been working perfectly for you. I would like to run vm's through iscsi (websites, SQL, applications). I would be putting in more smaller 10K or 15k drives instead of larger for the vm's and whatever SQL is running on. Larger drives just for storage on separate volume. From your build, how do you think it would work out? Any suggestions?
 
Status
Not open for further replies.
Top