Building an around 72TB FreeNAS for an approx $9000

[FreeME]

I need to build a NAS for video storage and I'm wondering what everyone thought of this setup. I'm not going to stream a lot from it. This is just for overflow storage. I do want to have some descent performance from it just in case we decide to have a couple users edit from it. I'm not really looking to build an ok Atom system. I need something that's going to help FreeNAS shine even more and be reliable.

So my question really is what Motherboard/cpu would you go with and what's a really well supported raid card with a total 24 ports? this can be 1 card or multiple. I'm really looking for a well supported system that I'm not going to have any surprises :)

Here is what I have so far. What would you do?

Supermicro - 36 Server Chassis - $1649.00
Processor - $279.99
Motherboard - $179.99
Raid Card x 2 - $1,620.00
Memory - $ 341.98
Hard Drives x 24 - $5,040.00

Thanks for the help

 

[jgreco]

Some immediate concerns:

1) Don't get the *5 version of processors for those boards; they have their own video. Get the *0 (E3-1240 in this case). We use the E3-1230's and are quite happy with them.

2) The MBD-X9SCL+-F-O is an updated version of that board. Primary difference is the ethernets.

Those two weren't fatal surprises. But these are:

3) The memory you've chosen is incompatible. You want Kingston KVR1333D3E9SK2/16G or other equivalent parts; use a memory manufacturer's memory selector, NOT the seat of your pants, to find compatible parts, because the Sandy/Ivy Bridge parts need ECC and unbuffered and a few other odd specifics. "You have been WARNED."

4) The RAID cards you've chosen are full height affairs. The chassis you've selected is a 2U chassis for purposes of system building; while it may "be" a 4U chassis, you only get 2U worth of space for the system. That means you need low profile RAID cards. Or you need to ditch the 36-drive chassis and go with a more conventional, happy-making 24-drive chassis. I'm assuming you've checked the RAID cards for FreeBSD compatibility.

So. Um. Rethink.

 

[Brand]

I'm really looking for a well supported system that I'm not going to have any surprises

If that is a concern you would be better off buying a prebuilt storage server with a warranty and support.

 

[jgreco]

That's one opinion. However, those typically come with a bunch of tradeoffs, cost or performance or ... well anyways, have you priced out what most places want for 72TB of storage? Take two iomega ix12-300r's, the chassis will run you about $3K each, then you still have to fit them with drives... Gets pricey quick.

 

[FreeME]

Thanks for all the replies guys. I did a little more research and I think I came up with something better. I do have some questions though.

1. FreeNAS seems to like memory and this board only supports 32GB. They recommend about 1GB per 1TB. I've looked at dual CPU motherboards that allow more memory but the processors are older and people tell me to go with newer CPU's. What's your take on this? I'm willing to spend $1000 on a good combo of MB/CPU. Would this affect any rebuild time or performance?

2. I'm not sure how comfortable I am with the thumb drive thing for booting. Should I Raid1 FreeNAS with 2 SSD's? I know FreeNAS/ZFS can operate on anything but I just want a solid system.


Supermicro - 36 Server Chassis - $1650
http://www.newegg.com/Product/Product.aspx?Item=N82E16811152540

Processor - $340
http://www.newegg.com/Product/Product.aspx?Item=N82E16819115081

Motherboard - $199
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182253

Memory 32GB Total - $ 268
http://www.superbiiz.com/detail.php?name=D38GE1600S

Raid Card x 3 - $775
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118112

Breakout Cable x 3 - $60
http://www.newegg.com/Product/Product.aspx?Item=N82E16816116097

SSD - OS Mirror
http://www.newegg.com/Product/Product.aspx?Item=N82E16820148442

Hard Drives x 24 - $5,040
http://www.newegg.com/Product/Product.aspx?item=N82E16822236344

NIC - $135
http://www.newegg.com/Product/Product.aspx?Item=N82E16833106015

CPU Cooler - $30
http://www.newegg.com/Product/Product.aspx?Item=N82E16835103065

Thanks again

 

[ramius]

Supermicro - 36 Server Chassis - $1650
http://www.newegg.com/Product/Product.aspx?Item=N82E16811152540

I don't know if you have noticed, but the chasis you selected has sas expanders. I'm not shure, but I think you will be needing only one SAS controller (One SFF-8087 for the front backplane and one for the rear one). To prevent a bottleneck I would go for this chasis http://www.newegg.com/Product/Product.aspx?Item=N82E16811152543 and use the double-bandwidth feature using 2 SFF-8087 controller ports for each backplane, if freenas takes advantage of this feature. You also may have to switch the breakout cables for regular SFF-8087 cables. Does the SAS controler include a Low Profile Bracket?

 

[jgreco]

I'm going to assume you've done your homework on the hardware, so I'm not going to dig through all that again.

As for 1), for "overflow storage", I'm not aware of any reason to be concerned that 32GB wouldn't be enough memory. Busier systems can make use of more memory. You're basically talking about a mostly not-busy system that will probably be relatively idle a lot of the time. The advice for memory is a loose rule-of-thumb and comes from an age where drives were an order of magnitude smaller and a fair bit slower.

1a) For rebuild times, I'd be more concerned about the attachment method of the drives than I would be about memory. If you're planning to put everything through SAS expanders, those have the potential to limit the bandwidth between the controller and the drive. Remember, during a rebuild, basically the system would like to read everything as fast as it can from all the drives in parallel. So you'll want to make sure that the SAS expanders are not a bottleneck. A single LSI controller might be capable of a theoretical 4000MB/sec (X8 PCIe) and 24 drives at 150MB/sec can almost certainly hit 3600MB/sec, so that's putting it solidly in the "could be a bottleneck" arena. In practice, it's a really good idea not to assume that a controller can actually push theoretical numbers anyways.

2) If the thumb drive thing bothers you, go ahead with SSD. The SSD's are more of a pain in a crisis; when you want to upgrade FreeNAS, there really is something pretty cool about just popping out the old and plugging in the new, and being able to roll back to the old setup instantly.

 

[joeschmuck]

If you are going to purchase RAID cards, buy an extra one. When you put this system together build it up. Once that is done and you have some data on the drives, turn it off and install the spare RAID card. Turn it on and hope it all runs fine. If there is any configuration to be made to the card ensure you take care of it then. The goal is to have a spare card ready for when/if you have a failure.

You may also want to figure out how you want to make your ZFS pools.

 

[survive]

Hi FreeME,

A couple of thoughts:

Supermirco actually makes SAS controllers, might want to see if they have something that "fits".....since it looks like you are going all Supermicro on everything else it might make sense.

Contact a Supermicro integrator, there are plenty of companies that do this and it might make sense to at least talk to a few of them and get your hardware specific questions answered. That will also get you some quotes you can refer to. The nice thing about going through an integrator is they should be able to help you out with a faster hardware RMA process.

-Will