BUILD Starting a 50-60TB Z2 Raid. Need help selecting hardware.

[manfromafar]

Hello all;

I assume (and am highly confident) that most people here know far more about hardware then I.

What I really need help with is selecting the hardware for the current build. Over the past 2 years I've accumulated over 40TB of "stuff". Now that I have money and the space I am looking to move the storage from single HDDs to one massive box that has room to grow.

The major things I need are:

1 cpu
Motherboard that supports ecc ram
I am looking at 16GB of Crucial ecc unbuffered ram
A power supply to run at the least 12 6TB WD Reds ( This will need to be able to expand to other pools later on as I expect my data to continue to grow at a steady pace)

As those are the major hurtles right now thats what I'll focus on.
I plan to be doing mostly light transcoding of video files and most machines I have in the house will be able to play back the video files perfectly fine without the use of transcoding... That being sad apple products still arnt the best with this so I will need to take that into account.

To house everything I'm planning on just building my own case as that will be the easiest to do.
To connect all the HDDs together I am looking at the Highpoint 750 HBA card as it allows up to 40 Disks to be combined into one PCIe slot. This will help with future expansion.

I hope to have the build down to around 600-700$ Canadian if possible. This excludes the cost of the HBA card and the drives of course.

Doing some of the research I have cobbled together a starter build:
I've been looking at the SeaSonic SSR-360Gp Plus Gold Certified Power supply
Crucial 16GB (2X8GB) ecc unbuffered Ram
A Pentium g3240
SuperMicro X10SLM-F-O (SUPERMICRO MBD-X10SLM-F-O uATX Server Motherboard LGA 1150 Intel C224 DDR3 1600 ) Mother board

Thanks for the help or if there is non thanks anyways.

 

[danb35]

Your chosen hardware is highly unlikely to be adequate for your stated storage needs. You might be OK if you bumped the RAM up to 32 GB, but then you're maxing out your motherboard. The rule of 1 GB RAM / 1 TB storage is only a guideline, not set in stone, but with 40 TB of storage and plans to expand as high as 60 TB, you'll probably be pushing it with only 32 GB of RAM. I'd suggest you look hard at a Socket 2011 board and low-end Xeon E5 instead. This will give you plenty of room to expand your RAM if necessary.

Reconsider the Highpoint card--what reports there are on it with FreeBSD aren't good. The IBM M1015/LSI 9211-8i is the go-to HBA around here, or some board include an onboard LSI HBA. With a SAS expander backplane, one of those will handle all of your drives.

You don't say how many drives you want to handle. This Supermicro chassis will handle up to 24 in hot-swap bays, and has a SAS expander backplane. At $375+shipping, it's about 2/3 of your budget, but it should last you a good long time.

 

[manfromafar]

Hmm interesting.

So even without doing any encryption I would have to bump up the cpu?
I'm not looking for amazing speeds or anything probably around 60MB/s write/read.

It's only job will basically be to store the data I push and server out the videos to a max of 4 devices at once. This shouldn't be a big deal considering most of the devices don't need transcoding.

 

[BigDave]

You can certainly start with the pentium, but I would just choose a board that's upgradeable to the xeon that danb35 is suggesting,
so that once you find the pentium lacking, just pop in a xeon. ;)

 

[Ericloewe]

Hmm interesting.

So even without doing any encryption I would have to bump up the cpu?
I'm not looking for amazing speeds or anything probably around 60MB/s write/read.

It's only job will basically be to store the data I push and server out the videos to a max of 4 devices at once. This shouldn't be a big deal considering most of the devices don't need transcoding.

The problem isn't computational. It's storage. ZFS needs a crapton of RAM and Xeon E3s are limited to 32GB of RAM. Performance will suffer, possibly a lot, with just 32GB of RAM if you end up with 60TB of data.

The only HBAs that we know work reliably are LSI SAS 2008/2308 cards, such as the M1015 or LSI SAS 9207.

A Seasonic G-360 is rather underpowered for a system this size, even in the best scenario (10 6TB drives). For 12 drives as a starting point, I'd look at a G-550, at least. Probably a Seasonic X-750 or Platinum 750 beyond that, if you want room to grow.

 

[Ericloewe]

You can certainly start with the pentium, but I would just choose a board that's upgradeable to the xeon that danb35 is suggesting,
so that once you find the pentium lacking, just pop in a xeon. ;)

Hate to be the bearer of bad news, but there are no LGA 2011 Pentiums. The problem here is RAM, mostly.

 

[BigDave]

I kinda missed that socket 2011 thing, sorry!

 

[marbus90]

To all the posters in this thread I may recommend this lecture: http://www.freenas.org/whats-new/20...design-part-i-purpose-and-best-practices.html

Hence the Socket 1150 route w/ 32GB RAM may be okay together with a 24bay chassis - however Socket 2011-3 systems aren't far off price-wise whilst providing an upgrade-path til 128GB RAM. The PSU needs to deliver a peak draw of ~60A on 12V, so I'd stick to new singlerail ~750+W PSUs.

 

[SweetAndLow]

You are going to need more disks also. If you did 11 disks at 6tb each in raid z3 like is suggested you will get 42tb. Now subtract out 20% because zfs doesn't like to be filled past 80% and you get 34tb or 31tib usable. Which is way less than you curvedly have. There are a couple other combinations possible and I'll let you figure out the math for those other combinations like 8 disk z2 and 6 disk z2. If I was building something like this I would go with a couple 6 disk z2 vdevs.

 

[Ericloewe]

To all the posters in this thread I may recommend this lecture: http://www.freenas.org/whats-new/20...design-part-i-purpose-and-best-practices.html

Hence the Socket 1150 route w/ 32GB RAM may be okay together with a 24bay chassis - however Socket 2011-3 systems aren't far off price-wise whilst providing an upgrade-path til 128GB RAM. The PSU needs to deliver a peak draw of ~60A on 12V, so I'd stick to new singlerail ~750+W PSUs.

My problem with the 32GB is that it's a limiting factor right now, which makes for an unpleasant future with even more data.
It's ok for certain values of ok , but it can be a real nuisance in other cases.

 

[manfromafar]

OK, So what CPU/Motherboard would be recommended for build then?

Also would the previous parts be fine for a 25TB build?

I will upgrade the PSU no matter what though since that I was unsure about.

 

[manfromafar]

Hmmm maybe I'll bump the Mother board up to a X10SL7 Since its only 140$ more and that way I can skip the HBA card and just use the built in one and save 300$

I believe it has a LSI 2308 raid card build in that can be reflashed.

http://www.newegg.ca/Product/Product.aspx?Item=N82E16813182821

If this motherboard works then I can sink the extra $200 into another kit of 16gb of ram

Also checking out the G3258 as I can get it cheaper then the g3240

 

[danb35]

The X10SL7 is a good board for many FreeNAS applications, but it maxes out at 32 GB of RAM. That may be enough for you for your current needs (or it may not), but it's unlikely to be enough for the future. That's why I recommended a Socket 2011 board and an E5 processor--those can handle much more RAM (up to 512 GB according to spec, though that would cost a fortune). Unfortunately, that's going to run up the cost a bit. The only Supermicro 2011 board with a built-in HBA is the X10SRH, which goes for about $400. You can get boards without the built-in HBA for about $270, but once you buy the HBA it's close to a wash. The least expensive E5 CPU I see is the E5-2603v2 for $206. Two 16 GB registered DIMMS would be $125-150 ea.

Using 16GB DIMMs, that board will allow you to expand to 128 GB of RAM. If you really wanted to go crazy, you could use 32GB DIMMs instead. Even with 8GB modules, you've got twice as many slots as on the Socket 1150 boards.

 

[manfromafar]

Interesting I can go upto a x10sri-f for the same amount of money.
I can also get the 16gb sticks for 250$ so 50$ increase isnt to back I just need to get a cpu for this thing.
Also with this Motherboard I dont have to worry about flashing the raid controller on it since there isn't one.

 

[marbus90]

best FreeNAS CPU would be the E5-1620 v3.

 

[joelmusicman]

For what you're talking about, if you're trying to stay low-budget, I'd recommend looking at used parts off of Ebay. Specifically, Xeon LGA1366 gear. You could get a processor (X5650) and motherboard for around $250. High ram capacity, and not stuck with the hideously expensive DDR4 RAM.

 

[joelmusicman]

Also, with the size of your array, you might want to consider upgrading to RAIDZ3.

 

[marbus90]

For what you're talking about, if you're trying to stay low-budget, I'd recommend looking at used parts off of Ebay. Specifically, Xeon LGA1366 gear. You could get a processor (X5650) and motherboard for around $250. High ram capacity, and not stuck with the hideously expensive DDR4 RAM.

"hideously expensive DDR4" is a myth long abolished.

Also, with the size of your array, you might want to consider upgrading to RAIDZ3.

should be clear that this isn't going to be a single vdev.

on another note, as I researched for another user I've found this system:
http://www.ebay.com/itm/151608402237

if you're scared of the expander (no reason to) you could request the A variant of the chassis with 3x9211-8i controllers. The CPUs are fine as long as you don't use encryption, the board takes up to 192GB RAM. 16GB RAM is a start, you can request more to be added - be sure to request 16GB Registered DIMMs, thereof at least one per CPU (2 total), better 3 per CPU (6 total). Anything below 150$ per 16GB added seems a good deal.

 

[joelmusicman]

"hideously expensive DDR4" is a myth long abolished.

should be clear that this isn't going to be a single vdev.

1. Ok, so the prices have come down some since I last looked, but there's still a premium over DDR3.

2. It wasn't clear to me... He was talking about having 12 6TB drives in a Z2 array. 10 x 6 = 60TB.