Feedback on a proposed high end build

[danjb]

I've built a number of servers using desktop components. I've been running one with 12 drives (4 in an external eSata enclosure) for Freenas for 2 years using recycled desktop components. I've become very dependent on it for shared storage, and it also runs a Crashplan and Transmission jail.

i'm thinking of building a high end (for home use) server using real server components like those recommended on this forum. I'd like something that can run reliably for 5+ years and allows for some room for growth over that time if needed. Here's the component list I came up with:

NORCO RPC-4224 4U Rackmount Server Case with 24 Hot-Swappable SATA/SAS Drive Bays
SUPERMICRO MBD-X10SRL-F Server Motherboard LGA 2011 R3
LSI LSI00244 (9201-16i) PCI-Express 2.0 x8 SATA / SAS Host Bus Adapter Card, Single Pack--Avago Technologies
iStarUSA WO22AB 22U 4 Post Open Frame Rack
SuperMicro PWS-920P-1R 920W high-efficiency (94%+) power supply with PMBus - OEM
Intel Xeon E5-2620 v2 Ivy Bridge-EP 2.1 GHz LGA 2011 80W BX80635E52620V2 Server Processor
Kingston 64GB (4 x 16GB) 288-Pin DDR4 SDRAM ECC Registered DDR4 2133 (PC4 17000) Server Memory Model KVR21R15D4K4/64
(12) Seagate NAS HDD ST6000VN0021 6TB 128MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive Bare Drive
SAMSUNG 850 EVO 2.5" 120GB SATA III 3-D Vertical Internal Solid State Drive (SSD) MZ-75E120B/AM

Any suggestions on changes?

 

[Nick2253]

Some comments: (take what you will)

• Norco products have a mixed reputation around here. The two big complaints are the fit/finish (primarily of the drive bays, but also of the entire case), and the reliability of the backplanes. Some would suggest going with a SuperMicro case, especially because they come with PSUs, and you can get a lightly used one on eBay for just about the cost of a Norco Case.
• Kingston memory also has a mixed reputation. There was some hullabaloo over something a while back, and since then I've generally stayed away from the brand.

Some questions:

• How do you anticipate configuring your HDDs?
• What's the 850 EVO for?
• How much of your disk space are you using right now, and what's your anticipated annual growth rate?

Otherwise, looks good to me.

 

[danb35]

I don't think that CPU will work with the motherboard--the board is Socket 2011R3, while I think the CPU is just Socket 2011. That's OK, though, because I think there are better choices in CPU for a single-socket board. The E5-1620v3 has two fewer cores, but 1.4 GHz more clock speed, for about $100 less than the 2620. Or the 1650 has the same six cores as the 2620, and the same 3.5 GHz clock as the 1620, but is about $600. Or pick up an older dual Socket 2011 board and a dirt cheap pair of E5-2670s like I did.

 

[danjb]

Some questions:

• How do you anticipate configuring your HDDs?
• What's the 850 EVO for?
• How much of your disk space are you using right now, and what's your anticipated annual growth rate?

Thanks very much for your feedback!! I was thinking a single large RAIDZ2 volume from the 12 new drives, with a primary storage dataset, a jails dataset, and maybe a system dataset.

The 850 EVO is a boot drive for FreeNAS. It's far bigger and faster than needed, but it will hopefully make for more flexibility in the future if requirements change, for not much additional cost.

On the old system I'm up to 90% of 13TB, and it's been increasing at the rate of maybe 250GB / month on average. If anything, I imagine that will increase in the future as media demands increase. As I say, I'd like to never have to worry about storage capacity in the next 5+ years. 12 6TB drives is probably overkill, but two years ago I thought the current setup was overkill.

I'll also look at the cpu's you mention danb35... that list was put together in January and I have not yet gone back to validate if it is still the right sweet spot. I assume more cores is good if more jails ends up on the machine. Today I try to avoid putting much cpu intensive stuff on the storage server (I have a separate server for Plex, for example). But, again, I would like flexibility if that should change.

 

[Dice]

SuperMicro PWS-920P-1R 920W high-efficiency (94%+) power supply with PMBus - OEM

How are you connecting this PSU?
Judging from the pictures it seem to be only the 'slot in unit' missing the "backplane" including cables to the motherboard etc?

 

[danb35]

I'll also look at the cpu's you mention danb35... that list was put together in January and I have not yet gone back to validate if it is still the right sweet spot.

I'm not an expert on the E5 chips, but it seems there's a pretty hefty price premium for the 26xx parts, in return for which you get the ability to use two sockets. If you aren't, and don't plan to be, using more than one socket, it seems the 16xx parts would warrant a hard look. But as I said, I'm not an expert with them.

 

[Robert Trevellyan]

I was thinking a single large RAIDZ2 volume from the 12 new drives

That's pretty wide. At least consider the option of striping two 6-disk RAIDZ2 vdevs.

 

[danjb]

That's pretty wide. At least consider the option of striping two 6-disk RAIDZ2 vdevs.

Is the advantage of two 6-disk vdevs versus one 12-disk vdev greater redundancy in case of hdd failure? Isn't it true, though, that is any one vdev fails the whole volume fails, so the failure of 2 hdd's in a single vdev would still take down the whole volume, right?

 

[Robert Trevellyan]

Is the advantage of two 6-disk vdevs versus one 12-disk vdev greater redundancy in case of hdd failure?

To some degree, but it's also a question of performance. The best thing about 12-wide is the capacity efficiency, but it's right at the limit of most recommendations that I've read. That doesn't mean you must not do it.

Isn't it true, though, that is any one vdev fails the whole volume fails, so the failure of 2 hdd's in a single vdev would still take down the whole volume, right?

Yes, that's true. EDIT: see correction below from @Sakuru

 

[Sakuru]

Isn't it true, though, that is any one vdev fails the whole volume fails, so the failure of 2 hdd's in a single vdev would still take down the whole volume, right?

No, you have to lose 3 disks to break a RAIDZ2 VDEV.

 

[danjb]

No, you have to lose 3 disks to break a RAIDZ2 VDEV.

Yes, that's right, sorry, I should have said 2 hdd's is the max that can fail. I also have seen the recommendations that say 12 hdd's per vdev is the top end of what is recommended, so I'll take your advice to do two 6 hdd vdevs rather than 1 12 hdd vdev. There is some additional redundancy / protection in that as well. Thanks!

 

[Mirfster]

You also gain IOPS with using 2 vDevs

 

[Nick2253]

It's a little misleading to say that two 6-drive RAIDZ2 VDEVs have the same failure rate as a single 12-drive RAIDZ2 VDEV on the basis that it takes "only" three drivers to bring down either one.

First off, image that three drives have failed. In a 12-drive RAIDZ2 VDEV, you have a 100% change of complete array failure. In the 2x 6-drive setup, your chance of data lose is only 25%: assuming failing drives are randomly distributed, only in the cases where all three drives fail in the same VDEV do you lose data.

Also, you array rebuild time is going to be much shorter on a 6-drive VDEV than a 12-drive VDEV. In the case that two drives fail, you're working against the clock, and the sooner you rebuild, the better.

Obviously, this all comes at a cost of HDD space (you lose an additional 2 drives), it's up to you if it's worth it for redundancy.

At the very least, I'd recommend RAIDZ3 over RAIDZ2 if you go the single VDEV route.

 

[Robert Trevellyan]

No argument with your analysis, except ...

It's a little misleading to say that two 6-drive RAIDZ2 VDEVs have the same failure rate as a single 12-drive RAIDZ2 VDEV on the basis that it takes "only" three drivers to bring down either one.

Did anyone say that on this thread?

 

[maglin]

If you are mostly just putting media on your NAS I would imagine a 12xRAIDz3 would work out great.


Sent from my iPhone using Tapatalk

 

[Nick2253]

No argument with your analysis, except ...

Did anyone say that on this thread?

I was referring to this:

Isn't it true, though, that is any one vdev fails the whole volume fails, so the failure of 2 hdd's in a single vdev would still take down the whole volume, right?

Which carried the implication that they were equivalent failure rates. I may have just been reading too much in to it.

 

[Robert Trevellyan]

Correcting the 2 to a 3, it's a completely accurate statement, and follows logically from the previous thought in the same post, that two 6-disk vdevs have more redundancy than one 12-disk vdev.