SuperChassis 847E16: MB/CPU/HDs recommendations

[Tenek]

I will be building data server first. Assuming I'm getting SuperChassis 847E16 and use all 36 slots. I'm thinking about stripped RAIDZ2 pools, 6 drives in each pool, at least 4Tb drives. It will get at least 64Gb of RAM.
Server will hold media and backups mostly. It should handle simultaneous read/write requests without significant performance degradation. I look forward to utilize my 1Gb Ethernet network as much as possible.
Also, server should handle some standard plugins for video/audio streaming.

Assuming all this, what MB/CPU/Drives would you recommend?
I'm not sure what drives (NL-SAS or SATA) will work for 847E16's backplane (never worked with it before)
As for hard drives I'm thinking so far about: Seagate Constellation ES 4TB SAS (ST4000NM0063) or Western Digital Red 6 TB SATA (WD60EFRX)

I would get something future proof since I don't want rebuild NAS solution too often :).

 

[jgreco]

The SAS expander backplanes will work with SATA drives. Generally speaking SAS slots will take SATA drives. There've been credible reports that SATA can cause some disruption to neighboring drives during error recovery, and it has something to do with the SAS expander, but I don't have much more information on that.

For a NAS, you don't need a dual board, so something like the X9SRL-F seems like an ideal starting point. I do not suggest the latest 2011-R3 (X10S) boards because you probably don't want to be spending your time debugging driver issues - nothing wrong with the boards though!

 

[Ericloewe]

The SAS expander backplanes will work with SATA drives. Generally speaking SAS slots will take SATA drives. There've been credible reports that SATA can cause some disruption to neighboring drives during error recovery, and it has something to do with the SAS expander, but I don't have much more information on that.

For a NAS, you don't need a dual board, so something like the X9SRL-F seems like an ideal starting point. I do not suggest the latest 2011-R3 (X10S) boards because you probably don't want to be spending your time debugging driver issues - nothing wrong with the boards though!

Are we talking SATA drives failing to time out on a read error, SATA drives acting weird during rebuilds, or something else?

 

[jgreco]

"but I don't have much more information on that."

 

[Ericloewe]

"but I don't have much more information on that."

That's what I feared...

 

[Tenek]

Thanks for recommendations, jgreco.
I've seen some posts about toxic SAS Expander <-> SATA drives relation.
It sounds like a minor risk. If something like that happens can it lead to ZFS pool lose?

Also, I noticed that some HDs available in SAS/SATA configurations for about the same price. For example: ST4000NM0023
So, if I will get mentioned above drive in SAS configuration should it resolve potential conflicts?
Or there will be the same SATA tunneling but inside harddrive (instead of SAS expander)?

 

[jgreco]

Since it isn't clear what happens, I can't say that it couldn't or wouldn't result in pool loss. If it damages the pool substantially enough to compromise redundancy, that'd be potentially fatal to the pool. If it is merely causing I/O to deadlock while one of the SATA drives "figures things out," then it's merely going to be messing with your pool throughput. The cases I've heard of sound more like the latter.

If you plan to go with expanders and can acquire SAS drives at a similar price, consider it. In the case of the NM0023, it is a dual-ported SAS drive but has 512 byte native sectors ... tradeoffs, tradeoffs. So you could even go for redundancy by getting the E26 chassis variant if that mattered to you.

 

[jgreco]

Oh, and, those drives are 7200 but eat a little more power. For anyone ELSE reading this, I do not suggest SAS drives on a chassis that has some SATA bays, because this creates some issues when you need to shuffle things around.

 

[Mlovelace]

I run the 3tb western digital SE drives and they've been good so far. Here is a link to the 4tb version.

WD SE WD4000F9YZ 4TB 7200 RPM 64MB Cache SATA 6.0Gb/s 3.5" Enterprise Hard Drive http://www.newegg.com/Product/Product.aspx?Item=22-236-520

 

[Urs]

1. look how big is your working dataset
2. does it easily fit into ram (first max out ram if you want it really fast)
3. how big are those files written?
4. are they sync or async writes?
5. consider using a read-cache ssd and two or three mirrored write cache ssds
6. how many iops per sec do you need?

i´m running 6 vms of a really small freenas host and there are really no problems, but it depends on what your vms are for...

 

[Urs]

i just see you want to stream media and hold backups... how many people are looking videos at once? do you only need to stream, to transcode or just a cifs share?

 

[diehard]

I can attest to some problems with this chassis/backplane and issues with the SAS expander with SATA drives.

Strangely enough the problems have seemed to be with older 1TB Seagate Constellation drives moreso than 3TB WD Red's.

 

[Ericloewe]

I can attest to some problems with this chassis/backplane and issues with the SAS expander with SATA drives.

Strangely enough the problems have seemed to be with older 1TB Seagate Constellation drives moreso than 3TB WD Red's.

That's interesting, since others have complained about older firmware versions on Seagate drives. Maybe there's a link...

 

[Tenek]

Urs, typical workload would be: to have a couple users copying/writing (>1GB) files over CIFS. And couple users watching video (transcoded and/or over CIFS).
Per jgreco suggestion I will be getting X9SRL-F and 4Tb SAS drives (maybe ST4000NM0023). All bays will be SAS drives for consistency.
I didn't check on CPU yet. I will be back when will take a look.

That's interesting, since others have complained about older firmware versions on Seagate drives.

I would expect that firmware issues should be sooner or later resolved. Maybe we are good now days?

 

[Ericloewe]

Urs, typical workload would be: to have a couple users copying/writing (>1GB) files over CIFS. And couple users watching video (transcoded and/or over CIFS).
Per jgreco suggestion I will be getting X9SRL-F and 4Tb SAS drives (maybe ST4000NM0023). All bays will be SAS drives for consistency.
I didn't check on CPU yet. I will be back when will take a look.


I would expect that firmware issues should be sooner or later resolved. Maybe we are good now days?

Firmware updates for HDDs are spotty, poorly publicized, messy and disorganized. It's not something that'll be solved before the majority of the drives affected is retired (dunno what drives specifically, so can't say anything more specific).

 

[Tenek]

For a NAS, you don't need a dual board, so something like the X9SRL-F seems like an ideal starting point. I do not suggest the latest 2011-R3 (X10S) boards because you probably don't want to be spending your time debugging driver issues - nothing wrong with the boards though!

Ok, that was a while since my original question. But, I'm getting closer to my build :).
I specifically looking at CSE-847E16-R1K28WB chassis.
Now days would you still recommend X9 over X10? And if X10s are fine what would be a good chouse for this chassis? I was thinking about MB with integrated HBA controller if possible.
Thanks!

 

[jgreco]

X9 vs X10 is a relatively slight update, but X9 is at this point ~2011-2012 era, and X10 has been mostly completely stable for a long time. If you get a great deal on an X9, by all means, go for it, you won't be sorry, but I wouldn't invest in it if an X10 was available instead.

Now you need to read this very closely. The "W" in your chassis denotes WIO. These chassis do not take standard ATX boards. They take Supermicro WIO format boards. These boards are specially designed to make better use of space in the chassis in the event you need full height, full length cards. The boards you would use for this chassis are almost certainly the X9SRW or X10SRW. That's a Xeon E5 (not E3) single socket board.



I'm showing you the 1018R-WC0R chassis with an X10SRW inside it. This 1U chassis allows two full size PCIe cards and a half length half height PCIe card too. It does this by not being ATX form factor and instead using riser cards.

Now the chassis you're looking at would support four full size PCIe on the left and one half height on the right, at least with the X10SRW. That's because the right riser only supports one card. But they're all x8 IIRC. We've got some gear in that form factor. It's sweet. But I have to tell you, if you think you might need more slots, then you may be better off with a non-WIO chassis.

For a pure NAS, WIO is generally a fine setup, and the X10SRW is nearly ideal IMO. Go get an E5-1620 V3 or 1650 V3 CPU, absolute best bang-for-the-buck in the E5 CPU's. You get the possibility to hit 128GB of RAM at a reasonable cost (128GB ~= $1000) at full DDR4-2133 speed. The E5-16XX's only take RDIMM, not LRDIMM, by the way.

The backplanes on your chassis are two separate ones; one for the rear-facing and one for the front. There are 24 drives on that front one so it is slightly bandwidth-challenged, but the reality is that this isn't ACTUALLY an issue because ZFS just can't pull that much data from all its disks simultaneously even during a scrub or resilver.

Our VM filer is built out of an X10SRW with E5-1650V3, 128GB, a M1015 HBA, a Chelsio card, an Intel 750 for SLOG, and then two Addonics M.2 adapter cards to hold some additional SSD storage for L2ARC. It's in a 2U 24 2.5" drive chassis. Powerhouse of a box.

We also use the X10SRW's for hypervisors. I've got a pair of them on the bunk above me right now as I take them out east for deployment. So I like them

 

[Tenek]

jgreco, thanks for the detailed answer! This is very helpful.

At first estimation I have to following:
- 847E16-R1K28WB (1600). I didn't find this chassis available on amazon or ebay.
- X10SRW-F (300)
- RAM (1376)
- M1015 (100)
- CPU (700)
- 2 SSD for OS (~ 100?). What brand would be recommended?
- I'm not sure if I need SLOG or L2ARC I guess it can be added later if necessary?
Total is ~$4200
I'm sure some parts can be found cheaper. Any recommendation will be appreciated.

OR

I never bought refurbished/used hardware on ebay. Some pullouts are are pretty cheap and tempting over there. For example this:
Supermicro 4U 36 Bay Storage Server X9DRI-LN4F+ 2x E5-2670 256GB 6047R-E1R36N
All together it is 2,899.95 (+ shipping).
All I will need to do is to replace raid card it comes with (AOC-SAS2LP-H8iR Raid card) with M1015.
Please let me know if you think this server would be a good option to consider.

Thanks again.

 

[tvsjr]

I have nearly that exact server, although I paid less for it (got lucky and snagged the chassis + mobo for $600, bought two E5-2670s for $200/ea., had the RAM laying around). Be careful on the backplanes... in this auction, you're good, but the part numbers matter. SAS2 indicates, well, SAS2 support... SAS1 backplanes will only recognize ~2.2TB drives. EL1 indicates an expander backplane, which you'll want... there are some backplanes that aren't expander-based, so you'd need to plug into 24/36 separate ports. That motherboard works very well for me... the quad NIC is very nice. I've got 2 running LACP to my client systems, and 2 for the storage networks.

I think you can do better on price if you watch for a while, but that depends on your timeline and risk tolerance.

OS drives aren't hit very hard. I picked up two Intel 320 40GB drives off eBay for about $25/ea. and they work very well.

If this is just a media server, you most likely don't need SLOG/L2ARC. VM workloads tend to be where SLOGs matter. As far as cache, max the memory on the board first (these boards support ludicrous amounts of memory).

 

[Tenek]

Alright!

I just ordered CSE-847E16-R1K28LPB with 256GB RAM.
Now I'm going to order AOC-USAS2-L8E on Amazon.
And I need to decide on hard drives. I would like to get 8tb SAS hard driver if any recommend available (please let me know if any reasonable 8tb SAS drives are available).
So far I looking at ST6000NM0014 and ST6000NM0034.
ST6000NM0014 are more expensive, but they are native to 4K block sizes. Please let me know if it makes sense to spend more but get ST6000NM0014 drives.
Again it will be 5 (or 6) Raidz-2 pools (6 drives each). So outcome will be about 144 Tb size.

Thanks!