BUILD Requesting opinions of potential build

[BilliusWillius]

Since I'm soliciting opinions, I'll start with why I want this built. I have a small home network which serves one user who has >13Tb and some other users (only one other who breaches a Tb in space requirements). Most of the data I intend to put on the server is inconsequential, and already exists on optical disk as a backup. I'd prefer to keep all the data on live disks so it can be accessed as desired and to aide ease of backup as well.

My idea:
Case+motherboard = SuperMicro: SSG-6047R-E1R36N (http://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1R36N.cfm)
CPU = Intel: E5-2603 v2 (http://ark.intel.com/products/64592...-2603-(10M-Cache-1_80-GHz-6_40-GTs-Intel-QPI))
Memory = Hynx: 4x8Gb DDR3 1600 Registered ECC (http://www.newegg.com/Product/Product.aspx?Item=N82E16820161697)
Heatsink = Supermicro: SNK-P0048AP4 (http://www.supermicro.com/support/resources/thermal/)

The HDDs are all consumer grade SATA3 (a mix of WD and Seagate, namely whatever was on sale), right now there will be 11 of them in a RAIDZ3 (all but two have been in use for about a month and so far no bad blocks or noticeable decay). The current CPU linked above is only a stopgap, as the system needs to be operational fairly soon (about a month or so before it must be running in a state which can start holding the network's data) and cost is a large issue (shaving $200 off startup cost is a big deal for the current build). Once more sufficient funds are available the CPU will be upgraded to something which supports turbo boost and hyperthreading. I'll be implementing encryption due to confidential documents which will need to be stored, and I'm aware that the CPU is underpowered for that kind of task, but as long as it can get the job done and won't have the computational equivalent of a heart attack, it can wait. Mostly the constraints on the system are price, as I had a rather difficult time finding validated ECC memory for the SuperMicro board and after reading some of the stickies about ECC I didn't want to roll the dice on my memory giving problems.
My reasoning for the case and board were for expansion later, since there might be a need for more storage later, and there will definitely be a need for better and beefier CPUs (to handle encryption mostly, possibly some transcoding for other users once their home movie collection is on active drives rather than locked away on DVD/CD/VHS).
I included the heatsink on the off chance someone's had issues with it before, I looked but I didn't see anyone report issues, and I'm a touch paranoid about overheating CPUs since I cooked one in a system some 10 years ago (I was younger and stupider then, and I don't use my computers as a footstool anymore regardless of how comfortable an ottoman they are).

 

[Ericloewe]

Sounds alright, but rather overkill-y. Wouldn't a cheaper chassis make more sense?

 

[marbus90]

The heatsink is already included. Also you should pick 2x16GB instead of 4x8GB DIMMs and preferably pick those who start at 1.35V.
Encryption-wise you don't have to worry too much with only 4xGBe. AES-NI is even on the Xeon E3 fast enough for that.
Since you want to do transcoding, I wouldn't go for Ivy EP anymore. The Haswell EP has _double_ the FLOPS compared to the Ivy.

My build for that use case would be:
http://www.supermicro.com/products/chassis/4U/847/SC847BE1C-R1K28LPB.cfm for SAS 12Gbps (not neccessary) or
http://www.supermicro.com/products/chassis/4U/847/SC847E16-R1K28LP.cfm with SAS 6Gbps Backplanes

http://www.supermicro.nl/products/motherboard/Xeon/C600/X10DRH-C.cfm for GBe Ethernet or
http://www.supermicro.nl/products/motherboard/Xeon/C600/X10DRH-CT.cfm for 10GBe

CPU-wise start out with an E5-2620 v3. I would first double up, after that choose the Xeon E5-2650L v3 or E5-2680 v3.
Why the 2620? You pay double for a CPU which already fulfills quite a lot of requirements, so that you don't need to waste 200$ during upgrading later on.
Memory of course 16GB DIMMs from the QVL.
Heatsinks you could use the SNK-P0048P or the SNK-P0048PS, absolutely no need for another fan in those cases.

I'm still thinking that UP could be enough for transcoding: http://www.supermicro.com/products/system/4U/5048/SSG-5048R-E1CR36L.cfm
You can still pop in a 2680 for 12x2.9GHz which should be good for plenty of streams.

 

[BilliusWillius]

Sounds alright, but rather overkill-y. Wouldn't a cheaper chassis make more sense?

Yes, for the current build. However I plan to expand the storage space later since I've found that once people are given access to something new they tend to eventually expand their usage. I somewhat expect the thing to be underutilized for the first few months (hopefully since that'll give me time to more fully test the whole rig) before they really start picking up. Unfortunately the next step down is about 24 drives, unless I want to go custom, which would require more time and skill than I have (my welding talent is sub-par, IMO, and I don't know the nearest machine shop who could make a design I've not created yet). Plus I somewhat like the idea of having all 36 drives in there and being able to loose 12 of them before risking data loss.

 

[BilliusWillius]

The heatsink is already included. Also you should pick 2x16GB instead of 4x8GB DIMMs and preferably pick those who start at 1.35V.
Encryption-wise you don't have to worry too much with only 4xGBe. AES-NI is even on the Xeon E3 fast enough for that.
Since you want to do transcoding, I wouldn't go for Ivy EP anymore. The Haswell EP has _double_ the FLOPS compared to the Ivy.

My build for that use case would be:
http://www.supermicro.com/products/chassis/4U/847/SC847BE1C-R1K28LPB.cfm for SAS 12Gbps (not neccessary) or
http://www.supermicro.com/products/chassis/4U/847/SC847E16-R1K28LP.cfm with SAS 6Gbps Backplanes

http://www.supermicro.nl/products/motherboard/Xeon/C600/X10DRH-C.cfm for GBe Ethernet or
http://www.supermicro.nl/products/motherboard/Xeon/C600/X10DRH-CT.cfm for 10GBe

CPU-wise start out with an E5-2620 v3. I would first double up, after that choose the Xeon E5-2650L v3 or E5-2680 v3.
Why the 2620? You pay double for a CPU which already fulfills quite a lot of requirements, so that you don't need to waste 200$ during upgrading later on.ports
Memory of course 16GB DIMMs from the QVL.
Heatsinks you could use the SNK-P0048P or the SNK-P0048PS, absolutely no need for another fan in those cases.

I'm still thinking that UP could be enough for transcoding: http://www.supermicro.com/products/system/4U/5048/SSG-5048R-E1CR36L.cfm
You can still pop in a 2680 for 12x2.9GHz which should be good for plenty of streams.

My transcoding was mostly a projection of what might be used (and it'll be VERY limited, perhaps 2-4 simultanious streams at most, all other streaming won't need transcoding since it will be to computers capable of handling the native files, my apologies for not being more precise on that).
My reason for the Super X9DRi-LN4F+ board was that I do plan on doing encryption and intel's stuff doesn't have a lot of cores per chip except on the really expensive chips, and if I understood the posts on it correctly encryption tends to use a thread per drive. Any expansion will require that I have many more threads to operate with, and I'm currently working at getting something operable (rather than functioning well).
My major reason for the 2603 was that it was so cheap. I would have gone for a 2630, but the additonal 400 is too much for the current budget (it's basically a "is it ABSOLUTELY necessary?" budget, and I've managed to argue it up to the $2800 I've got to work with). The 2620, I've seen posted that it's a horrible processor (and plus I plan to upgrade to something with hyperthreading later, so for now I kinda need a throwaway processor I can maybe repurpose to something else later when I can afford a second more potent CPU).
Ideally I'd go for the 1.35v DIMMs, but they are far outside the current budget (if I had the dosh I'd fully load the system with all 1.5Tb of memory and have it do data deduplication for me). I can upgrade later, but the 4 DIMMs I selected were the ones I could find off the QVL from a supplier I've done buisness with before (I'd be more willing to look elsewhere if I'd done buisness with someone who sold stuff of the list prior to making such a big purchase). I thought to go with 4 since 2 is only 16Gb, and 3 is an odd number (an odd number which wouldn't be in keeping with the manual on the motherboard).
Thanks for the info on the memory recomendation for encryption, I was somewhat concerned about running low doing that, my tests haven't had that problem yet as far as I've seen, but I've never run it on something quite this large.
I do like the idea, but I am not looking to get a SAS system (per drive price is far too high, I'd rather just go with consumer grade and loose ones that are broken or break down, even loosing a drive a year it'd cost me less per Tb of storage).
As for the ethernet I'd want to use a network backbone running fiberoptic, but I'm not going to bother with 10GbE, copper just isn't worth it. I'd rather just shove several cards in the box with 4 port of 1GbE cards, probably Intel, and use that to talk to the backbone, since the file server will need bandwidth more than individual port speed (Most of the computers in the house can't really reach 1GbE speeds).

 

[marbus90]

AES-NI is scaling with cores, yes. But the AES troughput of a single E5 v3 sixcore is already at >5GB/s. That is roughly 50 GBe ports, 25 if you count full duplex. But the storage subsystem will crap out earlier. A E5-2609 v2 should still be able to have an AES troughput of 3GB/s and it's just $100ish more than the 2603 v2. Hyperthreading won't help much either. Better add another 2609 v2 for upgrading, if ever needed. On top of that the 2609 v2 CPUs max out at DDR3-1333 anyway, so you might be able to get cheaper RAM. Also you can buy 2x16GB now or throw 4x8GB away soon. I'd pay a small surcharge, if any. 1.35V is just nice to have.
SATA drives will work on SAS Expander/Backplanes and you will need SAS Expanders/Controllers to meet the bay count needed for 33 drives.

You can save few $100 if you opt in for 16 instead of 24 DIMM slots:
http://www.supermicro.nl/products/system/4U/6047/SSG-6047R-E1R36L.cfm
If the primary intention is to store movies, deduping won't help unless every user stores the same videos over and over again. And even then you could put the dedup table to another SSD without the need to max out RAM. The LZ4 compression is doing quite a good job without taxing CPU and RAM much.

 

[Starpulkka]

Just have to ask that, do you have good place to mount that server. Dust free enviroment, and where air is changin and aside where that high pitch superloud noise dont bother anyone, and on summer perhaps an active cooling solution. I have mitsubishi fd-25 for active cooling, and dust is small broblem, but currently noise is my biggest broblem i have this Supermicro http://shop.delwi-itr.de/shop/Serve..._2x_Gigabit_LAN_bis_zu_16HDD_i260_78990_0.htm
Also looks like you backplane dont take full hight cards, but usually these days many comes with lower plate so its not a broblem.

What kindof UPS you planned to back up your server up? And do you have one decidated fuse for that system? You say "home" network but i guess you have somekind of workplace and im hoping you have a proper server room where you going to put that server, if so then my questions can be ignored. =)

 

[BilliusWillius]

Just have to ask that, do you have good place to mount that server. Dust free enviroment, and where air is changin and aside where that high pitch superloud noise dont bother anyone, and on summer perhaps an active cooling solution. I have mitsubishi fd-25 for active cooling, and dust is small broblem, but currently noise is my biggest broblem i have this Supermicro http://shop.delwi-itr.de/shop/Serve..._2x_Gigabit_LAN_bis_zu_16HDD_i260_78990_0.htm
Also looks like you backplane dont take full hight cards, but usually these days many comes with lower plate so its not a broblem.

What kindof UPS you planned to back up your server up? And do you have one decidated fuse for that system? You say "home" network but i guess you have somekind of workplace and im hoping you have a proper server room where you going to put that server, if so then my questions can be ignored. =)

Well I can say, with a completely ash covered face, nope! I've got a place that's sub-par for the server to sit, it'll get all right airflow but that's about the best I could say about the spot. There'll be an external home window not far from it, so I can setup a unidirectional venting system for it, but dust will be a problem for it (although I could make a HEPA filtered enclosure for it, but I'm loth to use my personal laminar flow hood for the purpose.)
I'll be hooking it into it's own UPS, but the room isn't really ideally suited to servers. Power is fairly stable, and the UPS should give it maybe 5 minutes of up-time in the event of a failure and I expect to be around for much of the time this thing is on until I can get the budget to really give it a more proper setup (and yes, you are smelling THAT much "if" on the plan.) For the moment I was planning on using this little APC 1500 I've got which is a bit too small for the full capability of the case, but I'm going to be swapping that out for something that's far more suited to a server with a far greater power draw after I've got the data on the system. Right now the system's purpose is just store data, being up for a few hours every week or so to absorb new stuff, then going back to sleep for a week. I plan to expand that as I accrue funding to do so.
As for the noise, the only person who'll be around the thing is myself; and I actually find the hum and drone of running computers to be somewhat relaxing, if not soporific. High pitched whines I tend to investigate and determine if it's concerning or not (hurray for sensory adaptation!) I do have a workplace, but it's about as cramped as a low budget would suggest, but it's not a closet, and it does get fairly decent airflow. I will be monitoring the temperature (being a bit of a geek I've got a lot of thermometers, who knows when I'll want to accurately measure the temperature of a distillation setup for euginol!) I might have to get an active cooling solution later, but by the time the temperature starts to go that way I should have a more acceptable budget. And now that I consider it more, I have a few ideas about that.
Thank you for the reminder about the thermal considerations.

 

[BilliusWillius]

AES-NI is scaling with cores, yes. But the AES troughput of a single E5 v3 sixcore is already at >5GB/s. That is roughly 50 GBe ports, 25 if you count full duplex. But the storage subsystem will crap out earlier. A E5-2609 v2 should still be able to have an AES troughput of 3GB/s and it's just $100ish more than the 2603 v2. Hyperthreading won't help much either. Better add another 2609 v2 for upgrading, if ever needed. On top of that the 2609 v2 CPUs max out at DDR3-1333 anyway, so you might be able to get cheaper RAM. Also you can buy 2x16GB now or throw 4x8GB away soon. I'd pay a small surcharge, if any. 1.35V is just nice to have.
SATA drives will work on SAS Expander/Backplanes and you will need SAS Expanders/Controllers to meet the bay count needed for 33 drives.

You can save few $100 if you opt in for 16 instead of 24 DIMM slots:
http://www.supermicro.nl/products/system/4U/6047/SSG-6047R-E1R36L.cfm
If the primary intention is to store movies, deduping won't help unless every user stores the same videos over and over again. And even then you could put the dedup table to another SSD without the need to max out RAM. The LZ4 compression is doing quite a good job without taxing CPU and RAM much.

Heh, well excuse me while I go wipe off the remnants of an oviparous animal's output from my face. I realized I was searching for RAM the wrong way round. So I think I will go with 2x16Gb reg ECC 1.35V memory. Plus the 2x16Gb sticks are cheaper by the price of a soft drink than the 4x8Gb, which means I think I'll go treat myself to some caffeine! Thanks for getting me to do some more checking.
Video files may make up the bulk of material, but there are a LOT of duplicate images on the network. I'm not so worried about de-duping the videos, since they are far smaller in number. It's the images I'm concerned with. There are some images where might have 10+ copies of the same thing scattered around the network. I'm fairly certain that even with block-level deduplication it wouldn't catch them all because there's bound to be some differences between them after all the years they've been on consumer-grade drives and file systems, and frankly I don't know how much deduping would really help since I've not managed to find all of these things and run DIFFs on them (if I had found them all I could do the thing manually). I had planned to use this thing to vacuum up the network, so there would be a centralized repo to sort through rather than all these other storage media scattered around the building. Not to mention having a backup place to dump new data as it rolls in, since the second largest data user on the network has been relying on the off-the-shelf Western DIgital network drives (and they make all the erector pili muscles attached to my follicles contract, especially because he uses it as primary storage with NO backup). I've given him a stack of 50 blu-rays as a gift with a big note to create backups, but I don't think he's used a single one. To get him to create backups I'll either have to make a script to do it invisibly, or something else that's basically a set-and-forget setup (I don't really care THAT much that his stuff is properly backed up, but I like to be accommodating and helpful if I can).
I've not used Intel in... years, back when hyperthreading was new (and I didn't have nearly enough disposable monies to afford a CPU that had it). I was planning on upgrading the CPU by basically getting an E5-2630 or better, maybe upwards of an E5-2660. That was because, as I understood it, the encryption requires about a thread per drive and I'd be running at least 11 of those in a RAIDZ3. Also I thought I'd put the extra computational muscle to use by crunching some RAIDZ parity calculations as I added more vdevs or setup a secondary zpool of mirrored vdevs in a RAIDZ3 zpool using all 36 drive bays.
Additionally I don't really know what other uses I might find for the system later so I thought I'd aim for something I could get running soon with a fair amount of room to expand into.

 

[BilliusWillius]

Since I've had this thing for over a year now, I figured I'd perform a touch of thread Necromancy and post the results.
It's been running for all but 2 months of the year, and aside from a slight SNAFU caused by a wetware error (THAT'LL teach me not to bloody RTFM, and completely, before starting something) that forced me to resliver two drives simultaneously, it's a trooper. I had to shut down for 2 months while I relocated from a room while the hardwood floors were resurfaced, and being without data for 2 months was a strange sensation. I was kinda expecting it to be a fair deal noisier, although after all the time it's been going I think I might have gotten used to the noise. I still have to get a decibel meter to measure I don't think it's noisy enough to cause permanent damage (although you can hear it through doors between floors). Even with the somewhat underpowered CPU it's crunching parity fast enough to deal with the limited writes it deals with. As for heat, it doesn't seem to have trouble or output too much, but then I've only got 18 drives in it at the moment.
As a side note, I initially got an enterprise grade UPS to power it, but didn't look carefully enough at the specs to find out that there wasn't a outlet in the whole house that could feed that monster (it weighs 120lbs, so I figured monster was appropriate). However even with the redundant 1250W PSUs a standard decent consumer grade UPS from APC seems to be able to power this thing for at least 20 minutes (not that I'm willing to test with the drives plugged in).

 

[solarisguy]

[...] I still have to get a decibel meter [...]

You may be able to get a good approximation by using someone's smartphone and a http://play.google.com/store/search?q=sound level meter