SuperMicro Rack Mount Config Opinions Please

[bollar]

Hi All,

I'm in the process of upgrading my home network and replacing my various Netgear NAS boxes. I have a server rack and have decided to unify the storage into a single FreeNAS system. (Two of the Netgear ReadyNAS Pro Business will become off-site backups)

Given the way it all prices out, I am leaning towards a 16 drive array with a RAID Z2 VDev of 8-2Gb drives and a RAID Z2 VDev of 8-3Gb drives in a single zpool. Sizes chosen because that's what I have lying around.

This is absolutely overkill and I recognize that -- part of it is just self-education, though, so I don't want to spend money foolishly if there's a better solution.

I have been looking at the following:

SUPERMICRO CSE-836E1-R800B Black 3U Rackmount Server Case

SUPERMICRO MBD-X9DRD-7LN4F-JBOD Extended ATX Server Motherboard Dual LGA 2011 DDR3 800

Since the chassis and motherboard are designed to work together, they seem like a safe choice. I did confirm that the onboard JBOD is compatible with FreeBSD. I also like the Quad GigE

As far as processor goes, my assumption has been that I don't need a whole lot of horsepower, so I was looking at one Intel Xeon E5-2603 Sandy Bridge-EP 1.8GHz 10MB L3 Cache LGA 2011 80W Quad-Core Server Processor, but if I need more power, I can add another one, or go for a faster processor.

And whatever RAM this system needs -- probably 60Gb or so.

From a compatibility perspective, I think this should work with FreeNAS, but I'd certainly appreciate any real world advice you have.

 

[bollar]

FYI, for those looking to setup something similar, here's what I wound up choosing:

SUPERMICRO CSE-836TQ-R800B Black 3U Rackmount Server Case
SUPERMICRO MBD-X9DRD-7LN4F-O Extended ATX Server Motherboard Dual LGA 2011 DDR3 800
4 x Kingston 16GB (4 x 4GB) 240-Pin DDR3 SDRAM DDR3 1333 ECC Unbuffered Server Memory w/TS Model KVR1333D3E9SK4/16G (64 GB total)

2 x Intel Xeon 4C E5 2609 2.4 GHz 4 LGA 2011 Processor BX80621E52609
1 x LSI LSI00301 (9207-8i) PCI-Express 3.0 x8 Low Profile SATA / SAS Host Controller Card
Also, a variety of old drives:
8 x Seagate Barracuda 7200 2TB drives
8 x Hitachi 3TB drives

Configured into 1 zpool with 2 RAID-Z2 VDEVs.

10TB loaded so far, peak throughput 1.6Gbps across 4 bonded GigE ports (LACP)

Seems to be working fine.

 

[jgreco]

Sounds like a hell of a system. What's the power consumption like?

 

[bollar]

It is way overpowered, but given some of the features I wanted, there was a narrow list of components that worked. Maybe Plex for FreeBSD will be released someday and I will have a use for the extra horsepower.

The UPS suggests that it's using 170W, but given the heat and noise, I won't be surprised if it's much more. Since it's been busy ingesting, I haven't had a chance to verify the draw when idle yet.

If it is 170W, that will be a relief - there are redundant 800W power supplies, so it could easily be worse.

But all I know for sure is that my neighbor's lights dimmed when I powered it on... ;)

 

[cyberjock]

My SR-2 rig dims the lights when I power it on. I love shocking friends that don't know the lights dim. :)

 

[jgreco]

See if you can get it to black out the neighborhood.

 

[bollar]

FYI, I tried to save a few bucks and use the Intel RMS25KB080 PCI-Express 2.0 x8 Low Profile Ready SATA / SAS Integrated RAID Module, but it didn't work -- it truly only works on Intel motherboards. :(

 

[bollar]

Okay, now that I have had this running for a while a couple of observations for those who may follow this path:

- LSI drivers need to be current, or the external card won't be recognized. The upgrade process requires UEFI, which isn't available on the LSI site, only the Supermicro site. The documentation on this process is basically nonexistent and I only found the instructions by accident on a seemingly unrelated FAQ.
- The system is way overpowered for my application. I have seen loads at 4+ once in awhile, but it's usually sitting around 0.5.

My original plan was to backup the FreeNAS to several Netgear ReadyNAS units, but I think my new plan is to get another X9DRD-7LN4F motherboard and a tower that will hold 8-10 drives and move a processor and half the RAM over to it. Then I can use that box not only for backups, but also for spare parts if the primary server goes down. Hopefully I can unload the ReadyNAS and Drobo units in my inventory to some people who are looking tp upgrade to appliance storage.

 

[jgreco]

That's awesome, and pretty much what I suspected/hoped you were going to find.

 

[bollar]

So, an update on this:

I have had good results selling the Netgear ReadyNAS units, so I can proceed with the plan to create a backup box using much of the same components as the original box. Here's what I've ordered:

SuperMicro X9DRD-7LN4F Motherboard -- will flash to IT mode

SuperMicro CSE-745TQ-R800B Black Tower Chassis
SuperMicro CSE-M35T-1B Black 5x 3.5" Hot-swap SATA Hard Disk Drive Trays

This system will have 13 hotswap drive slots and is capable of handling 12 SATA drives. I will move one Xeon processor and half the RAM from the original server and this will leave my only non-redundant part as the LSI-9207-8i

I think I'll configure this one as RAIDZ1 with 2 Vdevs of six disks -- once the dust settles and I know everything's working, I'll probably wipe the original server and recreate it as RAIDZ3 (and remember to force 4K blocks this time).

Will get this all configured next week sometime.

 

[c4rp3d13m]

I'm very interested to see this build Bollar.

Found this recommended config after I RAIDZ3 my 12 drives.

"For better performance, a mirror is strongly favored over any RAIDZ, particularly for large, uncacheable, random read loads.

When determining how many disks to use in a RAIDZ, the following configurations provide optimal performance. Array sizes beyond 12 disks are not recommended.

Start a RAIDZ1 at at 3, 5, or 9, disks.
Start a RAIDZ2 at 4, 6, or 10 disks.
Start a RAIDZ3 at 5, 7, or 11 disks.
The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups."

Now I'm running 2 vdev of 6 drives each.

 

[c4rp3d13m]

I am considering this case for my next build:

Rosewill RSV-L4500 Black Metal/ Steel, 1.0 mm thickness , 4U Rackmount Server Chassis, 15 Internal Bays, 8 Included Cooling Fans

with

Rosewill RSV-R26LX 26" 3-sections Ball Bearing Sliding Rail Kit for Rackmount Chassis

 

[bollar]

I am considering this case for my next build:

Rosewill RSV-L4500 Black Metal/ Steel, 1.0 mm thickness , 4U Rackmount Server Chassis, 15 Internal Bays, 8 Included Cooling Fans

with

Rosewill RSV-R26LX 26" 3-sections Ball Bearing Sliding Rail Kit for Rackmount Chassis

Nice price!

If the cooling is okay, I'd say it's a great choice.

 

[pirateghost]

thats almost the same exact case i have.

if it cools as good as the norco, then you shouldnt have any issues at all.

http://www.newegg.com/Product/Product.aspx?Item=N82E16811219020&Tpk=RPC-470

 

[bollar]

For those of you following along, here's how I decided to configure the arrays after testing with our primary workloads of AFP and iSCSI traffic:

Production:

Sixteen 3TB drives mirrored in eight groups of two.
HBA0:
8 - Seagate 3TB
HBA1:
8 - Hitachi 3TB
In all cases, a Seagate is mirrored to a Hitachi on different HBAs.

Offsite Backup System:
Twelve 2TB drives in three RAIDZ1 vdevs of four.

All absolutely irreplaceable data is also periodically backed up manually to hard drives stored offsite.