Final Freenas Critique - In Build Mode Now!

[jeffreyj900]

So, I posted one of these a couple of months ago. The original hardware I selected would make a great build, BUT based on all my research I've changed it! I'd like one final pass before I purchase....

I bought a case this week on ebay from one of the resellers that cyberjock mentioned. I went with a 16 bay supermicro chassis, SC836E26-R1200. I've been through the hardware recommendations and a ton of other threads.

* Motherboard - Supermicro X9SRH-7F-O (Capable of > 32GB memory; just in case)
* CPU - Intel E5-1650 v2 (6 core)
* CPU Cooler - SNK-P0048AP4 (from Supermicro site based on Motherboard)
* Memory - 2 x Samsung M393B2G70QH0-CMA (Samsung DDR3-1866 16GB, tested memory from Supermicro Site)
* Hard Drives - 8 x WD Red 4TB (Raid Z3)
* SATA DOM - Supermicro SSD-DMo32-PHI (yes, I know 32gb is big)
* CyberPower CP1500PFCLCD - PFC Sinewave UPS System - Pure Sine Wave | 100% Active PFC
* Silverstone Fan Splitter Cable
* Norco SFF-8087 Reverse Breakout Cable

I am planning Raid Z3 over Z2. I have a Synology DS 411+ with 4 2TB drives in Raid 5 (yeah, I know). My plan is to smoke this one to ensure all the hardware is good. Move everything from DS411+ to the new server. Use the Synology for backups. If I have the backups on the Synology, could I safely go with Raid Z2?

I plan on using the LSI 2308 controller on the motherboard to get started. If I add another 8 drives in the future, I'll add the IBM 1015.

I originally was going to get a Supermicro 4U Tower instead of rack mount. Given that I went this way, any recommendations on a rack mount UPS? The one I selected has been mentioned here as working with FreeNAS, and seems like a great pick.

I'll need to figure out how to get FreeNAS installed on the SATA DOM. I've already found a couple of threads on that. Will likely need to upgrade bios, etc.

Is there anything I've left out or doesn't make sense? This is a great site!

 

[survive]

Hi jeffreyj900,

The only thing "wrong" I see is running a pair of DIMMs on a quad-channel memory board, so you would only be using half your potential memory bandwidth.

Other than that the thing should be a beast.

-Will

 

[jeffreyj900]

Got it Will! So you would recommend 4x8GB. Would still allow me to go up to 64GB. I couldn't imagine needing more than that.

Thanks for that - Jeff

 

[SweetAndLow]

I would do raidz2 with 8x4tb but I think that is the upper edge

 

[marbus90]

Nope, better go for 16GB Registered DIMMs. they're usually cheaper per GB. And the DIMMs still won't idle but slap you with a bandwith in the dozens of GB - per DIMM. No need to be scared there.

Also your chassis seems to have a SAS Expander integrated. You need a SATA-SFF-8087 reverse breakout cable - but only one, and with that all the HDD bays are usable. No need for another controller to use the other 8 drives. Even if you don't have the expander, you can use the LSI SAS plus 8 onboard SATA Ports

Also the SATA DOM is available in 16GB too, on top of that I would take two since 9.3 is knocking on the door and allows for redundant boot drives. No special requisistes for the DOMs - they're SATA drives and seen as such by FreeNAS.

 

[jeffreyj900]

Thanks marbus90 - I have SFF-8087 cables on my list for the order. Thanks for calling it out. The case does have an expander with 6 SFF-8087 ports. The reverse breakout cables I've seen all have 4 sata connectors. I was assuming I'd need one per 4 drives or 2 given my current need. I have this SAS expander: BPN-SAS2-836EL2. My assumption was that I'd flash the LSI 2308 into IT mode, plug the sata connectors into the 8 ports and then the SFF-8087 into the correct SFF-8087 port on the expander. Of course, I need to figure out, by digging into the manual, which of the 6 SFF-8087 ports that would be. Is that correct?

I could also get lucky and the case could come with the existing connectors already plugged into the right ports.

 

[marbus90]

Yes, but you can only use 1 or at most 2 8087 ports. From what I understand, even the old expanders support 2x 8087 uplinks to the board. But you need only to connect PRI_J0 from the (rear view) right-hand side expander to run all 16 bays on SATA. The left expander is useless without SAS disks.

 

[jeffreyj900]

So my LSI 2308 would be my Host Bus Adapter (HBA). I would have two SFF-8087 reverse breakout cables, each connecting 4 of the 8 sata ports, the SFF-8087 would plug into each of the J0's on the expander? Or, do I just need one SFF-8087 plugged in and somehow the expander sees the other 4 ports not attached to the breakout cable because it sees the LSI 2308 as the HBA (my logical mind makes me think all 8 need to be connected to work).





If I do fill both Jo's with SFF-8087, how do I expand to another 8 drives in the future? Sorry for all the questions, just when you think you have it figured out.....

 

[marbus90]

Only the primary expander on the right of that picture is neccessary with SATA disks. Connect J0 of the primary expander to the HBA and all 16 disks can be used with that single connector cable. The expander multiplexes all 16 disks onto that single SAS cable. No, it won't be a slowdown, you can still use ~130MB/s per bay. That's in the ballpark of HDD speed.

 

[jeffreyj900]

Appreciate your patience, I get it now.

 

[jeffreyj900]

Parts ordered and beginning to come in. Received everything except the motherboard, cpu and memory. The case is used but in fantastic shape. Very solid build and HEAVY (71 lbs). UPS is pretty big. Can't wait to get it fired up! Gotta figure out how to get FreeNAS on the SATA DOM and whether to go with 9.2.1.9 or 9.3.

 

[jeffreyj900]

Got everything put together. Just fired it up. Not as loud as I thought it was going to be. Kinda loud when it fires up, but then slows the fans way down. Just got into IPMI, very nice by the way. Going to flash the controller, get the drives installed, then load FreeNAS tomorrow.

Any opinions on the following:
1. Should I go with FreeNAS 9.3 or stick with 9.2.1.9. I'm leaning towards 9.3. After burn in, tinkering and fine tuning, I suspect any issues with 9.3 would be shaken out? No rush really.... Figure a month before I feel good about moving everything over.
2. Last call on Raidz2 vs Raidz3. Critical data will be backed up. Non-critical data, and most of my space needs, includes DVD/Bluray/CDs. I can re-rip all of that if I absolutely had to. Leaning to Raidz2 which would give me ~19.2tb of usable space. I have 8 x 4tb WD Red drives.

I gotta tell you, the best thing I did was buy this case off of ebay. It is pretty sweet considering what I paid. If anyone is interested, I can post pics.

 

[Jailer]

Since you're starting fresh I'd go with 9.3. No sense starting on an older version if you don't have to.

If it were my build I'd go with RAIDZ2. It's still a non optimal number of drives but a better configuration than RAIDZ3 with 8 total drives.

 

[jeffreyj900]

Got it up and running with v9.3 and a RAIDZ2 volume. At this point, I'm in learning mode. I didn't encrypt the volume although it would be nice to. Figure I will play around for a while. Once I get it figured out, I will wipe it and set it up proper.