BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[pclausen]

Hi guys,

So I've decided to jump on the FreeNAS bandwagon with both feet!

I've been following DataKeeper's thread with great interest and my build will be very similar, only less CPU and RAM, but more spindles. :D

Back in circa 2006 I started with a hacked up chassis with the following internals:

Asus P5WDG2-WS Motherboard
Intel D805 CPU @ 3.8GHz
1 Gig RAM
Areca ARC-1170 RAID6 controller with 24 SATA II Ports
8 320GB Seagate perpendicular drives

Here are some pics of that contraption:

Then in 2008 I got my first 846 chassis and moved the internals from the above cabinet into it.

Things looked a lot better!

Next I got different length SATA cables and a full complement of 24 at that.

In 2009 I swapped out the Asus with a SuperMicro x7dwn+ and also added a Areca 1680 controller and a 2nd 846 chassis (with the dreaded SAS1 expander).

I ran like that until least year when I made the move to software raid a couple of really close calls with both of my 24 disk RAID6 arrays. Yes, I know I was pushing my luck big time!

So I picked up a pair of IBM 1015s along with a pair of SAS2 backplanes for the 846 chassis. I had a number of causalities making the transition, but no loss of data!

I ran FlexRAID for a while, but had various issues. I then switched to SnapRAID and while seemingly better than FlexRAID in most regards, I had issues with unrecoverable errors.

So here we are, lessons learned of the past, and ready to turn the page and accept that parity should represent more than 9% of your total data storage. :)

I took my secondary chassis off line, on right below:

I then pulled out that little SuperMicro power board, and installed the fresh X10 board:

There is only one chassis fan header on the back of this particular X10. I'll either have to extend the wire on the 2nd fan, or get a splitter. And yes, I plan to steal back all the 4 wire fans and cables from the other chassis ones this one becomes primary. :)

I went with the bottom of the barrel 1620 CPU for now. I figured if it ends up not cutting it, I can upgrade to the 1650 and hopefully recoup most of what I paid for this on eBay.

CPU installed:

I went with the SuperMicro P0048AP4 cooler. Here it is after converting it to narrow ILM:

Installed:

Memory should be here tomorrow (2 x SAMSUNG 16GB 288-Pin DDR4 SDRAM ECC Registered DDR4 2133 (PC4-17000)). So I'll hopefully be able to boot up and start running MEMTEST86 for a few days.

I know that this board likes 4 sticks at a time, but I'm going to see how 2 works out initially before biting the bullet on 2 more sticks.

After giving it a lot of thought, I think I've decided on doing 9 VDevs each with 8 disks in RAIDZ2. That will fill out 72 bays nicely. :)

More to come!

 

[Hobbel]

nice pics! hardware specs are similar to mine (https://forums.freenas.org/index.php?threads/x10slr-f-with-16-hdd-help.30169/), I will start building next week.

thx for the hint on the chassis fan. ;)

 

[DataKeeper]

Great photos! My wife would shoot me if a second 24-bay chassis showed up! That Supermicro CPU fan/heatsink looks so small after the Noctua I have :D

When I was attaching the fans I was kinda put off on that one single header back there as well. Here is how I plugged them in..

The Noctua NH-U9DX i4 has two CPU fans on it. It also came with a Y-splitter
The front Noctua CPU fan I placed on FAN1.
The rear Noctua CPU fan I placed on FAN2 using the included y-splitter into the 4-pin end.
The front 3 Chassis fans I placed on FAN3, FAN4 & FANA
The Rear Chassis fan by the PSUs I pulled forward and placed straight into the 3-pin end of the y-splitter going to FAN2
The Rear Chassis fan near the PCI slots I placed on FAN5 on the back corner.
​

This worked out perfectly for me with everything routing nice and neatly. Sensors show the following readings on HeavyIO Speed:

FAN1 - 900 rpm ; FAN2 - 900 rpm ; FAN3 - 4000 rpm ; FAN4 - 4100 rpm ; FAN5 - 3500 rpm ; FANA - 4100 rpm
​

With no drives installed ram temps are sitting at 26c, cpu & pch at 30c and system temp at 18c. Gotta love basements! :D

 

[marbus90]

I hope you removed all excessive standoffs. Common error it seems..

 

[pclausen]

Thanks DataKeeper, I plan to route my cables very similarly.

Good point marbus. I made sure to double check before mounting the board. I of course removed the 4 standoffs holding the little power board right away.

This mobo has 9 mounting holes and only 8 lines with my 846 chassis. I had to remove the "middle" standoffs as it was off by several inches. This one by the red arrow:

In this pic you can also see I got my first IBM 1015 installed. Flashed down to P16 and ready to go! :D I'll pull it back out for initial power on test, though. Decided to work from home today waiting for the RAM to show up... :D

 

[DataKeeper]

All the hole lined up perfectly for me when I installed it...

What? I just 2 minutes ago went to flash my M1015 to P16 IT mode and while the first 2 megarec cmds went fine and rebooted I got the damn PAL error on the first sas2flsh cmd!

 

[pclausen]

You will get the PAL error when flashing on a UEFI mobo using the DOS sas2flsh.

You have to use an older non UEFI mobo when flashing with the DOS sas2flsh.

Or

You have to boot into UEFI shell and use the sas2flash.efi utility.

This is what you want on your USB stick:

 

[pclausen]

So my memory showed up!

Installed in DIMM slots A1 and C1 for "symmetry" :)

It lives and sees all 32GB of RAM!

Time to check for latest BIOS and determine how to configure BIOS.

 

[marbus90]

Check out IPMI first. :P

IMHO the rear fans aren't neccessary since the middle fan wall should push enough air trough the chassis.

 

[DataKeeper]

LOL Whats with the monitor? I can happily say I've done everything via IPMI so far! I had a screen setup on the first boot but 2 minutes later I was upstairs in my office connected via IPMI. :p Java update messed me up for a bit but got through that on the X10 board. Its still not working right for the X8 however.

Always good to see them come alive!

 

[pclausen]

Haha, yeah I know. Old habits die hard I suppose. Got it up and running now though and this thing is cool as a cucumber compared to my X8!

Did a clean install of FreeNAS to a 16GB USB stick, which I have mounted in the internal USB 3 socket on the motherboard.

I also re-installed the 1015 and ran a single SFF-8087 cable for now to the backplane. Got 8 2TB drives in SAS slots 0-7. Waiting on a 9th drive to have all data moved off to other drive members in my old pool, so that I can start out by configuring a 9 disk RAIDZ2. I decided that 9 divides into 72 just as well as 8 does. :D This way each of my VDevs will have 9 members, and there will be 8 of them total instead of the other way around.

One thing I haven't looked into yet is if I should considder upgrading the firmware on my 2TB Hitachi HDS722020ALA330 drives.

EDIT: Looks like I have a mix of firmware versions running on my Deskstars. Some are on 28A and some on the latest 3MA. Might as well get them all on the most recent version.

 

[marbus90]

I'd rather do 10disk raidz2 and reserve 2 bays for hotspares.

 

[pclausen]

Agreed marbus. After tossing around pros and cons some more, doing the max recommended raidz2 size makes the most sense I think. So I got 10 2 TB drives (8 old DeskStars + 2 Seagate NAS) emptied from my Windows based pool and physically added to the new X10 server. The DeskStars were all upgraded to 3MA firmware prior to adding as seen here:

Memtest86 had also been running for 24+ hours with no errors. Here's a screenshot from fairly early on showing how it was doing at that time. No errors after 24 hours either.

After the memtest, I went ahead and stole these goodies from the current production 846 chassis:

After the conversion, I brought the system back up with "optimal" fan speed. It sure was nice and quiet, but I later found out that my disk drives were being cooked! I only discovered this as I removed drives that failed SMART test. Is there a way to monitor disk drive temps from the GUI?

Here's a shot of the X10 chassis guts in it's operational state. Note USB stick on on-board USB 3.0 connector and IBM 1015 installed. This shot also shows the 4 wire mid plane fans all wired up, but as I mentioned above, with optimal fan speed, the disk drives run way hot! You can also see that I'm not running any rear panel fans at this point. CPU and RAM temps are higher than before, but I think that is all related to mid plane fan speed. I've switched all midplane fans to full speed for now to cool down the HDs.

So I went ahead an setup 2 10 disk VDevs, each with 10 raidz2 members as follows:

Unfortunately, the physical bays don't like up with the da# IDs consistently, and they change after a reboot *sometimes*...

So my 2nd Vdev had has some SMART errors what I was very much aware of, and FreeNAS sure did pick up on those (as seen in the console window):

I'm concerned that I was not emailed about these critical errors. Email in general appears to be working fine however, so not sure why the SMART ones aren't getting through.

So here's a view of the way things are currently configured from a disk drive perspective:

A quick test copying data from the X8 based Windows server is showing excellent throughput:

Being a complete newbie to this, I'm wondering if I should worry about errors such as these in the log?

I have a ton of learning/reading in my immediate future, but any advice on my email error and how to get real time hd temp reading and overview of SMART errors would be much appreciated!

Once I stress test everything,a big challenge is going to be how to copy all of this from my X7 based server over to the new X10 FreeNAS system:

How far can I fill up the 2 Vdev's with data before I add a 3rd one? Getting my 3rd 846 chassis (filled with 1TB drive, so 2 more 10 disk VDevs, is probably still a few weeks out.

 

[marbus90]

Just drop everything in the current pool and add the other vdevs later on. You'll still get lots of performance out of that, even if the data is not optimally balanced between vdevs.

 

[DataKeeper]

I'd rather do 10disk raidz2 and reserve 2 bays for hotspares.

I thought hotspares were not looked well upon in freenas.. Or was that just old thought?

Once I thoroughly test my drives I was planning to have two additional spare, and tested, drives on hand. Perhaps even in spare caddies.

 

[danb35]

How do I monitor disk drive temps from the GUI?

You don't. You set a temperature threshold in your SMART configuration, and the server emails you when it's exceeded (see below).

I'm concerned that I was not emailed about this critical errors. Email in general appears to be working fine.

SMART email configuration is separate from the root user email configuration. For SMART, go to Services -> SMART -> click on the wrench and enter the email to report. That's the same place you can set the drive temps for notification.

 

[danb35]

I thought hotspares were not looked well upon in freenas.. Or was that just old thought?

A recent update (within the last month, IIRC) said it added hot spare functionality. I don't think I've seen any reports on if or how it works.

 

[marbus90]

it was the 201502271818 update. :)

 

[DataKeeper]

Ok so that's cool. Didn't realize it had changed. Thanks

 

[pclausen]

You don't. You set a temperature threshold in your SMART configuration, and the server emails you when it's exceeded (see below).

SMART email configuration is separate from the root user email configuration. For SMART, go to Services -> SMART -> click on the wrench and enter the email to report. That's the same place you can set the drive temps for notification.

Thanks. But I *did* already set my email address there, but it still isn't working. Sorry for leaving out that detail in my previous post. So I just went in and changed the notification values to 25 for Information and 30 for Critical to force some events.

All the drives were scanned and I could see the temps of each drive as they were read. However, I still didn't get any emails, and here's the error in the log:

Btw, this is a fresh install of FreeNAS-9.3-STABLE-201504152200 performed yesterday. So it looks like
/tmp/.smartalert is missing. Do I need to manually create that file?

UPDATE: I switched to the following nightly:

FreeNAS-9.3-Nightlies-201504250630

And SMART updates via email is now working!