BUILD Hardware selection - 1st FreeNAS build

[DataKeeper]

1. Yes I believe it's the same.
2. You only need 1 M1015 card though a spare on the closet is always good. Use both ports and connect the two cables to the backplane. This is all you need to do.
3. I don't know what fan wall he is using in the video. Any of the fans can be replaced with less noisy fans however your results will be your own. Personally, hacking up a solid chassis like these like this is not something I'd do. Please be aware this is not a low noise chassis and the fans and PSUs are designed to cool the system properly. By hacking them up the system will not pull as much air through the system for cooling.
4. Nope, just order the screws off eBay. When I was looking I seem to remember shipping was a long time so consider that.
5. No, I don't believe the bays will take the 2.5" drives without extra brackets. **Note however there is a tray you can order that inserts into the rear of the chassis that DOES hold 2 x 2.5" drives. I don't remember the SuperMicro part number but an email or call to them would likely be easy enough.. Ran about $50 bucks iirc. If you come by the part number please post it here for me. :D Someone else here might know what it is and toss it up.

 

[jgreco]

@DataKeeper Your build looks awesome and something I am planning to replicate. I presently have a very quiet 4U Rosewill RSV-L4500 case which i am planning to upgrade with the Supermicro chassis. I have questions on your build:
1. The Supermicro chassis selling on ebay is 846E160-R1200B with backplane BPN-SAS2-846EL1. Is this the same backplane in your build?
2. I will be using 4TB WD Red drives. My understanding is that the backplane has 3 connectors. To get the most out of the 24 drives it will need a bandwidth of 3.5Gbps (assuming 150MBps per drive). Do i need 3 LSI IBM m1015 cards connected to each of this connector or is 1 sufficient? I understand each IBM M1015 has two ports, so only 1 needs to be connected?
3. The chassis will be next to my desk so low noise it quite important to me. Do you know what mid fan wall is being using in this? Also, can the rear 80mm fans be replaced?
4. I guess i need the flat head screws for the HDD. Anything else Supermicro specific i may be missing?
5. I am planning to use mirrored 2.5 inch SSD for boot. Do the HDD trays take 2.5 inch drives?

Do not attempt to replace the fans on the fan bulkhead in the 846 chassis.

https://forums.freenas.org/index.ph...-hds-recommendations.24453/page-4#post-255614

 

[TXAG26]

Get a large Caselabs case if you need to house 30-40 drives quietly. Plenty of room to space the drives out and keep them cool using quiet fans.

I have one of these that nicely fits the bill. You do loose the backplane (unless you go with an aftermarket HDD cage solution), but if silence is a priority, this is a really good option. They now have a "Wide" magnum that holds even more HDDs.

http://www.caselabs-store.com/magnum-th10a/

 

[trumee]

Get a large Caselabs case if you need to house 30-40 drives quietly. Plenty of room to space the drives out and keep them cool using quiet fans.

I have one of these that nicely fits the bill. You do loose the backplane (unless you go with an aftermarket HDD cage solution), but if silence is a priority, this is a really good option. They now have a "Wide" magnum that holds even more HDDs.

http://www.caselabs-store.com/magnum-th10a/

I was looking for a rack mount case, these look like tower case. At the moment i am using a Rosewill RSV-L4500 which is very silent. However, it only has 15 bays and no back plane.

 

[trumee]

Do not attempt to replace the fans on the fan bulkhead in the 846 chassis.

https://forums.freenas.org/index.ph...-hds-recommendations.24453/page-4#post-255614

This looks like good for a basement but not in an apartment. My google-fu did not yield much information on high hdd temperatures with a modded fan wall.

Does anybody have a numbers to show the temperatures with 120/140mm fans?

How about a Norco RPC-4224, is it quieter than the Supermicro chassis?

 

[jgreco]

My google-fu did not yield much information on high hdd temperatures with a modded fan wall.

Right. Most of the people who bake their drives don't 'fess up.

Does anybody have a numbers to show the temperatures with 120/140mm fans?

You *definitely* don't want 120mm or larger fans with a chassis design that relies on static pressure for airflow.

How about a Norco RPC-4224, is it quieter than the Supermicro chassis?

Yup, especially if you get the 120mm fan wall. Side benefit: ability to bake cookies.

 

[trumee]

Right. Most of the people who bake their drives don't 'fess up. :)



You *definitely* don't want 120mm or larger fans with a chassis design that relies on static pressure for airflow.



Yup, especially if you get the 120mm fan wall. Side benefit: ability to bake cookies.

The Supermicro chassis are built for 15k drives. The WD red drives have lower power consumption/heat and should need lower heat dissipation?

Is there any other rack mount Supermicro chassis which can be made quiet?

 

[Ericloewe]

You can't physically build something really quiet that crams 24 drives into a 4U front panel without baking them. The fans definitely should not be at 100% constantly, though.

 

[jgreco]

The Supermicro chassis are built for 15k drives. The WD red drives have lower power consumption/heat and should need lower heat dissipation?

That's true. However, that is an incredibly tricky game in practice. The Red drives are not any smaller than the 15K drives, so you have to force the air through the same tiny gaps between drives that you have to with the 15K drives. In other words, you still need fans capable of generating the low pressure zone (in front of the fan bulkhead) that causes air to want to enter that space by working its way around your drives. It is important for this to be an actual low pressure plenum area, because you need air to be coaxed into the chassis around *all* the drives. That's why you can't run these things with their cover off.

Now the good news is that for the Reds, those fans might need to work less-hard and could be run at a lower RPM. And Supermicro's mainboards are definitely designed for this possibility. You can set the chassis fan thresholds in the BIOS for different use models, and then allow the temperature sensors on the mainboard try to regulate the fans.

Is there any other rack mount Supermicro chassis which can be made quiet?

Not really. Around 10 years ago, several companies (including AIC and Supermicro) recognized that the advent of SATA and SAS meant that they could make high density storage servers that would tackle both the enterprise high performance and data center mass storage markets. At the time, everything was about rack density, and it turns out that 12 3.5" drives fits into a 2U 4-wide 3-high very nicely, but with relatively little room for airflow. In both the enterprise IT room and in the data center, this is not deemed to be a problem because of high pressure differential fans. ==NOISE.

There are some chassis like the SC836 which are perhaps even worse, because they have some increased venting (see the top front) but this comes at the cost of a reduction in pressure differential which means your drives might be cooled less.

This problem wasn't so bad back in the '90's and we did actually have some good options for cooler, quieter running. I've still got some Kingston StorCase arrays in inventory here; 4U, 9 bay:

and then you put some 3.5 to 5.25 modules in that, and then yes you could definitely tweak the fans to something more quiet-y. But you're only getting 9 drives into 4U of space.

You can definitely still find things like this. Look at this:

http://www.newegg.com/Product/Product.aspx?Item=N82E16811165180

You wanna know how I know this could be made to work? I can *see* through it, and see lots of open space. That means it is possible to get airflow without wasting lots of energy on high pressure fans. But the thing is, you've got to be careful. The natural inclination here would be for someone to go "Ah! Three 5.25 bays on each side! I'll go get some five-3.5-in-three-5.25-bay modules" and wind up with 10 drives in there, but we're immediately back to the forced airflow issue. You could probably find an acceptable four-3.5-in-three-5.25-bay module out there somewhere, but most of those require fans as well. This gets very twitchy because doing a good job of cooling is just naturally at odds with low noise. I'm not even seeing any good trays for a six bay configuration. Of course if you don't need trays the easy solution here is just to go get some 3.5-to-5.25 brackets and mount six drives and call it a day.

 

[Mirfster]

I bet if you went all SSD, that it would help... ;) /Devils Advocate...

 

[jgreco]

I bet if you went all SSD, that it would help... ;) /Devils Advocate...

Actually this brings up an interesting point. One of the key assumptions with these chassis is that you're loading them with standard size (1" high) 3.5" drives.

In the 2.5" chassis, we often aren't using 2.5" 15mm height drives and this changes the airflow characteristics - in particular, it is kinda bad to mix and match drives because if you stick in some 15's they won't cool properly.

So you COULD potentially use a 24 bay chassis with reduced height ("slimline") drives such as the 20mm high ST1000DM003 1TB drives. The airflow is substantially improved (though goes *over* the drive, rather than both over and under). At this point, I suspect that the problem then merely moves back towards the backplane. The Supermicro looks something like this.

To me this looks like they've assumed that these cutouts provide enough space to allow the low pressure zone to extend to the front side of the backplane, which I'm guessing is true.

However, if you're looking at increasing the airflow around the drives by subbing in a slimline drive, and replacing the high static pressure fans with regular fans, then you're probably looking at some dead zones for airflow, especially around the bottom middle drives. And I'm guessing that you'd have problems even with the high static pressure fans because you've changed some of the fundamental assumptions made that allow this backplane to be used.

 

[Jatrabari]

OK so the system is together minus drives and M1015. On a quick power up I was seeing the front panel OverHeat LED come on and of course the fans were power cycling so I shutdown. Got into IPMI via IP and found the CPU is running at a cool 36C degrees however the FAN1 was causing the issue. Quick fix on that got it up and running. Changed the fan mode to "HeavyIO Speed" which stopped the fans from cycling to a nice continuous medium speed (~4000RPM). Changed the boot order for USB Flash Drive, popped in a flash drive with Memtest86, booted up and test has been running for 1 hour 55 minutes. Currently on Test 9 of 64GB ram.

IPMI, once figured out, ROCKS! :D

A note on the Noctua NH-U9DX cooler and this board, the X10SRL-F, you need to remove the mounting bars from the bottom of the cooler and then remove the plastic lock-rings (4), springs (4) and screws (4). Transfer these over to the included NM-XFB5 90° mounting bar and install that to the cooler replacing the stock ones. This is because of the LGA2011 Narrow ILM and the orientation of the cooler to this board. Takes about 3 minutes is all. The fans run at about 900RPM. *There is an included syringe of NT-H1 thermal Paste included with the CPU. It says a "drop" but its paste which doesn't drop out. I squeezed out about a 5mm circle in the center of the CPU which seems to have worked fine. CPU temp has remained between 38-40 during the memtest86.

Question: Does the IPMI port share MAC addresses/IPs, etc with the two onboard LAN ports? I think the older boards did but wanted to ask if the new ones did or not.

Quick photo of the CPU Cooler, RAM, Sata Cables, Intel X540-T1 10GbE card and wiring. Yes its automotive wire tubing :p left over from my profile pic build. Didn't have anything else and the bare wires were a disaster!
View attachment 7456

edit.. The APC SUA2200RM2U SMART-UPS 2200VA works perfectly... once I picked up a NEMA 5-15P to 5-15/20R adapter for it. Full charge and powering the the freenas chassis and the small 1U chassis which also has a memtest running.

Hello,

I am thinking the same motherboard and Noctua cooler as you have in your server. My question is that is there any problems with the clearance of the memory combs closest to the Noctua cooler so all slots can be populated if needed without issue?

 

[Stux]

How about a Norco RPC-4224, is it quieter than the Supermicro chassis?

It can be made so, without baking drives.

https://forums.freenas.org/index.ph...24-supermicro-x10-sri-f-xeon-e5-1650v4.46262/

 

[fx24]

It can be made so, without baking drives.

https://forums.freenas.org/index.ph...24-supermicro-x10-sri-f-xeon-e5-1650v4.46262/

Correct. I also have have a quiet 4224, but I use Noctua P12s.

 

[Ericloewe]

The upcoming low-clearance 120mm Noctuas should be interesting for these applications.

 

[DataKeeper]

Hello,

I am thinking the same motherboard and Noctua cooler as you have in your server. My question is that is there any problems with the clearance of the memory combs closest to the Noctua cooler so all slots can be populated if needed without issue?

I had no issues during the install nor since with its operations regarding the fitment of the ram and the cooler. Its close but clears fine with lots of air movement.

 

[DataKeeper]

OK... 1st issue in 2 years...

I've been away for the past 2 weeks on my dualsport motorcycle putting about 2500 miles on my bike (Yamaha WR250R) leaving DE, riding through MD and into WV, KY, TN & VA doing a 1000 mile off-road trail called the KAT (Kentucky Adventure Trail) then riding home. I get home and find the following errors:

Code:

CRITICAL: Device: /dev/da17 [SAT], Self-Test Log error count increased from 0 to 1
CRITICAL: Device: /dev/da17 [SAT], unable to open device
CRITICAL: The volume garage (ZFS) state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

When I built the system I installed 19 x 4TB drives so I went into Storage > Volume > Volume Status and sure enough da17 had been Removed and the Volume was in a Degraded state. The system much have Removed the drive automatically since I'm the only one with access and was away.

Clicked on the Removed drive (da17), clicked Replace and a window opened up with da18 as the Member Disk. I selected that and clicked Replace (iirc). It's now doing a Resliver, is at 7.90% complete (scanning 2.88T out of 36.4T) and continues.

Once finished.. should I shutdown the system before removing the da17 drive? I'd like to pull da18 (once the Resliver is done) and place it into the slot I'm pulling da17 from. I'll order a replacement drive later tonight and RMA the faulty drive back to WD for a replacement. Anything else I should do?

Thanks

 

[DataKeeper]

Resilver
Status: Completed
Errors: 0 Date: Mon Oct 30 13:06:00 2017
------------- Excellent! :D

 

[DataKeeper]

OK I had another 4TB drive still in the box from 2 years ago I forgot I had. So... I shut the system down after the successful Resliver today, Removed the bad drive and put the new one in again as a Spare. Before the Spare was simply a drive not used so I manually had to Replace & Resliver, even though the system had Removed the bad drive leaving itself in a Degraded state.

With the new spare installed and recognized by the system on reboot... Can I setup this up to automatically take the place of any future bad drives?

Storage >
Volume Manager >
Select "Volume To Extend" (garage in my case from the drop down) >
Click "1 - 4 TB" >
Select "Spare" under Volume Layout >

Click "Extend Volume" to finish...

Is this correct? Any Pros/Cons to this?
Is it recommended to do a full Snapshot after replacing a drive?

Thanks

 

[Jatrabari]

@DataKeeper Your build is using SATA DOM chips mirrored as boot drives. I am thinking of using also DOMs for boot but I noticed from Supermicro SATA DOM pdf that using Supermicro DOMs as RAID mirror is not recommended... But you haven't had any issues with your build for 2 years so they seem to work just fine.

Does any one know why they are not recommended? They are basically comparable to SSD drives?