Build Report: Norco RPC-4224, SuperMicro X10-SRi-F, Xeon E5-1650v4

[Stux]

I first started planning my build in this thread

The hardware is complete, the server is beginning to take on duties as we go through a commissioning phase.

So, since there has been a fair bit of interest in my build whenever I've mentioned it, and specifically, my impressions of the Norco RPC-4224 (a 4U 24 bay rackmount case), I'm going to post the nitty gritty.

(24 hot swap 3.5" bays + 2 hot swap USB boot drives ;))

Chassis: Norco 4U 24 Bay Rackmount RPC-4224
Rails: Chenbro 26" Slide Rails CSPC-SLIDE26N
Motherboard: Supermicro X10-SRi-F
CPU: Xeon E5-1650 v4
CPU Cooler: Noctua NH-U9DX i4
RAM: 2x Crucial 16GB ECC RDIMM DDR4-2400
PSU: Corsair RMx 1000
UPS: 5U SmartUPS 5000VAC
HBA: IBM ServeRaid M1115 (cross-flashed to LSI-9211-8i)
SAS Cables: 2x 0.5m 4 x SATA to SFF-8087 (Reverse Breakout), 4x 0.65m SFF-8087 to SFF-8087
HD Fans: 3x 120mm Noctua NF-F12 PWM
Exhaust Fans: 2x 80mm Noctua NF-A8 PWM
Boot: 2x Cruiser Fit 16GB USB3.0
HDs: 8x 4TB Seagate NAS-HD

Future plans include:
More Ram.
Another HBA.
2 more 8 drive vdevs.
PCIe NVMe SSD drives
10gbe Networking
And maybe some 2.5" SSDs.
And maybe one day a bigger Xeon.

Overall impressions... Well... I'm happy now... BUT... it took a lot of effort to get there. The box is super silent, and elegant enough. It looks good in the rack.

The biggest issue was that the Norco rails were not compatible with my rack. And I wanted to have the chassis on slide rails. In retrospect I should've just got a different chassis, but I wanted to go with this one because it was 'good value', in hand, and because of its 120mm fan wall had the potential for silence.

And then it became a challenge to be beaten into submission.

The Norco chassis came with the 120mm fan wall and the Dual 2.5" drive tray. As well as screws, bay trays, and stuff to use redundant PSUs. And no instructions. There are none. Nowhere. Nohow. Their website does not have them. They don't exist!

The issue with the rails is a 19" rack has an 18" opening and the nominal width between support posts is 18". The Norco chassis is 17" wide. This leaves 0.5" on either side for the rails. The Norco rails are wider than and this do not fit if your rack posts are subtantial, for example, my rack is heavy duty and uses S post rails, rather than the much flimsier L posts. The Norco rails are about 15mm thick and they need to be 12.5mm.

(Chenbro 4U 26" slide rails, modified to fit. These are the inner rails, the outers get installed in the rack. Easily, they click and slide in. Great design)


This led to me trying to find an alternative. I did find some ex-demo rails cheap, and I thought they looked like they just might work. Well, they did, but I had to do some hardware hacking to get them to fit onto the HD tray bracket hook thing... and then I had to find suitable screws (M4 I believe), washers and bolts, and then I had to modify the screws so the rails would slide...

(detail of the hacking I had to do to the chenbro's to make them compatible. Its hardened steel. I needed a tungsten carbide dremel bit)

(detail of the side rail screw hacking. I had to use M4 screws (or was it M3?). Then I had to use extra thin washers, and then I still had to file 0.5mm of the head of the screws. Also I had to drill a hole for one of the screws in the case)

(this was the drilled hole. Again, washers... a lot of potential weight... although its in a different plane)

(this was one of the pre-threaded holes. I didn't want to rely on just the thread, so I used some slightly larger washers the sat around the thread, then I could really torque down on the nut)

Okay, next issue, the HD backplanes. Actually they're quite good, 6 blackplanes, horizontally laid out, so barely any interference in air-flow, and then the 3 x 120mm fan wall.

(good shot of the backplane area, showing how I moved two of the fans from the factory location for better access. If you look at the 2.5" drive tray you can see the fans would interfere)

By default the fans are installed on the drive side of the fan wall, so that you can get an E-ATX board in. If you install them on the other side, then you get much better access to the SAS ports and the Molex 4-pin peripheral power connectors. BUT then you can't mount the 2.5" drive tray... so out came the dremel again.

(detail of dremel modified drive tray to resolve fan interference problem)

EDIT: This thread continues on the new forum, if you want to reply to something:

Build Report: Norco RPC-4224, SuperMicro X10-SRi-F, Xeon E5-2699A v4 Part 2

Continuing on from the old forum post I recently upgraded the CPU from E5-1650 v4 to the Unicorn model… the E5-2699A v4. This is an upgrade from 6 cores to 22! with a pretty good base clock. Broadwell era Xeons are cheap now and I get a kick out of dropping in a CPU with an RRP of 10K...

forums.truenas.com

 

[CookiesLikeWhoa]

I envy your quiet setup. The supermicro chassis are awesome and built really well, but even in "power saver mode" for the fans they are still loud.

How warm was the office if you had to guess? Those temps seem great for RPM controlled 120mm fans.

 

[Stux]

The backplanes are simple affairs. They each support 4 drives, they're just passive breakout backplanes, with hotswap, take one SFF-8087 connector, and a 4 pin molex connector. My PSU was big enough, and came with 3 sets of molex peripheral cables, so I ran all 3 to the backplanes. 2 backplanes per cable. Connected on the earliest connectors on the cable. This distributes the load best.

(PSU cabling. CPU fans, and HD fan cables can be seen. The PSU will run without the fan spinning up to about 400w, which is nice, although it does get warmish)

Now, that led to the next problem which is the stock fans are loud 3 pin fans. I wanted smooth, quiet, controllable PWM fans, so I replaced them with Noctua 120mm fans.

(detail under the 2.5" drive tray, sata reverse breakout cables can be see, also 2x internal USB3 boot drives. One on a header mounted USB3 port (front ports are USB2). FANA is in the corner)

I used 3 of the Noctua Y splitters to split the FANA connector to power the fans, and some of the Noctua extension cables they include. FANA is important later.

Next I decided that the exhaust fans were now annoying, even slowed down with the Noctua slow-fan cable. So I replaced them with Noctua 80mm fans. I used two extension cables to run the fans to FAN3&4.

The CPU Cooler fans were run to FAN1&2.

And that was the end of the hardward build.

Then I began commissioning the server, I started by running memtest, badblocks... then I actually installed Windows on a drive to do the thermal testing. I found the Supermicro board did not like the fans. Long story short, there were two problems. I had to adjust the BMC's fan threasholds... And the Supermicro default "Optimal" fan control algorithm liked to stall the fans... and then they would surge up.

It wasn't very good anyway, so I wrote my own Hybrid CPU/HD Fan Controller script. This script is great. Basically, it runs the fans at the right speed to provide the right cooling... which means... its pretty much silent, unless I'm *really* flogging the CPU. More on that later I guess. If I'm doing a scrub, the HD fans will spin up a bit, but you can't really hear them. If the CPU is getting a moderate flogging it will spin the CPU fan & exhaust to High. If its a real flogging, and runs for a few minutes, eventually the HD fans will spin up to maximum to provide the additional cooling. And as soon as the CPU abuse stops, everything goes back to 30% and relative silence.

Flogging the drives and CPU (mprime) in a warm office for a full weekend resulted in HDs at 28C and CPU no hotter than 65C.

Anyway,

I'm very happy with it now, but in retrospect I wasted a helluva lot of time on those damn rails!

I think it was worth it. My big concern with something like the Supermicro 4U chassis is I wanted something which could be made if not silent, then polite. This thing is quieter than my gigabit switch! I wouldn't put it in a living room, but basically, you have to hold your ear to within a few inches of it to hear it.

And I wanted to build a system that would last for 5-10 years. If the components are good, and I think they are, then this should.

About ready to go in the rack now...

(detail of rack rails in rack, showing the S-post that caused me so much grief. I really like these rails by the way. They just click in magically, support locking in, removal, long standout, etc)

 

[Stux]

I envy your quiet setup. The supermicro chassis are awesome and built really well, but even in "power saver mode" for the fans they are still loud.

How warm was the office if you had to guess? Those temps seem great for RPM controlled 120mm fans.

30C. Its gets very warm over the weekend because the building AC doesn't run on the weekends, unless someone is working back.

I have the Fan Controller targeting a maximum of 38C for the drives. The higher this setting the quieter. What this actually means is the low threshold for the CPU fans is <36C, ie 35C or lower.

The CPU is also targetted at 35C, so essentially the whole box runs at 35C, with all fans at 30% RPM. with the fans just kicking up a little bit to knock things back below 40C for the CPU occasionally, or 36C for the HDs.

Code:

#	tail -f ./fan_control.log
2016-09-24 01:32:09: Maximum HD Temperature: 35
2016-09-24 01:32:38: CPU Temp: 35 <= 35, CPU Fan going low.
2016-09-24 01:34:00: CPU Temp: 40 >= 40, CPU Fan going med.
2016-09-24 01:35:11: Maximum HD Temperature: 35
2016-09-24 01:36:19: CPU Temp: 35 <= 35, CPU Fan going low.
2016-09-24 01:38:01: CPU Temp: 40 >= 40, CPU Fan going med.
2016-09-24 01:38:12: Maximum HD Temperature: 35
2016-09-24 01:40:26: CPU Temp: 35 <= 35, CPU Fan going low.
2016-09-24 01:41:13: Maximum HD Temperature: 35
2016-09-24 01:42:01: CPU Temp: 41 >= 40, CPU Fan going med.

Now, if I were to run mprime with 1 thread, then my CPU fan would fluctuate between med/high getting an RPM somewhere between the two holding my CPU at about 49/50C, and the HD fans wouldn't need to kick in... this will work indefinately. And I can barely hear the CPU fan.

Code:

016-09-24 01:45:13: CPU Temp: 35 <= 35, CPU Fan going low.
2016-09-24 01:47:15: Maximum HD Temperature: 35
2016-09-24 01:48:00: CPU Temp: 40 >= 40, CPU Fan going med.
2016-09-24 01:50:11: CPU Temp: 50 >= 50, CPU Fan going high.
2016-09-24 01:50:16: Maximum HD Temperature: 35
2016-09-24 01:50:17: CPU Temp: 48 >= 40, CPU Fan going med.
2016-09-24 01:50:34: CPU Temp: 52 >= 50, CPU Fan going high.
2016-09-24 01:50:45: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:50:50: CPU Temp: 52 >= 50, CPU Fan going high.
2016-09-24 01:50:52: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:51:02: CPU Temp: 50 >= 50, CPU Fan going high.
2016-09-24 01:51:22: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:51:26: CPU Temp: 52 >= 50, CPU Fan going high.
2016-09-24 01:51:30: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:51:38: CPU Temp: 52 >= 50, CPU Fan going high.
2016-09-24 01:51:41: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:51:46: CPU Temp: 51 >= 50, CPU Fan going high.
2016-09-24 01:51:47: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:51:51: CPU Temp: 51 >= 50, CPU Fan going high.
2016-09-24 01:51:52: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:51:53: CPU Temp: 51 >= 50, CPU Fan going high.
2016-09-24 01:51:56: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:51:58: CPU Temp: 51 >= 50, CPU Fan going high.
2016-09-24 01:51:59: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:52:00: CPU Temp: 51 >= 50, CPU Fan going high.
2016-09-24 01:52:10: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:52:24: CPU Temp: 50 >= 50, CPU Fan going high.
2016-09-24 01:52:57: CPU Temp: 48 >= 40, CPU Fan going med.

but with 4 thread (ie 33% serious usage)

Code:

2016-09-24 01:54:03: CPU Temp: 39 dropped below 40, CPU Fan going med.
2016-09-24 01:54:58: CPU Temp: 50 >= 50, CPU Fan going high.

The CPU will stay below 60C for a long time without any additional cooling from the HD fans

But if I now set 12 threads, for 100% maximum cpu heat generation, then

Code:

2016-09-24 01:54:03: CPU Temp: 39 dropped below 40, CPU Fan going med.
2016-09-24 01:54:58: CPU Temp: 50 >= 50, CPU Fan going high.
2016-09-24 01:56:17: Maximum HD Temperature: 35
2016-09-24 01:56:38: CPU Temp: 49 >= 40, CPU Fan going med.
2016-09-24 01:56:47: CPU Temp: 54 >= 50, CPU Fan going high.
2016-09-24 01:56:49: CPU Temp: 60 >= 60, Overiding HD fan zone to 100%
2016-09-24 01:59:21: Maximum HD Temperature: 34
2016-09-24 02:02:22: Maximum HD Temperature: 33
2016-09-24 02:05:24: Maximum HD Temperature: 33
2016-09-24 02:08:25: Maximum HD Temperature: 32
2016-09-24 02:11:31: Maximum HD Temperature: 32

And although all fans in the case are now at 100% from about 3M away, I can basically just hear a vague hum. Eventually the HDs will continue to drop in temperature to about 28C if I keep the CPU pegged like this because of the HD fans being on 100%, and the CPU will probably climb to 65C and stop there. Last time I tested that the HDs were undergoing a stress test, heating up the intake air.

And now I kill mprime...

Code:

2016-09-24 02:14:01: CPU Temp: 46 >= 40, CPU Fan going med.
2016-09-24 02:14:01: Restoring HD fan zone to 30%

The response is immediate.

 

[CookiesLikeWhoa]

That's why Noctua fans are incredible! I have our NAS set up in the office sitting at one end. The fans are at 100% and when no ones in you can hear them humming away but once people are in the office working the easily get drowned out by talking or the radio.

You would really have to be slamming that FreeNAS to get all of those threads used up. Those most I've ever seen our NAS get hit for was 50%. Three plex streams going, 2 editors transferring to it, while a VB Win10 instance was running and I was working to set up wordpress in a jail (still can't get it to work -.-).

Mad props on your cable management by the way. Looks really good!

 

[Stux]

That's why Noctua fans are incredible! I have our NAS set up in the office sitting at one end. The fans are at 100% and when no ones in you can hear them humming away but once people are in the office working the easily get drowned out by talking or the radio.

You would really have to be slamming that FreeNAS to get all of those threads used up. Those most I've ever seen our NAS get hit for was 50%. Three plex streams going, 2 editors transferring to it, while a VB Win10 instance was running and I was working to set up wordpress in a jail (still can't get it to work -.-).

Mad props on your cable management by the way. Looks really good!

Yeah. We do video encoding and compilations and stuff like that sometimes, and the plan is to put a number of different VMs on here.... Although mprime can saturate the CPUs like that... not much else will to that extent even if it was 100% utilized, and the VMs will be idling most of the time.

The rest of the time, its just a backup target (a TB of deltas a day or so) and a file server.

 

[Dice]

Cool to have a 'report-post' on the build.
Love it.
I'm looking forward to trying to implement your script at one point.

 

[droeders]

Great post and thanks for taking the time to put all of this together. Despite your frustrations, I think I'm going to build my next NAS using this case. Might not have tried it without seeing your results, so appreciate you being the guinea pig. :)

I love my Supermicro gear, but it sits close to me and the noise is bothersome. If I add another SM chassis, it will be unbearable.

I have some universal rails that I used with my Norco RPC-370 and I haven't had any issues with fit on this case. I used the RL-01 rails from here:

http://www.rackmountmart.com/html/chAss.htm

If I build with the RPC-4224 and these rails, I'll report back my findings.

 

[Mlovelace]

I have two of Norco RPS-4224 and they're well worth the money. I also swapped out the fans, but not with quiet noctuas, I put in 120mm 253cfm delta fans. They are loud but so is everything in our data center.
http://www.newegg.com/Product/Product.aspx?Item=35-706-015

 

[Stux]

Those paying attention may have noticed that I installed the first 8 drives in a zig zag order in the top 4 rows. I actually did this to test both onboard and HBA SATA/SAS connectors initially, and then because its a pretty pattern, and then realised that it provided greatly enhanced cooling because of the large air channels between each HD.

( 8 x 4TB actively serving NAS duties, 5 x 3TB WDC Red/Green undergoing badblocks tortue testing)

Today, I added another 5 drives by filling the upper most 'holes' in the array. I wanted to run badblocks on the drives, without suffering from bottlenecks from inadequate hardware, and it occurred to me that it would be a good test of the cooling system, as well as hot swap capabilities. I had some ideas on how to improve cooling if needed.

So first things first, my Hybrid Fan Controller script needed to actually deal with hotswap. Now it does.

Fairly quickly after starting the bad blocks pass the drives started heating up waaaay too fast.

Code:

2016-09-26 17:19:32: Maximum HD Temperature: 35
2016-09-26 17:25:45: Maximum HD Temperature: 36
2016-09-26 17:25:45: Drives are warming, going to 50%
2016-09-26 17:28:47: Maximum HD Temperature: 36
2016-09-26 17:31:48: Maximum HD Temperature: 36
2016-09-26 17:34:50: Maximum HD Temperature: 36
2016-09-26 17:37:50: Maximum HD Temperature: 37
2016-09-26 17:37:50: Drives are warm, going to 75%
2016-09-26 17:40:52: Maximum HD Temperature: 38
2016-09-26 17:40:52: Drives are too hot, going to 100%
2016-09-26 17:43:52: Maximum HD Temperature: 39

This is when I said "Oh Shit." and ran to implement my first course of action.

It was quite noticable that all the incoming air was by-passing the block of HDs at the top of the bay area and instead coming in the remaining holes below the bay area... you could feel it with your hand.

Norco provides some packaging material with their chassis. It just so happens that its a nice density and just slightly thicker than an HD.

(Norco packing foam is just the perfect thickness to make drive tray dummy blocks)

So, I cut some strips slightly wider than the drive trays, then cut those into smaller blocks. They seemed to fit perfectly into the drive trays... and in went my custom designed dummy drive trays. The fit is very good, and the foam is easily removed with the friction fit.

Thankyou Norco.

Code:

2016-09-26 17:49:54: Maximum HD Temperature: 39
2016-09-26 17:52:55: Maximum HD Temperature: 39
2016-09-26 17:55:56: Maximum HD Temperature: 39

cutouts installed... and temperature starts dropping

Code:

2016-09-26 17:58:57: Maximum HD Temperature: 39
2016-09-26 18:01:58: Maximum HD Temperature: 39
2016-09-26 18:04:59: Maximum HD Temperature: 38
2016-09-26 18:08:00: Maximum HD Temperature: 38
2016-09-26 18:11:02: Maximum HD Temperature: 38
2016-09-26 18:14:02: Maximum HD Temperature: 38
2016-09-26 18:17:04: Maximum HD Temperature: 38
2016-09-26 18:20:04: Maximum HD Temperature: 38
2016-09-26 18:23:06: Maximum HD Temperature: 38
2016-09-26 18:26:07: Maximum HD Temperature: 38
2016-09-26 18:29:07: Maximum HD Temperature: 38
2016-09-26 18:32:09: Maximum HD Temperature: 37
2016-09-26 18:32:09: Drives are warm, going to 75%
2016-09-26 18:35:11: Maximum HD Temperature: 37
2016-09-26 18:35:11: Drives are warm, going to 75%

This is where I realised I have a minor cosmetic bug in the fan controller on the "Drives are warm, going to 75%" case. Ie it repeats itself. While I was fixing that, I bumped the med-high duty to 80%.

About now, the building AC turned off... and it started getting warmer. Decided to leave it warm to see how things go.

Code:

2016-09-26 18:38:11: Maximum HD Temperature: 38
2016-09-26 18:38:11: Drives are too hot, going to 100%
2016-09-26 18:41:12: Maximum HD Temperature: 38
2016-09-26 18:44:14: Maximum HD Temperature: 38
2016-09-26 18:45:18: Maximum HD Temperature: 38
2016-09-26 18:45:18: Drives are too hot, going to 100%
2016-09-26 18:47:09: Maximum HD Temperature: 38
2016-09-26 18:47:09: Drives are too hot, going to 100%
2016-09-26 18:50:09: Maximum HD Temperature: 38
2016-09-26 18:53:10: Maximum HD Temperature: 38
2016-09-26 18:56:11: Maximum HD Temperature: 38
2016-09-26 18:59:12: Maximum HD Temperature: 38
2016-09-26 19:02:14: Maximum HD Temperature: 38
2016-09-26 19:05:15: Maximum HD Temperature: 38
2016-09-26 19:08:16: Maximum HD Temperature: 38
2016-09-26 19:11:17: Maximum HD Temperature: 38
2016-09-26 19:14:19: Maximum HD Temperature: 38
2016-09-26 19:17:19: Maximum HD Temperature: 38
2016-09-26 19:20:20: Maximum HD Temperature: 38
2016-09-26 19:23:21: Maximum HD Temperature: 38
2016-09-26 19:26:22: Maximum HD Temperature: 38
2016-09-26 19:29:23: Maximum HD Temperature: 38
2016-09-26 19:32:24: Maximum HD Temperature: 38
2016-09-26 19:35:25: Maximum HD Temperature: 38
2016-09-26 19:38:26: Maximum HD Temperature: 38
2016-09-26 19:41:28: Maximum HD Temperature: 38

Not cooling enough.

Time to engage course of action #2

There are some extremely large 'holes' in the fan wall for cabling to be routed through, there are also some smaller holes. I figured air-flow was taking the easy path and just circulating from the motherboard side back to the HD side and through the fan wall.

(HD packing foam seems like the perfect draft stopper...)

So I cut some strips of foam and plugged the drafts...

(The HD packing foam nicely seals the drafts around the various cables/wires which transit through the holes in the fan wall.)

While I was there I also sealed some smaller holes in the fan wall

(yes, scotch tape is maybe not so professional... but it is effective... and I was in a hurry. Maybe I'll replace it with actual duct tape oneday... but I doubt it.)

While the lid had been off, the HD temperatures had spiked....

Code:

2016-09-26 19:44:28: Maximum HD Temperature: 39
2016-09-26 19:47:29: Maximum HD Temperature: 39
2016-09-26 19:50:30: Maximum HD Temperature: 39
2016-09-26 19:53:32: Maximum HD Temperature: 39

BUT, as soon as I put the lid back on, the HDs began dropping relatively rapidly.

Code:

2016-09-26 19:56:32: Maximum HD Temperature: 38
2016-09-26 19:59:33: Maximum HD Temperature: 38
2016-09-26 20:02:34: Maximum HD Temperature: 37
2016-09-26 20:02:34: Drives are warm, going to 80%
2016-09-26 20:05:35: Maximum HD Temperature: 37
2016-09-26 20:08:36: Maximum HD Temperature: 37
2016-09-26 20:11:37: Maximum HD Temperature: 37
2016-09-26 20:14:38: Maximum HD Temperature: 37
2016-09-26 20:17:40: Maximum HD Temperature: 37
2016-09-26 20:20:40: Maximum HD Temperature: 37
2016-09-26 20:23:41: Maximum HD Temperature: 37
2016-09-26 20:26:42: Maximum HD Temperature: 37
2016-09-26 20:29:43: Maximum HD Temperature: 37
2016-09-26 20:32:44: Maximum HD Temperature: 37
2016-09-26 20:35:45: Maximum HD Temperature: 37
2016-09-26 20:38:46: Maximum HD Temperature: 37
2016-09-26 20:41:47: Maximum HD Temperature: 37

And once the HDs get to 37C, then the fans die down.... The badblocks is still going, the building A/C is still off.

Code:

# ./show_hd_temps.pl
2016-09-26 20:44:39: /dev/da0: 31
2016-09-26 20:44:39: /dev/da1: 30
2016-09-26 20:44:39: /dev/da2: 32
2016-09-26 20:44:39: /dev/da3: 32
2016-09-26 20:44:39: /dev/da4: 37
2016-09-26 20:44:39: /dev/ada0: 32
2016-09-26 20:44:39: /dev/ada1: 37
2016-09-26 20:44:39: /dev/ada2: 32
2016-09-26 20:44:39: /dev/ada3: 37
2016-09-26 20:44:40: /dev/ada4: 36
2016-09-26 20:44:40: /dev/ada5: 32
2016-09-26 20:44:40: /dev/ada6: 36
2016-09-26 20:44:40: /dev/ada7: 32
2016-09-26 20:44:40: Maximum HD Temperature: 37

I'm happy again.

Be interesting to see how it goes with all the disks working hard... I've started a scrub on the other drives... we'll see.

BTW, I filtered it out, but CPU Temp never went above 40C.

 

[Dice]

Nicely done!
I did some similar, less well documented, but still hardware modifications to my SC847 with great results too.
There is A LOT to be gained from managing air flow and pressure from simply tricks such as scotch tape (I used gorilla tape xD )

 

[Stux]

Nicely done!
I did some similar, less well documented, but still hardware modifications to my SC847 with great results too.
There is A LOT to be gained from managing air flow and pressure from simply tricks such as scotch tape (I used gorilla tape xD )

Apparently, well designed airflow is key :)

The cooling is now more than sufficient with 13 drives. I won't know for sure until I get more drives in there, but there is always the option to spin up the exhaust fans in response to HD temps, and then also upgrade the HD fans.

Be interesting to see how it goes with all the disks working hard... I've started a scrub on the other drives... we'll see.

With a scrub and the badblocks, the drives actually fluctuated between 36 and 37C with fans fluctuating between 50 and 80%. No drive ever exceeded 37C.

I'd call it a success :)

Code:

2016-09-27 11:40:58: Maximum HD Temperature: 37
2016-09-27 11:43:59: Maximum HD Temperature: 36
2016-09-27 11:43:59: Drives are warming, going to 50%
2016-09-27 11:47:00: Maximum HD Temperature: 36
2016-09-27 11:50:01: Maximum HD Temperature: 36
2016-09-27 11:53:02: Maximum HD Temperature: 36
2016-09-27 11:56:03: Maximum HD Temperature: 36
2016-09-27 11:59:04: Maximum HD Temperature: 37
2016-09-27 11:59:04: Drives are warm, going to 80%
2016-09-27 12:02:06: Maximum HD Temperature: 37
2016-09-27 12:05:07: Maximum HD Temperature: 37
2016-09-27 12:08:08: Maximum HD Temperature: 37
2016-09-27 12:11:09: Maximum HD Temperature: 37
2016-09-27 12:14:10: Maximum HD Temperature: 37
2016-09-27 12:17:12: Maximum HD Temperature: 37
2016-09-27 12:20:12: Maximum HD Temperature: 37
2016-09-27 12:23:13: Maximum HD Temperature: 37
2016-09-27 12:26:14: Maximum HD Temperature: 37
2016-09-27 12:29:16: Maximum HD Temperature: 36
2016-09-27 12:29:16: Drives are warming, going to 50%
2016-09-27 12:32:16: Maximum HD Temperature: 36
2016-09-27 12:35:17: Maximum HD Temperature: 36
2016-09-27 12:38:18: Maximum HD Temperature: 36
2016-09-27 12:41:19: Maximum HD Temperature: 36
2016-09-27 12:44:21: Maximum HD Temperature: 37
2016-09-27 12:44:21: Drives are warm, going to 80%

Ideally I'd use a PID algorithm to control the fluctuations, but seriously, its not necessary. As it is, the fan duty only changes every 15-30 minutes under load.

HD temps with badblocks running on 5 drives, other drives have finished their scrub. Fans are back to 50%, barely audible up close.

Code:

# ./show_hd_temps.pl
2016-09-27 13:17:42: /dev/da0: 31
2016-09-27 13:17:42: /dev/da1: 30
2016-09-27 13:17:43: /dev/da2: 32
2016-09-27 13:17:43: /dev/da3: 32
2016-09-27 13:17:43: /dev/da4: 36
2016-09-27 13:17:43: /dev/ada0: 31
2016-09-27 13:17:43: /dev/ada1: 36
2016-09-27 13:17:43: /dev/ada2: 33
2016-09-27 13:17:43: /dev/ada3: 36
2016-09-27 13:17:43: /dev/ada4: 35
2016-09-27 13:17:43: /dev/ada5: 32
2016-09-27 13:17:43: /dev/ada6: 36
2016-09-27 13:17:44: /dev/ada7: 31
2016-09-27 13:17:44: Maximum HD Temperature: 36

Worth pointing out that when I say the "drives" reached 37C, what I actually mean is out of all the drives, the hottest drive, was 37C, many others were much cooler.

I think its fair to say that WDC 3TB NAS drives run hotter than Seagate 4TB NAS drives. Either that, or writing to a drive with badblocks generates a lot more heat than reading with a scrub.

Code:

# camcontrol devlist
<ATA ST4000VN000-1H41 SC46>		at scbus0 target 2 lun 0 (pass0,da0)
<ATA ST4000VN000-1H41 SC46>		at scbus0 target 3 lun 0 (pass1,da1)
<ATA ST4000VN000-1H41 SC46>		at scbus0 target 5 lun 0 (pass2,da2)
<ATA ST4000VN000-1H41 SC46>		at scbus0 target 7 lun 0 (pass3,da3)
<ATA WDC WD30EFRX-68E 0A82>		at scbus0 target 9 lun 0 (pass4,da4)
<ST4000VN000-1H4168 SC46>		  at scbus1 target 0 lun 0 (pass5,ada0)
<WDC WD30EZRX-00DC0B0 80.00A80>	at scbus2 target 0 lun 0 (pass6,ada1)
<ST4000VN000-1H4168 SC46>		  at scbus3 target 0 lun 0 (pass7,ada2)
<WDC WD30EFRX-68EUZN0 80.00A80>	at scbus4 target 0 lun 0 (pass8,ada3)
<WDC WD30EFRX-68EUZN0 80.00A80>	at scbus5 target 0 lun 0 (pass9,ada4)
<ST4000VN000-1H4168 SC46>		  at scbus6 target 0 lun 0 (pass10,ada5)
<WDC WD30EZRX-00DC0B0 80.00A80>	at scbus7 target 0 lun 0 (pass11,ada6)
<ST4000VN000-1H4168 SC46>		  at scbus8 target 0 lun 0 (pass12,ada7)
<SanDisk Ultra Fit 1.00>		   at scbus12 target 0 lun 0 (pass13,da5)
<SanDisk Ultra Fit 1.00>		   at scbus13 target 0 lun 0 (pass14,da6)

 

[Stux]

Code:

839G scanned out of 10.0T at 3.38G/s, 0h46m to go

Damn.

Not to shabby for a single vdev :)

 

[SweetAndLow]

839G scanned out of 10.0T at 3.38G/s, 0h46m to go

Damn.

Not to shabby for a single vdev :)

That number doesn't mean anything. It will change depending on where it's at in the disk.

Sent from my Nexus 5X using Tapatalk

 

[Stux]

That number doesn't mean anything. It will change depending on where it's at in the disk.

Sent from my Nexus 5X using Tapatalk

It stabilized at about 2.8GB/s for the duration of the resilver.

 

[Chris Moore]

By default the fans are installed on the drive side of the fan wall, so that you can get an E-ATX board in. If you install them on the other side, then you get much better access to the SAS ports and the Molex 4-pin peripheral power connectors. BUT then you can't mount the 2.5" drive tray... so out came the dremel again.

What are you going to use that 2.5 drive tray for?? Laptop drives?

 

[Stux]

What are you going to use that 2.5 drive tray for?? Laptop drives?

Boot drives? L2arc? Mirrored ssd pool?

Haven't quite decided.

 

[trumee]

Can you post the model numbers of the fans you used?

 

[Stux]

Can you post the model numbers of the fans you used?

I did, first post

 

[Jason Evans]

Very nice. Something I have been looking at for a future upgrade to my box. Kudos