Modest CPU Utilization to Test Thermal Performance During HDD Stress Tests

[GrumpyBear]

While I'm stress testing my hard disks using bad blocks and jgreco's scripts I would like to also use this opportunity to verify my cooling. The Cougar fans I'm using are only running at their slowest speed while running badblocks and the stress tests as on my SuperMicro X10SL7 the SuperMicro secret sauce for the fan controller for the FAN-A appears to be heavily influenced by the the CPU temperature and as the Xeon E1231v3 CPU is chugging along at 0.01 to 0.4% utilization it is not generating any heat. Meanwhile one of the disks is running at 39C and another hitting 41C occasionally.

I suspect that this low CPU utilization is low for a "Normal" FreeNAS box during this much i/o.

1. Can anyone give me a benchmark for this CPU's utilization under more normal conditions that would generate this much disk activity (say under medium-high FreeNAS load - a scrub?)
2. Anyone have any ideas on how to generate a similar CPU load under FreeNAS so I can see the resulting impact of the FAN-A PWM output and how well the cooling works then? (is "stress" included with FreeNAS and, if so, is texinfo also included so I can see the complete docs?)

 

[Bidule0hm]

If you want to load your CPU you can use this (works on bash, from the top of my head it doesn't work on csh, just type bash in the shell if you use csh):

Code:

[root@freenas ~]# for i in 1 2 3 4; do while : ; do : ; done & done
[1] 85125
[2] 85126
[3] 85127
[4] 85128
[root@freenas ~]#

It's for 4 cores, you can adjust to load less or more cores if you want ;)

In theory on a 4 cores CPU loading 2 cores at 100% should give the same result than loading the 4 cores at 50% so even if this command load the cores at 100% you can adjust the total load by varying the number of loaded cores.

To stop the process (one per core obviously) use kill PID like this:

Code:

[root@freenas ~]# kill 85125
[root@freenas ~]# kill 85126
[root@freenas ~]# kill 85127
[root@freenas ~]# kill 85128

FYI the higher peak CPU load I have is when MiniDLNA rebuilds the DB at startup but it's only during 5 to 6 minutes. The higher long CPU load I have is during a scrub.

 

[GrumpyBear]

... FYI the higher peak CPU load I have is when MiniDLNA rebuilds the DB at startup but it's only during 5 to 6 minutes. The higher long CPU load I have is during a scrub.

What kind of utilization on a scrub for your 4360? (I'd have to adjust for the Xenon)

 

[Bidule0hm]

The MiniDLNA pluging produce a peak of 23-25% for 1 min and then it ramp down to 15-20% during 5 min.

The scrub is pretty constant and loads the CPU to 15-18%.

Now I'm curious to know if the scrub is multi-threaded... :)

 

[GrumpyBear]

If you want to load your CPU you can use this (works on bash, from the top of my head it doesn't work on csh, just type bash in the shell if you use csh):

Code:

[root@freenas ~]# for i in 1 2 3 4; do while : ; do : ; done & done
[1] 85125
[2] 85126
[3] 85127
[4] 85128
[root@freenas ~]#

Thanks, I looked at a bunch of shell scripts but was hoping for something more granular than busying out a thread. Stress appears to be calculating Pi.

So running your do-while I get 50%CPU as expected. With the "Standard" Fan Controller Mode the CPU jumps from 50 to 68C and the Cougar Vortex 120mm System fans move from 500,600 to 800,900rpm and the CPU fan from 1100 to 1400,1500rpm with the core temperature dropping to 64C after 10 min or so. The upper HDD which were running at 39,41,41,39 are now at 35,37,37,34 so about a 4C drop.

Dropping down to 1 thread (12.5% Utilization) the core temp drops down to 54C and the fans revert back to their slow speed.

Using 2 threads (25% utilization) the CPU jumps to 64C then drops down to 59,60C and the CPU fan runs at 1300,1400 and the case fans at 800rpm. The disks run again at 35, 37, 37, 34C.

So with under 25% utilization things run a bit hot. Its not severe but I'm looking forward to see what the Noctua Industrial 140mm fans I have on order will do. They are rated at the same minimum speed (800rpm +/- 20%) as the Cougars but with a higher static pressure and larger size should pump out more air. Also having a 3000rpm maximum speed versus 1500 on the Cougars should mean they increase speed at a faster rate. Judging by some of the reviews I could even use a passive cooler but I might need 4000rpm Deltas for that! Hopefully they will, for the most part, operate in the lower 30 to 50% band of the PWM duty-cycle where they should be fairly quiet.

 

[Ericloewe]

My 120mm IndustrialPPC 3000s idle at a comfortable level, drowned out by the fixed speed 80mm fans on the drive cages, which are audible, but tolerable. About 800RPM, IIRC.

 

[GrumpyBear]

My 120mm IndustrialPPC 3000s idle at a comfortable level, drowned out by the fixed speed 80mm fans on the drive cages, which are audible, but tolerable. About 800RPM, IIRC.

I thought you were the one with the screaming Deltas?? Good to know that the 120mm Noctuas actually operate at their minimum spec'ed speed (well it is supposed to be 750rpm but the IPMI interface appears to round the values compared to those seen with IPMIutil) unlike the Cougars which are rated at 800 min and 1500 max and operate at 500 and 1600 respectively. A delta of 3/8 under the rated value is one heck of a variance. Ive ordered 3 of the 140s and 1 120 of the IPPPC PWMs. Why a 120? Because NewEgg only had 3 of the 140s in stock.

I hope you got the fluorescent green version of the Sharkoon case! Nothing screams enterprise grade NAS more :p (says the guy who got the windowed version of the Define R4)

 

[GrumpyBear]

The MiniDLNA pluging produce a peak of 23-25% for 1 min and then it ramp down to 15-20% during 5 min.

The scrub is pretty constant and loads the CPU to 15-18%.

Now I'm curious to know if the scrub is multi-threaded... :)

Interesting, The Xenon E3-1231v3 has twice the cores and cache. Though it has a lower native clock speed at 3.4GHz the Turbo at 3.8GHz is 100MHz faster so a rough estimate might be that the Xenon would be twice as fast. The CPU PassMark Scores (9634 and 3728) would seem to support this. Of interest is that the CPUmark price/performance figures are almost identical so both these CPUs have very good bang for the buck.

So an estimate of CPU stress under a typical high load would be 7.5-9% so I would be operating at really low fan speeds most of the time (The Xenon likely has twice the TDP of the i3 if one were to remove the integrated graphics).

 

[Bidule0hm]

Yep, that's the theory but I think the load may depends on the RAM size, the pool config, the moon, ... :D

I'm really curious to see if the test will match the prediction ;)

 

[Ericloewe]

I thought you were the one with the screaming Deltas?? Good to know that the 120mm Noctuas actually operate at their minimum spec'ed speed (well it is supposed to be 750rpm but the IPMI interface appears to round the values compared to those seen with IPMIutil) unlike the Cougars which are rated at 800 min and 1500 max and operate at 500 and 1600 respectively. A delta of 3/8 under the rated value is one heck of a variance. Ive ordered 3 of the 140s and 1 120 of the IPPPC PWMs. Why a 120? Because NewEgg only had 3 of the 140s in stock.

No, but I did look at Deltas for the hot swap bays, but they haven't been needed (yet).

I hope you got the fluorescent green version of the Sharkoon case! Nothing screams enterprise grade NAS more :p (says the guy who got the windowed version of the Define R4)

Nope, black for me. :p The neon green effect would be a bit lost without the stock fans, though, so I'd probably have gone with one of the crazy colors if they'd been significantly cheaper than black.

I get the enterprise NAS look by adding a bunch of useless little LEDs - some know them as "hot swap bay drive activity/presence LEDs". It's almost like looking at a busy network switch. :D

 

[Bidule0hm]

You find your temps a bit high? look at the Delta TFC1212DE :) Yes, it sucks roughly 50 W and is very quiet with only 66 dB of noise...

 

[GrumpyBear]

... very quiet with only 66 dB of noise...

LOL - We have different definitions of "Quiet". Yours seems to be quiet like a heavy metal concert!

Just installed the Noctuas. Running at 600 and 700rpm. again no rhyme or reason. One at the front at 600 and one at the back.

I'm running dd reads and writes to generate some heat then I'll look at the Disk temperatures after a while and add some CPU load to see the slope. I'm testing "Optimal" now and I'll do "Standard" on the next test.

After 1:10 with the disks highly loaded I'm seeing:

da0: 29 Celsius
da1: 34 Celsius
da2: 32 Celsius
da3: 31 Celsius
da4: 32 Celsius
da5: 34 Celsius
da6: 33 Celsius
da7: 31 Celsius

Where on previous test:

da0: 35 Celsius
da1: 39 Celsius
da2: 37 Celsius
da3: 35 Celsius
da4: 39 Celsius
da5: 41 Celsius
da6: 41 Celsius
da7: 39 Celsius

So a 5-6 degree drop at the bottom of the stack of disks and 7-8 degrees at the top. This is with the Noctuas running at their lowest (30% PWM) speed. So I would tentatively say that the larger size combined with the 200rpm increase and speed and likely higher static pressure makes these fans look like a winner. Lets see how loud they get as the CPU heats up.

 

[Bidule0hm]

It was ironic of course :D

Be careful, HDDs under 30 C is also bad for the lifetime of the drive (see the Google paper for more details) ;)

 

[GrumpyBear]

Be careful, HDDs under 30 C is also bad for the lifetime of the drive (see the Google paper for more details) ;)

I have studied the Google Paper quite a bit. And yes, it seems the Average failure rate doubles from 5% for a drive operating at 40C to 10% for a drive operating at 30C but of more concern is the long term rates where they look at average temperature by age and after the 3-month "bathtub curve" failures where they observe this doubling again the curve flattens out and at the three year point reverses where there is a marked spike in failures for drives over 40C (12-15% versus 5-7%).

The sweet spot appears to be between 35 and 40C but as someone once said: "Lies, dammed lies, and statistics"

 

[Bidule0hm]

Yes, exactly, target temp: 35 C +/- 5 C :) It's not as bad as going over 40 C but I think the AFR is significantly higher so it's worth considering ;) In fact I just thought about something: if the AFR were higher not only because the average temp of the drive but also because of the variations?

I mean it's well known that thermal variations are bad for mechanical systems and materials, and maybe the drives in the lower and higher sections of the temp stats are subjected to higher temp variations than the drives in the average temp section and it's why they are in these sections.

 

[GrumpyBear]

Well I have some data. It's probably worth starting another thread for but the principle observations so far are:

With these Noctua Fans the best Fan mode appears to be "Optimal". In this mode with the CPU connected to FAN-1 and the Case fans connected to FAN-2 through FAN-4 and being driven (PWM) from FAN-A the CPU fan increases as load (core temp) increases and the case fans remain at the 30% duty cycle (about 600-700rpm) which is sufficient to keep the disks between 28-32C.

In "Standard" mode the CPU temp has a greater influence on the case fans and the case fans gradually ramp up from 500-600rpm at 0% utilization to 1500-1700 rpm at 87.5% cpu utilization with the hard disk temperatures dropping slowly to 23-28C. Both tests were run when i was running staggered 5 dd writes to one set of 4 disks and one set of staggered dd reads.

In "Heavy I/O" mode at 87.5% CPU utilization the case fans run at 2200-2400rpm and the drive temperatures drop down to 22-27C after 5 min.

Time to get some sleep. I'll analyze the data some more tomorrow night and put some graphs together. And then try another run with the Cougar fans. With their lower speed and static pressure they may be able to run in Standard mode and let the drives run a little hotter (which seems strange to say as electronics usually like being dipped in Liquid Nitrogen).

 

[Bidule0hm]

In standard mode FAN-A PWM = FAN-1 to 4 PWM so yeah, optimal mode is, well, more optimal :D

"as electronics usually like being dipped in Liquid Nitrogen" actually in the realm of breaking overclocking world records extreme cooling systems are used (-50 to -200 C) and some CPU coldbug at only -80/-90 C. I don't know why some CPUs coldbug at - x0 C when others are perfectly happy even at -150 C but it's interesting :)