Thermal and Accoustical Design Validation

Status
Not open for further replies.

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
While running extensive tests to validate my disks in preparation for a FreeNAS build using these guides:
as suggested by jgreco in the above thread I used this opportunity of having the disks at heavy load to validate my thermal and acoustical design.

Recommendations (TL;DR)
If you are using a SuperMicro X10-series motherboard in a pedestal (tower) case and desire adequate cooling for the hard drives with a minimum of noise:
  • Use the "Optimal" fan mode.
  • Use 120mm or 140mm PWM (4-pin) case fans connected to the FAN-A header.
  • Connect the CPU cooler to one of the FAN-1 through FAN-4 headers.
  • Use IPMItool to set the upper and lower thresholds for the fans.
  • Verify that the disks are adequately cooled under heavy load.
Background
SuperMicro Motherboards are server grade motherboards and as such are designed for installation in rack-mounted cases in Data Centers. Adapting these boards for use in pedestal (tower) cases and with a goal of reducing system noise requires testing to ensure that the proposed thermal solution provides adequate airflow to keep the hard drives cool under high loads.
Electrical components tend to be more efficient at lower temperatures. Hard disks are electro-mechanical systems and the effects of high and low temperatures on their operation and predicted failure rate is not well understood. A widely quoted paper written in 2007 by a research team from Google "Failure Trends in a Large Disk Drive Population" which studied the failure trends in over 100,000 consumer-grade 5400-7200rpm drives deployed in their company suggesting that operating hard disks over 40 degrees Celsius may cause them to fail sooner than drives operating at lower temperatures.
Fans tend to be the largest source of noise in computer systems. In a data center where space is at a premium and high quality environmental controls are in place noise is acceptable and most rack-mounted systems have processors with passive heatsinks. In order to adequately cool these processors shrouds and baffles are utilized in chassis thermal designs with small, high rpm fans to move large volumes of air. Typically fan noise is directly proportional to rotational speed and the smaller a fan is the higher speed at which it must operate to move the same volume of air.
Computer fans typically operate at 12V DC. As they utilize DC motors the speed of the fans can be adjusted to a limited extend by lowering or raising the voltage (voltage control).
Most computer fans these days have a third connection (3-pin) that outputs a tachometer (TAC) signal (2 pulses-per-revolution). This TAC signal allows a controler on the motherboard to monitor the fan speed and detect a failed fan.
Higher end computer fans support varying the fan speed on the motor itself. These fans have a fourth lead (4-pin), in addition to the TAC signal, that accepts a Pulse-Width Modulated (PWM) signal from a controler on the motherboard to control the fan speed. The duty-cycle (percentage of on to off) of the PWM lead determines the speed at which the fan operates. At 0% duty cycle the fan is not running. At 100% duty-cycle the fan is running at its highest speed.
Like most rotating devices fans have inertia when stopped and require a certain amount of force to overcome this when started. To overcome inertia most fans are started by sending a PWM signal which starts at a higher duty-cycle and then is dropped down to the desired duty cycle. For this reason duty cycles lower than 30% are undefined.

Motherboard Capabilities
The SuperMicro X10SL7-F motherboard has 5 4-pin fan headers labelled FAN-1 through -4 and FAN-A. Each of the headers is rated at 1.5A (18W) with a combined rating not to exceed 3.0A
The BIOS and IPMI web interface has 4 possible fan modes; Standard, Full Speed, Optimal or PUE2 and, Heavy I/O. There is sparse information on the intended use of the fan headers and which fan mode should be used.
The motherboard monitors CPU, PCH, VRM and Memory temperatures using hardware built-in to the dies. There is additionally 2 thermistors monitored on the motherboard.
index.php

"RT1" is located between the SAS ports and the LSI2308 and is the "System" sensor
index.php

RT2 is located at between the ASPEED BMC and the I/O slots and is the "Peripheral" or "Aux" sensor.

Hardware Choices and Design Goals
I chose to use the Fractal Designs Define R4 case. This case was designed for the enthusiast gaming market segment and has a large volume and provision for up to six 120 or 140mm case fans. The case includes 2 140mm Silent Series R2 3-pin fans as well as a "fan controller" with a switch capable of operating 3 fans at 5, 7 or 12V (likely a resistive divider).
I replaced the 3-pin case fans with 4 Cougar Gaming 120mm Vortex PWM (4-pin) fans. The intent was to use large PWM controlled fans running at slow enough speeds to minimize noise and provide adequate cooling under load.
Extensive research on-line and on the Frequently Asked Question (FAQ) section of SuperMicro's support site led me to the initial supposition that the FAN-1 through FAN-4 headers had their PWM output controlled by CPU temperature and the FAN-A header had it's PWM output possibly controlled by another temperature sensor in fan modes other than Standard.
In order to drive the 4 case fans from the FAN-A header a 5-way 4-pin fan Splitter was used. The splitter was modified to extend the TAC leads from the additional fans back to the unused FAN-2 -4 headers:
index.php
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Part 2.

Methodology
Set fan thresholds
Select a FAN mode
Load up Disks using dd
Let System Come up to temperature
Record system temperatures
Record Disk Temperatures
Record Fan Speeds
Loop until utilization ~ 100%
Increase CPU Utilization
wait 5 minutes
record temperatures and fan speeds​
repeat​
As well explained by Eric Lowe here setting the fan thresholds using IPMItool can be accomplished either remotely or locally on the system. The 120mm Cougar Vortex PWM Fans I used to replace the case fans with are rated at 800 to 1500rpm. Initial tests however revealed the actual installed range was 500-1600rpm so the following commands were issued:
Code:
ipmitool sensor thresh "FANA" upper 1700 1900 2100
ipmitool sensor thresh "FANA" lower 200 300 400
ipmitool sensor thresh "FAN2" upper 1700 1900 2100
ipmitool sensor thresh "FAN2" lower 200 300 400
ipmitool sensor thresh "FAN3" upper 1700 1900 2100
ipmitool sensor thresh "FAN3" lower 200 300 400
ipmitool sensor thresh "FAN4" upper 1700 1900 2100
ipmitool sensor thresh "FAN4" lower 200 300 400
Note that I did NOT have to load the kernel module as stated in the HOWTO.
I also noted that the FAN1 upper thresholds were abnormally high and after initial tests were set to:
Code:
ipmitool sensor thresh FAN1 upper 2500 2600 2700
The fan mode, as previously mentioned, can be set to "Standard", "Full Speed", "Optimal" or "Heavy I/O". Some of these can be set in the BIOS and all via IPMI via the GUI available from the BMC. Page 2-50 of the SMT IPMI User's Guide has perhaps the clearest explanation of these modes:
• Standard speed, for standard power efficiency
• Full speed, for maximum system performance
• Optimal speed or PUE2 speed, for most efficient cooling under normal use
• Heavy IO speed, for running the system at maximum I/O (maximizing cooling around PCI-E add-on cards)
Note: Heavy IO mode is used in X10 motherboards with dual cooling zones. PUE Optimal mode is used in certain X10 motherboards with single cooling zones to consume the least amount of power possible.​
IPMI can also be accessed by a number of third-party tools. I use IPMItool as it is included in the latest 9.3 stable releases and works from the shell and does not require Java. The fan mode setting is a OEM extension to the IPMI specification but is not exposed via the SuperMicro IPMI extensions contained in IPMItool. However further investigation found this post on SuperMicro's website detailing the "raw" IPMI commands to set and query the fan mode. A little experimentation found the following commands:
Code:
ipmitool raw 0x30 0x45 0x1 0x0 (Set Mode to Standard)
ipmitool raw 0x30 0x45 0x1 0x1 (Set Mode to Full)
ipmitool raw 0x30 0x45 0x1 0x2 (Set Mode to Optimal)
ipmitool raw 0x30 0x45 0x1 0x4 (Set Mode to Heavy I/O)

ipmitool raw 0x30 0x45 0x0 (Get Fan Mode)

Disk Stress Tests are well documented here in the threads mentioned above. For these tests I simply used the following non-destructive script loosely based on jgreco's thread above.
Code:
#! /bin/sh -

echo "Reading from All Disks"
for i in 1 2 3 4 5; do
  sleep 60
  for disk in 0 1 2 3 4 5 6 7; do
  dd if=/dev/da${disk} of=/dev/null bs=1048576 &
  done
done
replacing the number of disks and type (i.e. "0 1 2 3 4 5 6 7" and "da") as appropriate.
Disk stress tests by their very nature tend to force the hard drives to produce the maximum amount heat we would typically see in production. It was noted during initial testing that the disks took about 30 minutes to reach thermal equilibrium. The disk temperatures can be read from the SMART data. I used a simple script:
Code:
#! /bin/sh -

  for disk in 0 1 2 3 4 5 6 7; do
  echo -n "da"$disk " "
  smartctl -l scttempsts /dev/da${disk} | grep 'Current Temperature'
  done
While this is useful for observing the maximum disk temperatures these same disk tests typically produce minimal processor utilization. The heat produced by a processor is usually proportional to its utilization. As previously mentioned it appears that the fan modes on the SuperMicro X10SL7-F motherboard are controlled by the processor temperature.
A method is therefore required to vary the CPU utilization, and hence its temperature, in order to observe the changes in fan speed and temperature of the disks. As suggested by user bidule0hm here starting an infinite loop from a shell will typically create 100% utilization on a single thread. The Xeon E31231-V3 processor has 4 cores and 8 threads running at 3.4/3.8GHZ. A simple script to busy a single core is:
Code:
#! /bin/bash
while : ; do : ; done &
echo $!
Note that the processes that "busy out" a core should be stopped after the tests are run. This can be done by observing which Process IDs (PIDs) are consuming 100% utilization on a thread and issuing the "kill <PID>" command.
The temperatures for the system components and the fan speeds can be obtained via IPMI using:
Code:
ipmitool -c sdr type Temperature
ipmitool -c sdr type FAN
During initial testing it was determined that more than 85% of any final change in value for disk and system temperatures was reached within 5 minutes so the interval for changing utilization set to 300s.
Putting it all together we get:
Code:
#! /bin/bash -
#
# Run this while loading disks with reads or writes to also load the CPU so the
# behavior of the fan control can be verified to keep the disks at a desired
# temperature.

## Load the disks with reads
#echo "Reading from All Disks"
## Start 5 disk reads spaced 60s apart
#  for i in 1 2 3 4 5; do
#  sleep 60
##  for each disk
#  for disk in 0 1 2 3 4 5 6 7; do
#  dd if=/dev/da${disk} of=/dev/null bs=1048576 &
#  done
#  done
## Wait 30 min for temps to settle
#  echo "Waiting 30 minutes for disks to warm up"
#  sleep 1800

# Print Initial Values
  date
  ipmitool -c sdr type Temperature
  ipmitool -c sdr type FAN
  for disk in 0 1 2 3 4 5 6 7; do
    echo -n "da"$disk " "
    smartctl -l scttempsts /dev/da${disk} | grep 'Current Temperature'
  done
# Add a infinite loop for each core every 5 minutes to increase CPU Utilization
  for Cores in 1 2 3 4 5 6 7; do
    while : ; do : ; done &
      sleep 300
#  After 5 minutes display temperatures and fan speeds
     date
     ipmitool -c sdr type Temperature
     ipmitool -c sdr type FAN
     for disk in 0 1 2 3 4 5 6 7; do
        echo -n "da"$disk " "
        smartctl -l scttempsts /dev/da${disk} | grep 'Current Temperature'
     done
  done
 
Last edited:

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Part 3.

Results
The CPU fan was connected to the FAN-1 header and the 4 case fans were initially connected to the FAN-2 through -4 and FAN-A headers.
The fan mode was changed from "Standard" to "Full Speed" and the following speeds were observed:
  • FAN-1 - 2000rpm
  • FAN-2 - 1600rpm
  • FAN-3 - 1600rpm
  • FAN-4 - 1500rpm
  • FAN-A - 1500rpm
So as expected all fans run at full speed (100% PWM Duty Cycle)

The fan mode was then changed to "Heavy I/O" and the following speeds were observed:
  • FAN-1 - 1000rpm
  • FAN-2 - 600rpm
  • FAN-3 - 600rpm
  • FAN-4 - 500rpm
  • FAN-A - 1400rpm
So it appears that in Heavy I/O mode FAN-A is controlled separately from FAN-1 through FAN-4 and runs at about 80-90% PWM duty-cycle.

The fan mode was changed to "Optimal" and the disks were loaded using "dd" reads and were allowed to sit for 30min. to reach thermal equilibrium. The initial reading at 0% utilization were
Code:
Wed Mar 11 19:58:48 EDT 2015
CPU Temp,46,degrees C,ok
System Temp,47,degrees C,ok
Peripheral Temp,42,degrees C,ok
PCH Temp,46,degrees C,ok
VRM Temp,44,degrees C,ok
DIMMA1 Temp,35,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1000,RPM,ok
FAN2,600,RPM,ok
FAN3,600,RPM,ok
FAN4,500,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  37 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  34 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  38 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  36 Celsius

and the reading taken 10 minutes apart were:
Code:
Wed Mar 11 20:08:50 EDT 2015
CPU Temp,54,degrees C,ok
System Temp,47,degrees C,ok
Peripheral Temp,42,degrees C,ok
PCH Temp,46,degrees C,ok
VRM Temp,45,degrees C,ok
DIMMA1 Temp,35,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1200,RPM,ok
FAN2,600,RPM,ok
FAN3,600,RPM,ok
FAN4,600,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  37 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  34 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  38 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  36 Celsius
Wed Mar 11 20:18:53 EDT 2015
CPU Temp,58,degrees C,ok
System Temp,46,degrees C,ok
Peripheral Temp,42,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,47,degrees C,ok
DIMMA1 Temp,36,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,34,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1300,RPM,ok
FAN2,800,RPM,ok
FAN3,800,RPM,ok
FAN4,800,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  37 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  34 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  38 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  36 Celsius
Wed Mar 11 20:28:55 EDT 2015
CPU Temp,63,degrees C,ok
System Temp,46,degrees C,ok
Peripheral Temp,41,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,48,degrees C,ok
DIMMA1 Temp,37,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,34,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1400,RPM,ok
FAN2,900,RPM,ok
FAN3,900,RPM,ok
FAN4,800,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  37 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  33 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  37 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  35 Celsius
Wed Mar 11 20:38:58 EDT 2015
CPU Temp,64,degrees C,ok
System Temp,44,degrees C,ok
Peripheral Temp,41,degrees C,ok
PCH Temp,46,degrees C,ok
VRM Temp,48,degrees C,ok
DIMMA1 Temp,36,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1500,RPM,ok
FAN2,1000,RPM,ok
FAN3,1000,RPM,ok
FAN4,900,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  36 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  32 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  37 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  34 Celsius
Wed Mar 11 20:49:01 EDT 2015
CPU Temp,66,degrees C,ok
System Temp,44,degrees C,ok
Peripheral Temp,41,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,49,degrees C,ok
DIMMA1 Temp,37,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1500,RPM,ok
FAN2,1000,RPM,ok
FAN3,1000,RPM,ok
FAN4,900,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  35 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  32 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  36 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  33 Celsius
Wed Mar 11 20:59:03 EDT 2015
CPU Temp,68,degrees C,ok
System Temp,45,degrees C,ok
Peripheral Temp,41,degrees C,ok
PCH Temp,48,degrees C,ok
VRM Temp,50,degrees C,ok
DIMMA1 Temp,37,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1500,RPM,ok
FAN2,1000,RPM,ok
FAN3,1000,RPM,ok
FAN4,900,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  35 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  32 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  36 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  33 Celsius
Wed Mar 11 21:09:06 EDT 2015
CPU Temp,69,degrees C,ok
System Temp,43,degrees C,ok
Peripheral Temp,41,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,51,degrees C,ok
DIMMA1 Temp,37,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,32,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1600,RPM,ok
FAN2,1100,RPM,ok
FAN3,1100,RPM,ok
FAN4,1000,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  34 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  32 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  35 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  32 Celsius

So we see that in "Optimal" mode FAN-1 through -4 increase as the CPU temperature increases under load and FAN-A remains at a low speed.

The fan mode was next set to "Standard" and the following readings were obtained:
Code:
Wed Mar 11 21:39:41 EDT 2015
CPU Temp,45,degrees C,ok
System Temp,47,degrees C,ok
Peripheral Temp,42,degrees C,ok
PCH Temp,46,degrees C,ok
VRM Temp,43,degrees C,ok
DIMMA1 Temp,34,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1000,RPM,ok
FAN2,600,RPM,ok
FAN3,600,RPM,ok
FAN4,500,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  36 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  34 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  37 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  35 Celsius
Wed Mar 11 21:44:44 EDT 2015
CPU Temp,56,degrees C,ok
System Temp,48,degrees C,ok
Peripheral Temp,42,degrees C,ok
PCH Temp,48,degrees C,ok
VRM Temp,46,degrees C,ok
DIMMA1 Temp,36,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1000,RPM,ok
FAN2,600,RPM,ok
FAN3,600,RPM,ok
FAN4,500,RPM,ok
FANA,600,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  36 Celsius
da2  Current Temperature:  36 Celsius
da3  Current Temperature:  34 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  38 Celsius
da6  Current Temperature:  38 Celsius
da7  Current Temperature:  36 Celsius
Wed Mar 11 21:49:46 EDT 2015
CPU Temp,60,degrees C,ok
System Temp,47,degrees C,ok
Peripheral Temp,41,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,47,degrees C,ok
DIMMA1 Temp,36,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,34,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1300,RPM,ok
FAN2,800,RPM,ok
FAN3,800,RPM,ok
FAN4,700,RPM,ok
FANA,700,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  36 Celsius
da2  Current Temperature:  35 Celsius
da3  Current Temperature:  34 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  38 Celsius
da6  Current Temperature:  36 Celsius
da7  Current Temperature:  36 Celsius
Wed Mar 11 21:54:49 EDT 2015
CPU Temp,63,degrees C,ok
System Temp,46,degrees C,ok
Peripheral Temp,41,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,48,degrees C,ok
DIMMA1 Temp,37,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,34,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1400,RPM,ok
FAN2,900,RPM,ok
FAN3,900,RPM,ok
FAN4,800,RPM,ok
FANA,900,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  36 Celsius
da2  Current Temperature:  35 Celsius
da3  Current Temperature:  34 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  37 Celsius
da6  Current Temperature:  36 Celsius
da7  Current Temperature:  35 Celsius
Wed Mar 11 21:59:51 EDT 2015
CPU Temp,65,degrees C,ok
System Temp,45,degrees C,ok
Peripheral Temp,40,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,49,degrees C,ok
DIMMA1 Temp,36,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1400,RPM,ok
FAN2,1000,RPM,ok
FAN3,1000,RPM,ok
FAN4,900,RPM,ok
FANA,900,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  36 Celsius
da2  Current Temperature:  35 Celsius
da3  Current Temperature:  33 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  37 Celsius
da6  Current Temperature:  36 Celsius
da7  Current Temperature:  34 Celsius
Wed Mar 11 22:04:54 EDT 2015
CPU Temp,66,degrees C,ok
System Temp,45,degrees C,ok
Peripheral Temp,39,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,50,degrees C,ok
DIMMA1 Temp,37,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1500,RPM,ok
FAN2,1000,RPM,ok
FAN3,1000,RPM,ok
FAN4,900,RPM,ok
FANA,1000,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  35 Celsius
da2  Current Temperature:  35 Celsius
da3  Current Temperature:  32 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  37 Celsius
da6  Current Temperature:  36 Celsius
da7  Current Temperature:  33 Celsius
Wed Mar 11 22:09:57 EDT 2015
CPU Temp,68,degrees C,ok
System Temp,44,degrees C,ok
Peripheral Temp,39,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,51,degrees C,ok
DIMMA1 Temp,37,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,33,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1500,RPM,ok
FAN2,1000,RPM,ok
FAN3,1000,RPM,ok
FAN4,1000,RPM,ok
FANA,1000,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  35 Celsius
da2  Current Temperature:  35 Celsius
da3  Current Temperature:  32 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  36 Celsius
da6  Current Temperature:  36 Celsius
da7  Current Temperature:  33 Celsius
Wed Mar 11 22:14:59 EDT 2015
CPU Temp,70,degrees C,ok
System Temp,44,degrees C,ok
Peripheral Temp,39,degrees C,ok
PCH Temp,47,degrees C,ok
VRM Temp,52,degrees C,ok
DIMMA1 Temp,37,degrees C,ok
DIMMA2 Temp,,,ns
DIMMB1 Temp,32,degrees C,ok
DIMMB2 Temp,,,ns
FAN1,1600,RPM,ok
FAN2,1100,RPM,ok
FAN3,1100,RPM,ok
FAN4,1000,RPM,ok
FANA,1100,RPM,ok
da0  Current Temperature:  32 Celsius
da1  Current Temperature:  34 Celsius
da2  Current Temperature:  35 Celsius
da3  Current Temperature:  32 Celsius
da4  Current Temperature:  36 Celsius
da5  Current Temperature:  35 Celsius
da6  Current Temperature:  36 Celsius
da7  Current Temperature:  32 Celsius

In Standard mode all the fans including FAN-A now are controlled by the CPU temperature.

Next the 5-way PWM Splitter was installed and all case fans were connected through it to the FAN-A header and the fan mode again set to "Optimal" with the fans connected thus:

FAN-1 - CPU
FAN-2 - Front Upper Fan
FAN-3 - Front Lower Fan
FAN-4 - Rear Top Fan
FAN-A - Rear Back Fan​

The tests were again run and the following readings obtained:
Code:
Utilization: 0%

Temperatures:
   CPU Temp   45 degrees C
   System Temp   48 degrees C
   Peripheral Temp   42 degrees C
   PCH Temp   47 degrees C
   VRM Temp   44 degrees C
   DIMMA1 Temp   35 degrees C
   DIMMB1 Temp   33 degrees C

Fan Speeds:
   FAN1   1000 R.P.M
   FAN2    600 R.P.M
   FAN3    600 R.P.M
   FAN4    500 R.P.M
   FANA    600 R.P.M

Disk Temps:
   da0   33 Celsius
   da1   39 Celsius
   da2   36 Celsius
   da3   35 Celsius
   da4   37 Celsius
   da5   39 Celsius
   da6   38 Celsius
   da7   37 Celsius

Utilization: 12.5%

Temperatures:
   CPU Temp   55 degrees C
   System Temp   48 degrees C
   Peripheral Temp   42 degrees C
   PCH Temp   48 degrees C
   VRM Temp   45 degrees C
   DIMMA1 Temp   36 degrees C
   DIMMB1 Temp   33 degrees C

Fan Speeds:
   FAN1   1200 R.P.M
   FAN2    600 R.P.M
   FAN3    600 R.P.M
   FAN4    500 R.P.M
   FANA    600 R.P.M

Disk Temps:
   da0   33 Celsius
   da1   38 Celsius
   da2   36 Celsius
   da3   35 Celsius
   da4   37 Celsius
   da5   39 Celsius
   da6   38 Celsius
   da7   37 Celsius

Utilization: 25%

Temperatures:
   CPU Temp   59 degrees C
   System Temp   46 degrees C
   Peripheral Temp   43 degrees C
   PCH Temp   48 degrees C
   VRM Temp   47 degrees C
   DIMMA1 Temp   36 degrees C
   DIMMB1 Temp   34 degrees C

Fan Speeds:
   FAN1   1300 R.P.M
   FAN2    600 R.P.M
   FAN3    600 R.P.M
   FAN4    500 R.P.M
   FANA    600 R.P.M

Disk Temps:
   da0   33 Celsius
   da1   38 Celsius
   da2   36 Celsius
   da3   35 Celsius
   da4   37 Celsius
   da5   39 Celsius
   da6   38 Celsius
   da7   37 Celsius

Utilization: 37.5%

Temperatures:
   CPU Temp   62 degrees C
   System Temp   46 degrees C
   Peripheral Temp   42 degrees C
   PCH Temp   48 degrees C
   VRM Temp   48 degrees C
   DIMMA1 Temp   37 degrees C
   DIMMB1 Temp   34 degrees C

Fan Speeds:
   FAN1   1400 R.P.M
   FAN2    600 R.P.M
   FAN3    600 R.P.M
   FAN4    500 R.P.M
   FANA    600 R.P.M

Disk Temps:
   da0   33 Celsius
   da1   38 Celsius
   da2   36 Celsius
   da3   35 Celsius
   da4   37 Celsius
   da5   39 Celsius
   da6   38 Celsius
   da7   37 Celsius

Utilization: 50%

Temperatures:
   CPU Temp   65 degrees C
   System Temp   46 degrees C
   Peripheral Temp   42 degrees C
   PCH Temp   48 degrees C
   VRM Temp   49 degrees C
   DIMMA1 Temp   37 degrees C
   DIMMB1 Temp   34 degrees C

Fan Speeds:
   FAN1   1500 R.P.M
   FAN2    600 R.P.M
   FAN3    600 R.P.M
   FAN4    500 R.P.M
   FANA    600 R.P.M

Disk Temps:
   da0   33 Celsius
   da1   38 Celsius
   da2   36 Celsius
   da3   35 Celsius
   da4   37 Celsius
   da5   39 Celsius
   da6   38 Celsius
   da7   37 Celsius

Utilization: 62.5%

Temperatures:
   CPU Temp   67 degrees C
   System Temp   46 degrees C
   Peripheral Temp   43 degrees C
   PCH Temp   48 degrees C
   VRM Temp   50 degrees C
   DIMMA1 Temp   38 degrees C
   DIMMB1 Temp   35 degrees C

Fan Speeds:
   FAN1   1600 R.P.M
   FAN2    600 R.P.M
   FAN3    600 R.P.M
   FAN4    500 R.P.M
   FANA    600 R.P.M

Disk Temps:
   da0   33 Celsius
   da1   38 Celsius
   da2   36 Celsius
   da3   35 Celsius
   da4   37 Celsius
   da5   39 Celsius
   da6   38 Celsius
   da7   37 Celsius

Utilization: 75%

Temperatures:
   CPU Temp   67 degrees C
   System Temp   46 degrees C
   Peripheral Temp   43 degrees C
   PCH Temp   48 degrees C
   VRM Temp   51 degrees C
   DIMMA1 Temp   38 degrees C
   DIMMB1 Temp   35 degrees C

Fan Speeds:
   FAN1   1600 R.P.M
   FAN2    600 R.P.M
   FAN3    600 R.P.M
   FAN4    500 R.P.M
   FANA    600 R.P.M

Disk Temps:
   da0   33 Celsius
   da1   38 Celsius
   da2   36 Celsius
   da3   35 Celsius
   da4   37 Celsius
   da5   40 Celsius
   da6   38 Celsius
   da7   37 Celsius

Utilization: 87.5% (133W)
** Note: heard humming from system - appears Power Supply fan turned on)

Temperatures:
   CPU Temp   69 degrees C
   System Temp   46 degrees C
   Peripheral Temp   42 degrees C
   PCH Temp   48 degrees C
   VRM Temp   51 degrees C
   DIMMA1 Temp   38 degrees C
   DIMMB1 Temp   35 degrees C

Fan Speeds:
   FAN1   1600 R.P.M
   FAN2    600 R.P.M
   FAN3    600 R.P.M
   FAN4    500 R.P.M
   FANA    600 R.P.M

Disk Temps:
   da0   33 Celsius
   da1   38 Celsius
   da2   36 Celsius
   da3   35 Celsius
   da4   37 Celsius
   da5   40 Celsius
   da6   38 Celsius
   da7   37 Celsius
So we can see from this test that the CPU fan speed increases as expected with the CPU temperature but the case fans remain at their lowest speed but keep the disks at or just below 40C with the disks under heavy load.

An additional set of tests were run with three 140mm and one 120mm Noctua Industrial PPC 3000rpm fans replacing the Cougar fans. These fans are about twice the price of the Cougar fans and move considerably more air. Under "Optimal" mode these fans kept the disks at or under 33C at 87.5% utilization.

Power Consumption
A Kill-a-Watt electrical power meter was connected between the UPS and the computer and reading were taken:

Parasitic Load: 10-11W (IPMI on, Computer off)
Peak Startup Load: 75W (No Disks Connected)
Peak Startup Load: 158W (8 Disks Connected)
Idle Load: 54W (No Disks, Fan Mode Optimal)
Idle Load: 68W (0% CPU, 0% Disk, Fan Mode Optimal)
Peak Load: 135W (87.5% CPU, Disk Stress Test, Fan Mode Optimal)

Miscellaneous
The Power Supply in use (Corsair HX650) is vastly overpowered and is thus running at a low point in it's efficiency curve. It is likely that a 350W power supply would be more than adequate.

It was also noted during the stress tests that while the Western Digital and Seagate disks performed similarly in the single threaded mode for reads and writes with the Seagate disks performing faster as expected due to their slight speed advantages in the parallel read tests they were observed to perform at roughly 20% of the performance of the Western Digital disks. This behavior was not observed during the single and parallel write stress tests nor were any such significant variations observed during tests using badblocks and SMART long tests. The next phase of testing will be using iozone on a ZFS filesystem so it will be interesting to see if a significant deviation occurs between the disks again. An issue with mixing disks from manufacturers may be that in an effort to be less susceptible to manufacture-specific fault issues you can end up with performance that is the worst of both worlds.

Conclusions
The initial design using 4 120mm Cougar fans connected to a 5-way 4-pin fan splitter connected to the FAN-A header works well using a fan mode of "Optimal". The Noctua fans while impressive in performance at low speeds are significantly louder as expected at higher speeds and do not offer a significant advantage.
Keeping the disks "around" 40C under heavy load was the goal I had set and this has been verified.
 
Last edited:

Fraoch

Patron
Joined
Aug 14, 2014
Messages
395
Very impressive writeup, thank you!
 

dj_jazzn

Dabbler
Joined
Apr 14, 2014
Messages
31
I have been following your post and working on a very similar build. BTW, thanks for all the documentation.

I have wired my fans according the diagram you provided. All is good: fans are all spinning and RPM values are being reported. However, I am a bit confused on one point, the lower fan threshold values.

I followed the commands you outline in your post:

Code:
ipmitool sensor thresh "FANA" upper 1700 1900 2100
ipmitool sensor thresh "FANA" lower 400 300 200
ipmitool sensor thresh "FAN2" upper 1700 1900 2100
ipmitool sensor thresh "FAN2" lower 400 300 200
ipmitool sensor thresh "FAN3" upper 1700 1900 2100
ipmitool sensor thresh "FAN3" lower 400 300 200
ipmitool sensor thresh "FAN4" upper 1700 1900 2100
ipmitool sensor thresh "FAN4" lower 400 300 200


Note that I did NOT have to load the kernel module as stated in the HOWTO.
I also noted that the FAN1 upper thresholds were abnormally high and after initial tests were set to:
Code:
ipmitool sensor thresh FAN1 upper 2500 2600 2700

When I review the results, I get the following:

Code:
 ~# ipmitool sensor list all
CPU Temp         | 36.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 95.000    | 98.000    | 100.000
System Temp      | 50.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
Peripheral Temp  | 45.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000
PCH Temp         | 52.000     | degrees C  | ok    | -11.000   | -8.000    | -5.000    | 90.000    | 95.000    | 100.000
VRM Temp         | 40.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 87.000    | 92.000    | 97.000
DIMMA1 Temp      | 33.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
DIMMA2 Temp      | 33.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
DIMMB1 Temp      | 32.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
DIMMB2 Temp      | 32.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000
FAN1             | 900.000    | RPM        | ok    | 400.000   | 600.000   | 800.000   | 2500.000  | 2600.000  | 2700.000
FAN2             | 700.000    | RPM        | ok    | 400.000   | 300.000   | 200.000   | 1700.000  | 1900.000  | 2100.000
FAN3             | 600.000    | RPM        | ok    | 400.000   | 300.000   | 200.000   | 1700.000  | 1900.000  | 2100.000
FAN4             | 600.000    | RPM        | ok    | 400.000   | 300.000   | 200.000   | 1700.000  | 1900.000  | 2100.000
FANA             | 600.000    | RPM        | ok    | 400.000   | 300.000   | 200.000   | 1700.000  | 1900.000  | 2100.000
Vcpu             | 1.809      | Volts      | ok    | 1.242     | 1.260     | 1.395     | 1.899     | 2.088     | 2.106
VDIMM            | 1.320      | Volts      | ok    | 1.096     | 1.124     | 1.201     | 1.642     | 1.719     | 1.747
12V              | 12.051     | Volts      | ok    | 10.164    | 10.521    | 10.776    | 12.918    | 13.224    | 13.224
5VCC             | 4.969      | Volts      | ok    | 4.225     | 4.380     | 4.473     | 5.372     | 5.527     | 5.589
3.3VCC           | 3.329      | Volts      | ok    | 2.804     | 2.894     | 2.969     | 3.554     | 3.659     | 3.689
VBAT             | 3.000      | Volts      | ok    | 2.400     | 2.490     | 2.595     | 3.495     | 3.600     | 3.690
AVCC             | 3.329      | Volts      | ok    | 2.399     | 2.489     | 2.594     | 3.494     | 3.599     | 3.689
VSB              | 3.269      | Volts      | ok    | 2.399     | 2.489     | 2.594     | 3.494     | 3.599     | 3.689
Chassis Intru    | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na
 ~#


It seems that the lower Lower Non-Recoverable and Lower Non-Critical values for Fan(2, 3, 4, and A) are reversed when I compare them to Fan1.
Non-Recoverable = 400
Lower Non-Critical = 200​

In another post here, the values appear to be in line with what I was expecting.

Have I missed something?
 
Last edited:

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
No, you're right. The parameters in the command are in the same order as in the table. So the lower and upper parameters should be in the opposite order of each other ;)
 

dj_jazzn

Dabbler
Joined
Apr 14, 2014
Messages
31
Thanks for the confirmation. I will correct the order of my lower threshold limits.
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
... It seems that the lower Lower Non-Recoverable and Lower Non-Critical values for Fan(2, 3, 4, and A) are reversed when I compare them to Fan1.
Non-Recoverable = 400
Lower Non-Critical = 200​

In another post here, the values appear to be in line with what I was expecting.

Have I missed something?
Thanks for catching this - correction made.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Added a warning to my guide about the order of the thresholds.

I'm curious, however: Anyone notice any weird behavior that could possibly have been caused by the wacky thresholds?
 

GrumpyBear

Contributor
Joined
Jan 28, 2015
Messages
141
Added a warning to my guide about the order of the thresholds.

I'm curious, however: Anyone notice any weird behavior that could possibly have been caused by the wacky thresholds?
Nope - ipmitool does not appear to do any validation of input. The result of having the non-critical and non-recoverable values reversed appears to not have had any impact. I would have noticed this had it thrown an alert or warning in the log like what occurs when I had the default values and the Noctua or Cougar fans spun down to their 30% PWM speeds which tripped the non-recoverable threshold and forced the fans into high-speed mode permanently (only resetting the BMC or a reboot fixed this - never did figure out how to clear the alarm after setting the new thresholds)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Doesn't the IPMI GUI alarm reset button do the trick?
 

dj_jazzn

Dabbler
Joined
Apr 14, 2014
Messages
31
Addressing some uncomfortable HDD temperature values during a burn-in procedure...

I was expecting the fans to respond in some way to the higher drive temperatures, but they did not. Is the badblocks continuous write/read operation any more stressful to the HDDs than other stress test methods? The CPU is certainly not being loaded while running badblocks.

Wondering if I have missed configuration step?

My system is currently configured as stated in the post above:
Next the 5-way PWM Splitter was installed and all case fans were connected through it to the FAN-A header and the fan mode again set to "Optimal" with the fans connected thus:

FAN-1 - CPU
FAN-2 - Front Upper Fan
FAN-3 - Front Lower Fan
FAN-4 - Rear Top Fan
FAN-A - Rear Back Fan​

While following this Hard Drive Burn in Testing How To and testing 5 drives concurrently using:
Code:
badblocks -ws /dev/daX

two drives temps were above the 40°C mark. (one drive was at 41°C and another at 42°C). Meanwhile, the case fan speeds (2-4 and A) were around 500-600RPM while in OPTIMAL mode.

To address this, I set the fan mode to FULL using:
Code:
ipmitool raw 0x30 0x45 0x1 0x1

With the case fans now set at full speed (1600RPM), temperatures of the drive returned to around 34-37°C. I continued with FULL fan mode for the remainder of the burn-in process.

All of the MB temperature sensor thresholds are currently unchanged at there default values:
Code:
CPU Temp         | 42.000     | degrees C  | ok    | 0.000     | 0.000     | 0.000     | 95.000    | 98.000    | 100.000  
System Temp      | 42.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000   
Peripheral Temp  | 41.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000   
PCH Temp         | 49.000     | degrees C  | ok    | -11.000   | -8.000    | -5.000    | 90.000    | 95.000    | 100.000  
VRM Temp         | 39.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 87.000    | 92.000    | 97.000   
DIMMA1 Temp      | 33.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000   
DIMMA2 Temp      | 32.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000   
DIMMB1 Temp      | 32.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000   
DIMMB2 Temp      | 31.000     | degrees C  | ok    | 1.000     | 2.000     | 4.000     | 80.000    | 85.000    | 90.000  


System hardware details at the time of this post:
FreeNAS 9.3 Stable
Intel Xenon E3-1231v3 | Supermicro X10SL7-F-O | 4X8GB Crucial 1600Mhz ECC CT2KIT102472BD160B
2X16GB Cruzer Fit Mirrored (Internally mounted 2 Port USB Motherboard Header Adapter USBMBADAPT2)
5X4TB WD40EFRX (INSTALLED) | 6X4TB - ZFS2 (PLANNED)
Fractal Define R4 | Seasonic G-450 | 4X Cougar Vortex PWM CF-V12HP | UPS TBD

The hard drives are mounted in the bottom 5 bays with the power and data cables pointing toward the RHS removable case panel. The two front mounted fans pulling air into the case and two rear mounted fans pushing air out of the case.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
It's normal, the drive temps aren't used in the fan control on this MB (sadly).

The only way to do this is to use some external controller with temp sensor on the drives.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Drive temperatures are very hard to include generically. For this reason, to include them, you have to narrow your scope by making assumptions, which means you can't realistically do it on a motherboard.

The cleanest realistic solution is a custom fan controller that gets fed drive and sensor temperatures from the OS (say, via serial connection).
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Or a MCU with some temp sensor, or a more analogue circuit as I plan to do... But for this you need some knowledge in the electronic realm.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Or a MCU with some temp sensor, or a more analogue circuit as I plan to do... But for this you need some knowledge in the electronic realm.

Sure, but I'm way to lazy to tape temperature sensors to my hard drives. When I get around to it, I'll go for the digital approach, since that allows for greater scalability.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
In theory one sensor is ok as all the drives should be roughly at the same temp. But, yeah, it's a tradeoff: on one side you have the scalability and on the other you have the standaloness (yeah, it's a word... :P) ;)
 

dj_jazzn

Dabbler
Joined
Apr 14, 2014
Messages
31
Would it be possible to indirectly influence the fans speeds by lowering one or more of the temperature sensor thresholds? For example, the closest temperature sensor on the X10SL7-F to my HDD cages is the RT1 System sensor. Although not a direct measure of the disk temperature, could it be influenced by disk temperature and the direction of air flow through the case? Perhaps some experimenting is in order while stress testing the system.
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
The thresholds don't do anything to the fan speed, it's just thresholds for IPMI alerts.

And I don't think the air temp will vary by much with the drive's temp, but more likely with the room air temp...

But I have a nasty idea: write a script who loads the CPU in function of the drives temp :D

Edit: this reminds me that: https://xkcd.com/1172/
 
Status
Not open for further replies.
Top