PID Fan Control Script with Detailed Logs
EDIT 2: Updated spinpid.sh to include a setting variable (FAN_MIN) to specify minimum fan duty cycle (%). This was suggested by
@Kevin Horton and can be adjusted to prevent fan stalling at too low a speed. 2017-01-01
EDIT: I found when the processor was under heavy load, its temperature spiked. Of course the script only responds to drive temperature. So, like
@Stux did for X10 boards, I modified the script substantially to respond to both drive and CPU temperature. My board has only one fan zone, so it is not split as in Stux's script. The relationship is a bit complicated, but I have tested for days with a lot of scrubs and processor-intensive tasks, and it has worked very well. When the drives need cooling, PID is still used. When the CPU needs more attention, the fans simply ramp up with temperature, and there are two settings to regulate that. New scripts attached.
****************************************
Since being inspired by the ideas and facts presented in this thread, especially the tricks that Kevin discovered to control fans, I’ve spent way too much time working on this. My motivation is that the PWM fan control logic in my motherboard is lousy. Much of the time, setting the mode to Standard is inadequate and HeavyIO is overkill. Either way, the temperatures fluctuate a lot, so I don’t see switching between them in a script as an ideal solution.
Instead, adjusting fan duty cycle to maintain a steady temperature seems to be the way to go. There are a couple of problems though:
- BMC won’t give up control (EDIT: @PigLover found the solution to this: leaving fan mode on Full. That is incorporated into the script now). It’s hard to develop control logic that works when the BMC independently decides to change duty cycle. This may happen when you set a duty cycle near the upper/lower end, >80% or <30%. Limiting your own settings between those figures doesn’t help much, since the logic also doesn’t work right if you hit a control limit. I’ve taken two approaches to dealing with this:
- When setting the duty cycle ≥ 50%, first set the mode to HeavyIO, and when setting <50%, first set mode to Standard. This has nothing to do with regulating temperature, it is just less likely the BMC will feel the need to make adjustments in those ranges.
- Control the temperature so well that fans rarely, if ever, will get near the extremes of duty cycle.
- Temperature readings are way too coarse for fine control. Using the temperature of the hottest drive (Tmax) is a good strategy in principle, but you only have 1 C resolution. Let’s say your setpoint (SP) is 36 C. You can successfully keep Tmax in the range 35-37, but you will never do better than that, and it will be very hard to keep it from oscillating. The solution I’ve found is using the mean temperature of all drives (Tmean). Even with slight temperature change, chances are that one of the drives will cross the degree threshold and its temperature reading will change. Using Tmean (with at least 1 decimal place) gives you much more and earlier information about temperature trends in your drives.
First, a few basics in case there are any fan newbies. There are two fan settings you can control: mode and duty cycle. You can read those plus RPM.
- Mode. There are 3-5 modes, depending on the board. Mine has Standard, HeavyIO, and Full. Full is full speed all the time (unless you change it with an ipmitool raw command). Standard and HeavyIO have some degree of speed control based on board temperatures. HeavyIO is a higher speed at a given temperature. The details don’t seem to be well documented, but that doesn't matter, since it works very poorly anyway.
- Duty cycle and RPM. Duty cycle is a percentage of full power applied to the fans. It is correlated with RPM, actual fan speed, but you can’t set RPM directly. And when you read RPM it is rounded to the nearest 100.
I read whatever I could understand on PID control. Turns out, as BiduleOhm said, it doesn’t have to be as complicated as they make it out to be. Here’s how it works. Based on temperature error from your setpoint, you calculate three corrections (P, I, and D) to adjust the duty cycle. Each has a tuning constant: Kp, Ki, and Kd. Note that T (time between control cycles) is sometimes included in the calculation of I and D. It makes it a bit more complicated, but I think then your tuning constants don’t break if you change T. However, I’ve never changed T to see what happens.
First you calculate the current error as ERRc = Tmean – SP. Most sources say to calculate error the other way, SP-Tmean, but it makes more sense to have error be positive when temps are too high and you need to increase fan speed.
- P is for proportional. This is a correction proportional to the current error. Just multiply by the tuning constant. So the formula for P is simply Kp * ERRc.
- I is for integral. This is a correction for cumulative error. So every cycle, you add the current error (ERRc) * T to cumulative error (ERR), and multiply by a tuning constant. So ERR = ERRc*T + ERR, then I = Ki * ERR. My understanding is this helps correct offset; where the temperature stays a bit below or above SP.
- D is for derivative. That is change in current error with respect to time, or basically the slope of the error line. In practice it’s even simpler, you just need to subtract ERRc from the previous error (ERRp), divide by T and multiply by another tuning constant. So D = Kd * ((ERRc – ERRp) / T). This really does two things. When you start a scrub or the sun hits your NAS, you get a large positive error. The bigger the increase in error, the bigger D is. In this case, D and P are additive, aggressively increasing duty cycle. But then when the temps are cooling fast, coming back down to SP, P is still positive and D is negative, so D counters P and puts the brakes on the fans, reducing overshoot and subsequent oscillation.
Most of what I’ve read says, in the great majority of cases, D is not needed, and you can just use PI. So I worked with PI for a long time, trying all kinds of tuning, and always had oscillation. By then I was using Tmean instead of Tmax, and I put the derivative term in and it was MUCH more stable. On the other hand, I have not seen any sign of offset, and I does more harm than good, so I ended up setting Ki to 0.
Here are a few graphs showing some preliminary trials. First using Tmax as the process variable, SP is 36. Kp is too high.
View attachment 11763
Tried adding the I term. Many experiments I won't bore you with. This one using Tmean, still bad oscillation. This shows cumulative error too. No improvement
View attachment 11764
And the script and tuning I ended up with, showing Tmax, Tmean, and duty cycle.. As a test, this starts with a large error. See how Tmean comes down to SP without overshooting. Then there are minor corrections through the day, and you can see the fans increase as the sun comes in in the afternoon.
View attachment 11765
Here is what the log looks like. The stuff on the right (starting with ERRc) can be turned off; I just use it for diagnosis and tuning. This log begins with starting the script. It gets equilibrated about 30 minutes in.
Code:
Saturday, May 14
da0 da1 da2 da3 ada0 ada1 ada2 Tmax Tmean RPM MODE Fan% Curr/New
15:01:03 Spin 35 Spin 35 Spin 36 Spin 35 Spin 31 Spin 32 Spin 32 ^36 33.71 800 Standard 49/51 ERRc= 0.14; P= 0.58; I= 0.00; D= 1.15
15:06:05 Spin 35 Spin 35 Spin 36 Spin 36 Spin 31 Spin 32 Spin 32 ^36 33.86 800 HeavyIO 51/53 ERRc= 0.29; P= 1.15; I= 0.00; D= 1.14
15:11:06 Spin 35 Spin 35 Spin 36 Spin 36 Spin 32 Spin 33 Spin 32 ^36 34.14 800 HeavyIO 53/58 ERRc= 0.57; P= 2.29; I= 0.00; D= 2.28
15:16:07 Spin 35 Spin 35 Spin 36 Spin 36 Spin 32 Spin 33 Spin 32 ^36 34.14 900 HeavyIO 58/60 ERRc= 0.57; P= 2.29; I= 0.00; D= 0.00
15:21:08 Spin 35 Spin 35 Spin 36 Spin 36 Spin 32 Spin 33 Spin 32 ^36 34.14 900 HeavyIO 60/62 ERRc= 0.57; P= 2.29; I= 0.00; D= 0.00
15:26:09 Spin 35 Spin 35 Spin 36 Spin 35 Spin 32 Spin 33 Spin 32 ^36 34.00 900 HeavyIO 62/63 ERRc= 0.43; P= 1.72; I= 0.00; D= -1.13
15:31:10 Spin 35 Spin 34 Spin 36 Spin 35 Spin 32 Spin 32 Spin 32 ^36 33.71 1000 HeavyIO 63/61 ERRc= 0.14; P= 0.58; I= 0.00; D= -2.28
15:36:12 Spin 35 Spin 34 Spin 36 Spin 35 Spin 31 Spin 32 Spin 32 ^36 33.57 900 HeavyIO 61/60 ERRc= 0.00; P= 0.00; I= 0.00; D= -1.14
15:41:13 Spin 35 Spin 34 Spin 36 Spin 35 Spin 31 Spin 32 Spin 32 ^36 33.57 900 HeavyIO 60/60 ERRc= 0.00; P= 0.00; I= 0.00; D= 0.00
15:46:14 Spin 35 Spin 34 Spin 36 Spin 35 Spin 31 Spin 32 Spin 32 ^36 33.57 900 HeavyIO 60/60 ERRc= 0.00; P= 0.00; I= 0.00; D= 0.00
15:51:15 Spin 35 Spin 34 Spin 36 Spin 35 Spin 31 Spin 32 Spin 32 ^36 33.57 900 HeavyIO 60/60 ERRc= 0.00; P= 0.00; I= 0.00; D= 0.00
15:56:16 Spin 35 Spin 34 Spin 36 Spin 35 Spin 31 Spin 32 Spin 32 ^36 33.57 900 HeavyIO 60/60 ERRc= 0.00; P= 0.00; I= 0.00; D= 0.00
This is MUCH better than the control built into the boards. Tmean normally stays within 0.3 C of SP unless there is a disturbance, then within 0.5 C. It is damn near perfect. I guess SuperMicro doesn’t do something like this because (a) they’re more interested in protecting the board than the drives, and the board is less sensitive to temperature variation, (b) accessing drive temperatures depends on the OS, and (c) it requires tuning.
Tuning
PID tuning advice on the internet generally does not work well for controlling drive temperatures in my experience.
- First run the script spincheck.sh (logs detailed data only, no control) and get familiar with your temperature and fan variations without any intervention.
- Now in the settings of spinpid.sh, choose a setpoint that is an actual observed Tmean, given the number of drives you have. It should be the Tmean associated with the Tmax that you want.
- Set Ki=0 and leave it there. You probably will never need it.
- Start with Kp low. Use a value that results in a rounded correction=1 when error is the lowest value you observe other than 0 (i.e., when ERRc is minimal, Kp ~= 1 / ERRc). However, if you have few drives and thus coarser temperature monitoring, you may need a larger Kp. I would not go below 4.
- Set Kd at about Kp*10
- Get Tmean within ~0.3 degree of SP before starting script. At this stage you don’t want to test a large error, you want an equilibrated system.
- Start script and run for a few hours or so. If Tmean oscillates (best to graph it), you probably need to reduce Kd. If no oscillation but response is too slow, raise Kd.
- Stop script and get Tmean at least 1 C off SP. Restart. If there is overshoot and it goes through some cycles, you may need to reduce Kd.
- If you have problems, examine P and D in the log and see which is messing you up. Most likely Kd needs tuning. You can try raising Kp, though too high and changes become too aggressive and you get overshoot and oscillation. You can even try using Ki. If you use Ki, make it small, ~ 0.1 or less.
Scripts
There are two bash scripts attached (change extension to .sh after you download) (EDIT: scripts updated 2016-07-02).
spincheck.sh logs data only, it does not control anything.
spinpid.sh logs and controls. Both scripts log:
- disk status (spinning or standby)
- disk temperature (Celsius) if spinning
- max and mean disk temperature
- fan rpm and mode
- current fan duty cycle (plus new one for spinpid.sh)
- optional diagnostic variables
The scripts include disks on motherboard as well as on an HBA. They get a list of devices from camcontrol devlist. I edit that to delete my SanDisk flash drives from the list; you may have to change that for your system. Suggestions in the script.
The mode code is primarily based on my board. If you have different modes you may need to make some minor tweaks. I have no idea if any of this is applicable to non-Supermicro boards.
As usual, you are responsible for anything you do on your system. This works for me, but for all I know it could make your box catch fire or cause a zombie apocalypse. I suggest you monitor closely at first.
After testing, if you decide to use it, you can run it as a post-init script (Tasks in the GUI) so it starts automatically after booting. In this case, to avoid ‘windup’ (a large error when starting with possibly cold drives), you may want to add a ‘sleep 1200’ before the main loop. Then it will wait 20 minutes for the drives to warm up before doing anything.
