Fan Scripts for Supermicro Boards Using PID Logic 2020-08-20, previous one was missing a file

[Glorious1]

The script is still showing a "drive" da12 that apparently is not a spinning drive. It isn't in the camcontrol devlist output that you posted. Is it possible that the last line of that output was not pasted correctly? Otherwise I can't imagine what is going on. If you don't see it there, try smartctl -a /dev/da12.

As for a graph, you can paste it into an excel spreadsheet and go from there. Of course that's not live. Alternatively, go back through the thread to find the posts by @barbierimc. He added a few functions to the script that send the data to a program called graphana, as I recall.

 

[spacecabbie]

The script is still showing a "drive" da12 that apparently is not a spinning drive. It isn't in the camcontrol devlist output that you posted. Is it possible that the last line of that output was not pasted correctly? Otherwise I can't imagine what is going on. If you don't see it there, try smartctl -a /dev/da12.

As for a graph, you can paste it into an excel spreadsheet and go from there. Of course that's not live. Alternatively, go back through the thread to find the posts by @barbierimc. He added a few functions to the script that send the data to a program called graphana, as I recall.

Owww yea thats the usb boot drive (lexar jumpdriver p20 16gb
That got cut apparently sorry.

Code:

<Samsung SSD 840 EVO 500GB EXT0DB6Q>  at scbus6 target 0 lun 0 (pass0,ada0)
<Samsung SSD 840 EVO 500GB EXT0AB0Q>  at scbus7 target 0 lun 0 (pass1,ada1)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus8 target 0 lun 0 (pass2,ses0)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 8 lun 0 (pass3,da0)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 9 lun 0 (pass4,da1)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 10 lun 0 (pass5,da2)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 11 lun 0 (pass6,da3)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 12 lun 0 (pass7,da4)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 13 lun 0 (pass8,da5)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 14 lun 0 (pass9,da6)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 15 lun 0 (pass10,da7)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 16 lun 0 (pass11,da8)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 17 lun 0 (pass12,da9)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 18 lun 0 (pass13,da10)
<HGST HUS724040ALS640 A1CD>        at scbus9 target 19 lun 0 (pass14,da11)
<HP HP SAS EXP Card 2.10>          at scbus9 target 20 lun 0 (pass15,ses1)
<Lexar USB Flash Drive 1.00>       at scbus11 target 0 lun 0 (pass16,da12)

Code:

smartctl -a /dev/da12
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/da12: Unknown USB bridge [0x05dc:0xa836 (0x1aa2)]
Please specify device type with the -d option.

 

[Glorious1]

Owww yea thats the usb boot drive (lexar jumpdriver p20 16gb
That got cut apparently sorry.

Well, you probably know how to fix it. Attached is spinpid2.sh with that fix and the others. Also the config file that goes with it. You'll need to go through all the config settings, which involves first setting your fan thresholds (see the link to @Ericloewe 's method in the resource overview) and then running spintest.sh to get some of the fan data.

 

[Glorious1]

Glorious1 updated Fan Scripts for Supermicro Boards Using PID Logic with a new update entry:

2020-06-17 (minor update)

This is a minor update with no changes to the actual fan control logic. If the fan scripts are working fine, you can safely ignore this. There are some changes to make the scripts compatible with running in a virtual machine, and with SAS drives. Changes:

1. Moved IPMITOOL definition to config file for spinpid.sh and spinpid2.sh (other scripts don't use config file). If doing VM, this makes it easier to add your IP, user, and password and survive updates. This is the only real...

Read the rest of this update entry...

 

[mistermanko]

Any idea why IPMI would report different RPM values than your script?

EDIT: just recognized your script accounts for that.

Mismatch between CPU Duty and RPMs -- DUTY_CPU=20; RPM_CPU=1700
Attempting to fix CPU mismatch

Good work!

 

[Glorious1]

Any idea why IPMI would report different RPM values than your script?

View attachment 39436
View attachment 39437

The script gets the numbers from ipmitool, so most likely it's just a matter of timing. Try it a number of times when the temps and fans are stabilized, not changing. If you post back, please send a whole line of the script output from the spinpid2.log. You can get up-to-date ipmi data by entering [sudo] ipmitool sdr in CLI or SSH session rather than the IPMI GUI.

 

[mistermanko]

The script gets the numbers from ipmitool, so most likely it's just a matter of timing. Try it a number of times when the temps and fans are stabilized, not changing. If you post back, please send a whole line of the script output from the spinpid2.log. You can get up-to-date ipmi data by entering [sudo] ipmitool sdr in CLI or SSH session rather than the IPMI GUI.

Thanks, actually your script had to adjust the reading and did it by itself. Very impressive!
Another question, why did you settle for a drive temp setpoint of 33.57? Is there any science behind that or just personal preference?

 

[Glorious1]

Thanks, actually your script had to adjust the reading and did it by itself. Very impressive!
Another question, why did you settle for a drive temp setpoint of 33.57? Is there any science behind that or just personal preference?

I'm not sure what adjustment you're referring to. As far as I can recall the script just reports the fan speeds it gets from ipmitool.

The mean drive temperature is a mean of integers (drives report temp to the nearest degree C). So the possible mean values are not continuous. For a setpoint, I watched the logs and picked an actual mean value that was about what I wanted. That way, the script isn't bouncing back and forth trying to reach a mean value it never can exactly get to. But it's probably not that important.

 

[mistermanko]

I'm not sure what adjustment you're referring to. As far as I can recall the script just reports the fan speeds it gets from ipmitool.

The mean drive temperature is a mean of integers (drives report temp to the nearest degree C). So the possible mean values are not continuous. For a setpoint, I watched the logs and picked an actual mean value that was about what I wanted. That way, the script isn't bouncing back and forth trying to reach a mean value it never can exactly get to. But it's probably not that important.

I was referring to that message:

Mismatch between CPU Duty and RPMs -- DUTY_CPU=20; RPM_CPU=1700
Attempting to fix CPU mismatch

 

[Glorious1]

I was referring to that message:

Oh, I never saw that actually happen, just an attempt to monitor and fix if it did. If you still have that log, and the CPU log, could you send them to me so I can see what was going on? Have you only seen that once? Have you set your fan thresholds per @Ericloewe 's post?

 

[mistermanko]

Oh, I never saw that actually happen, just an attempt to monitor and fix if it did. If you still have that log, and the CPU log, could you send them to me so I can see what was going on? Have you only seen that once? Have you set your fan thresholds per @Ericloewe 's post?

It appears after the first duty cycle and never again after it, well because it's fixed than.

Have you set your fan thresholds per @Ericloewe 's post?

Yes, of course. Supermicros standard thresholds are bogus if used with standard ATX cooling solutions.

Here is the relevant portion of the log. The script registers the mismatch and fixes the RPM. unrelevant note: I've set the disk temp set point to 35 C, below that my 3 Noctua A14 are not sufficient anymore.

Code:

21:23:12  *35  *35  *34  *32  *36  *32  ^36  34.00  -1.57  -6.28   4.00   41 Full     20  20    300  1000   ---   ---   ---
21:28:44  *36  *35  *35  *33  *37  *33  ^37  34.83  -0.74  -2.96   6.64   59 Full     50  24    300  1700   ---   ---   ---
Mismatch between CPU Duty and RPMs -- DUTY_CPU=20; RPM_CPU=1700
Attempting to fix CPU mismatch 

21:34:21  *36  *35  *35  *34  *37  *34  ^37  35.16  -0.41  -1.64   2.64   45 Full     20  25    300  1000   ---   ---   ---
21:39:52  *37  *36  *35  *34  *38  *34  ^38  35.66   0.09   0.36   4.00   46 Full     20  29    400  1000   ---   ---   ---
21:45:24  *37  *36  *35  *34  *38  *35  ^38  35.83   0.26   1.04   1.36   48 Full     20  31    400  1000   ---   ---   ---
21:50:54  *37  *36  *35  *34  *39  *35  ^39  36.00   0.43   1.72   1.36   48 Full     20  34    500  1000   ---   ---   ---

 

[Glorious1]

Here is the relevant portion of the log.

Please send the full log as an attachment, including CPU log if you have it. You probably will have to add ".txt" suffix.

 

[mistermanko]

Please send the full log as an attachment, including CPU log if you have it. You probably will have to add ".txt" suffix.

Here you go.

 

[bestboy]

I just updated the script from one of the ancient, early versions to the most recent one and noticed a regression: The New_Fan% columns are always 0.

New version:

Code:

root@cygnus:/mnt/tank/scripts/spin # ./spinpid2.sh

****** SETTINGS ******
CPU zone 1; Peripheral zone 0
CPU fans min/max duty cycle: 20/100
PER fans min/max duty cycle: 1/100
CPU fans - measured RPMs at 30% and 100% duty cycle: 700/2000
PER fans - measured RPMs at 30% and 100% duty cycle: 3400/7100
Drive temperature setpoint (C): 35
Kp=4, Kd=40
Drive check interval (main cycle; minutes): 3
CPU check interval (seconds): 2
CPU reference temperature (C): 45
CPU scalar: 4
Reading fan duty from board
Getting CPU temperatures via sysctl

Key to drive status symbols:  * spinning;  _ standby;  ? unknown                              Version 2020-06-17

Monday, Aug 17                                                                                                                  CPU         New_Fan%  New_RPM_____________________
          da0  da1  da2  da3  da4  da5  da6  da7  da8  da9  da10 da11 da12 da13 da14 da15 ada0 Tmax Tmean   ERRc      P      D TEMP MODE    CPU PER   FANA  FAN1  FAN2  FAN3  FAN4
18:43:17  *33  *35  *33  *32  *35  *34  *32  *34  *38  *35  *38  *37  *37  *34  *36  *35  *33  ^38  34.76  -0.24  -0.96  -1.92   41 Full      0   0    400  1700  1800  1600   800^C

Old version:

Code:

root@cygnus:/mnt/tank/scripts/spin # ./_old/spinpid2_mod.sh

****** SETTINGS ******
CPU zone 1; Peripheral zone 0
CPU fans min/max duty cycle: 20/100
PER fans min/max duty cycle: 1/100
CPU fans - measured RPMs at 30 0x1.2cp+9nd 100 2000uty cycle: /
PER fans - measured RPMs at 30 0x1.a9p+11nd 100 7200uty cycle: /
Drive temperature setpoint (C): 35
Kp=4, Ki=0, Kd=40
Drive check interval (main cycle; minutes): 3
CPU check interval (seconds): 2
CPU reference temperature (C): 45
CPU scalar: 4

Key to drive status symbols:  * spinning;  _ standby;  ? unknown                              Version 2017-04-10

Monday, Aug 17                                                                                                                             CPU         New_Fan%  New_RPM_____________________
          da0  da1  da2  da3  da4  da5  da6  da7  da8  da9  da10 da11 da12 da13 da14 da15 ada0 ses0 Tmax Tmean   ERRc      P     I      D TEMP MODE    CPU PER   FANA  FAN1  FAN2  FAN3  FAN4
18:43:51  *33  *35  *33  *32  *35  *34  *32  *34  *38  *35  *38  *37  *37  *34  *36  *35  *33  _0   ^38  34.76  -0.24  -0.96  0.00  -3.20   42 Full     50  46   1100  4100  4200  3900  2000^C

Any idea what might cause this?

 

[Glorious1]

Thanks for the report. Off the top of my head, I'm not sure. I looked at my log from testing that version and it doesn't give 0 for new duty cycles. I assume this is not happening for most people since this is the first I've heard. A couple of things to try:

1. Let it go two full cycles (i.e., run it for over 3 minutes). See if it gets it right the second time.
2. In your config file, change the HOW_DUTY variable to 0 instead of 1 and see if that fixes it.
3. When you ran the newer script, the fans were running very slow compared to your 30% duty. That might make sense for your peripheral fans since you're allowing them to go down to 1% minimum. I'm not sure about the CPU fan. You might try going into your IPMI and setting fan mode to some other mode, then switch it to full, before running the script, and see what happens.
4. If all else fails, go into the script. Near the bottom of the DRIVES_check_adjust function, there are two lines preceded by a comment "DIAGNOSTIC variables . . .". Uncomment the two lines (228 and 234) that begin with "printf". Run the script (at least 2 cycles) and send me the output.

Keep me posted.

 

[bestboy]

2. In your config file, change the HOW_DUTY variable to 0 instead of 1 and see if that fixes it.

This did the trick.

Code:

root@cygnus:/mnt/tank/scripts/spin # ./spinpid2.sh

****** SETTINGS ******
CPU zone 1; Peripheral zone 0
CPU fans min/max duty cycle: 20/100
PER fans min/max duty cycle: 1/100
CPU fans - measured RPMs at 30% and 100% duty cycle: 700/2000
PER fans - measured RPMs at 30% and 100% duty cycle: 3400/7100
Drive temperature setpoint (C): 35
Kp=4, Kd=40
Drive check interval (main cycle; minutes): 3
CPU check interval (seconds): 2
CPU reference temperature (C): 45
CPU scalar: 4
Assuming fan duty as set
Getting CPU temperatures via sysctl

Key to drive status symbols:  * spinning;  _ standby;  ? unknown                              Version 2020-06-17

Monday, Aug 17                                                                                                                  CPU         New_Fan%  New_RPM_____________________
          da0  da1  da2  da3  da4  da5  da6  da7  da8  da9  da10 da11 da12 da13 da14 da15 ada0 Tmax Tmean   ERRc      P      D TEMP MODE    CPU PER   FANA  FAN1  FAN2  FAN3  FAN4
20:07:13  *30  *33  *31  *29  *32  *32  *31  *33  *37  *34  *36  *35  *34  *33  *33  *33  *33  ^37  32.88  -2.12  -8.48 -28.27   39 Full     50  13   1100  2500  2700  2500  1200
Mismatch between CPU Duty and RPMs -- DUTY_CPU=20; RPM_CPU=1100
Attempting to fix CPU mismatch

20:10:30  *31  *33  *31  *30  *33  *33  *31  *33  *37  *34  *36  *35  *34  *33  *34  *33  *33  ^37  33.17  -1.83  -7.32   3.87   42 Full     20   1    400  1700  1800  1600   800
20:13:42  *32  *34  *32  *31  *33  *33  *32  *34  *37  *35  *36  *35  *35  *34  *35  *34  *33  ^37  33.82  -1.18  -4.72   8.67   44 Full     20   5    400  1900  2000  1800   800
20:16:53  *32  *34  *32  *31  *34  *33  *32  *34  *37  *35  *37  *36  *35  *34  *35  *34  *33  ^37  34.00  -1.00  -4.00   2.40   44 Full     20   3    400  1800  1900  1800   800
20:20:05  *32  *34  *32  *31  *34  *33  *32  *34  *37  *35  *37  *36  *36  *34  *35  *35  *33  ^37  34.11  -0.89  -3.56   1.47   44 Full     20   1    400  1700  1800  1600   800
20:23:17  *33  *35  *32  *32  *34  *34  *32  *34  *38  *35  *38  *36  *36  *35  *35  *35  *32  ^38  34.47  -0.53  -2.12   4.80   44 Full     20   4    400  1800  1900  1800   800
20:26:28  *33  *35  *33  *32  *34  *34  *33  *34  *38  *35  *38  *36  *36  *35  *36  *35  *32  ^38  34.64  -0.36  -1.44   2.27   44 Full     20   5    400  1900  2000  1900   800
20:29:40  *33  *35  *33  *32  *35  *34  *33  *34  *38  *35  *38  *36  *36  *35  *36  *35  *32  ^38  34.70  -0.30  -1.20   0.80   44 Full     20   5    400  1900  2000  1900   800
20:32:51  *33  *35  *33  *32  *35  *34  *33  *34  *38  *35  *38  *37  *37  *35  *36  *35  *32  ^38  34.82  -0.18  -0.72   1.60   44 Full     20   6    400  2000  2100  2000   800
20:36:02  *33  *35  *33  *32  *35  *34  *33  *34  *38  *35  *38  *37  *37  *35  *36  *35  *32  ^38  34.82  -0.18  -0.72   0.00   43 Full     20   5    400  1900  2000  1900   800
20:39:14  *33  *35  *33  *32  *35  *34  *33  *34  *38  *35  *38  *37  *37  *35  *36  *35  *33  ^38  34.88  -0.12  -0.48   0.80   46 Full     24   5    500  1900  2000  1900   800
20:42:25  *33  *35  *33  *32  *35  *34  *33  *34  *38  *35  *38  *37  *37  *35  *36  *35  *33  ^38  34.88  -0.12  -0.48   0.00   43 Full     20   5    400  1900  2000  1900   800
20:45:37  *33  *35  *33  *32  *35  *34  *33  *34  *38  *36  *39  *37  *37  *35  *36  *35  *33  ^39  35.00   0.00   0.00   1.60   45 Full     20   7    400  2000  2100  1900   800

I actually read about the HOW_DUTY option in the config file, but didn't really understand the non-1 option. Maybe consider to improve the description to make it idiot proof :)
Anyways: Thank you.

BTW: Is there anything I can do about the "Mismatch between CPU Duty and RPMs"? I set the 30% and 100% marks according to my spintest results:

Code:

root@cygnus:/mnt/tank/scripts/spin # ./spintest.sh

[...]

Monday, Aug 17, 20:05:05
                                 ___Duty%___  Curr_RPM____________________
                        MODE     Zone0 Zone1  FANA  FAN1  FAN2  FAN3  FAN4
Duty cycle 100%         Full         0     0  2000  6900  7100  6800  2400
Duty cycle 90%          Full         0     0  1800  6400  6400  6200  2300
Duty cycle 80%          Full         0     0  1700  5900  6000  5600  2300
Duty cycle 70%          Full         0     0  1500  5400  5400  5100  2200
Duty cycle 60%          Full         0     0  1300  4900  4900  4600  2200
Duty cycle 50%          Full         0     0  1100  4200  4400  4200  2100
Duty cycle 40%          Full         0     0   900  3800  3900  3600  1900
Duty cycle 30%          Full         0     0   700  3300  3400  3100  1700
Duty cycle 20%          Full         0     0   400  2700  2800  2600  1300
Duty cycle 10%          Full         0     0   100  2200  2300  2100  1400

 

[Glorious1]

I'm guessing your board is reporting incorrect duty cycles (as I've been told some do), and therefore the calculations got screwed up. No need, but if you want to investigate that, you would have to set HOW_DUTY back to 1, and uncomment those diagnostic variables in the script. I'll look into rewriting the description of HOW_DUTY.

The mismatch notice probably would only happen on the first cycle. When HOW_DUTY is not 1 (we're not reading the erroneous duty cycle from the board), we have to assume it's something before the first cycle (we use 50) and go from there. But still I think there is a little bug that I could fix to avoid that. I'll look into it.

 

[bestboy]

If I can help please let me know. Here is the run with the diagnostics output:

Code:

root@cygnus:/mnt/tank/scripts/spin # ./spinpid2_diagnostics.sh

****** SETTINGS ******
CPU zone 1; Peripheral zone 0
CPU fans min/max duty cycle: 20/100
PER fans min/max duty cycle: 1/100
CPU fans - measured RPMs at 30% and 100% duty cycle: 700/2000
PER fans - measured RPMs at 30% and 100% duty cycle: 3400/7100
Drive temperature setpoint (C): 35
Kp=4, Kd=40
Drive check interval (main cycle; minutes): 3
CPU check interval (seconds): 2
CPU reference temperature (C): 45
CPU scalar: 4
Reading fan duty from board
Getting CPU temperatures via sysctl

Key to drive status symbols:  * spinning;  _ standby;  ? unknown                              Version 2020-06-17

Monday, Aug 17                                                                                                                  CPU         New_Fan%  New_RPM_____________________
          da0  da1  da2  da3  da4  da5  da6  da7  da8  da9  da10 da11 da12 da13 da14 da15 ada0 Tmax Tmean   ERRc      P      D TEMP MODE    CPU PER   FANA  FAN1  FAN2  FAN3  FAN4
21:48:08  *34  *36  *33  *32  *35  *35  *33  *35  *39  *36  *39  *37  *38  *35  *37  *36  *33  ^39  35.47   0.47   1.88   6.27   45 Full      0   0    400  2100  2200  2000   900
 DUTY_PER=8, DUTY_PER_LAST=0, DUTY=--, Tmean=35.47, ERRp=0
 DUTY_PER=8, DUTY_PER_LAST=0, DUTY=8, Tmean=35.47, ERRp=0

21:51:20  *34  *36  *33  *32  *35  *35  *33  *35  *39  *36  *39  *37  *38  *35  *36  *36  *33  ^39  35.41   0.41   1.64  -0.80   47 Full      0   0    600  1700  1800  1700   800
 DUTY_PER=1, DUTY_PER_LAST=0, DUTY=28, Tmean=35.41, ERRp=0.47
 DUTY_PER=1, DUTY_PER_LAST=0, DUTY=1, Tmean=35.41, ERRp=0.47

21:54:33  *34  *36  *33  *33  *35  *35  *33  *35  *39  *36  *39  *38  *38  *35  *37  *36  *33  ^39  35.58   0.58   2.32   2.27   48 Full      0   0    700  1900  2000  1900   800
 DUTY_PER=5, DUTY_PER_LAST=0, DUTY=32, Tmean=35.58, ERRp=0.41
 DUTY_PER=5, DUTY_PER_LAST=0, DUTY=5, Tmean=35.58, ERRp=0.41

Here is some basic mainboard/IPMI firmware info of my system:

Code:

root@cygnus:/mnt/tank/scripts/spin # dmidecode -t 2
[...]
Base Board Information
        Manufacturer: Supermicro
        Product Name: X10SLH-F/X10SLM+-F
        Version: 1.02
[...]

root@cygnus:/mnt/tank/scripts/spin # ipmitool mc info
Device ID                 : 32
Device Revision           : 1
Firmware Revision         : 3.84
IPMI Version              : 2.0
Manufacturer ID           : 10876
Manufacturer Name         : Supermicro
Product ID                : 2051 (0x0803)
Product Name              : Unknown (0x803)
Device Available          : yes

EDIT: Oops. I forget to set HOW_DUTY back to 1. I'll update this post later with the proper data.
EDIT2: updated the diagnostic output

 

[Glorious1]

Glorious1 updated Fan Scripts for Supermicro Boards Using PID Logic with a new update entry:

2020-08-20 Minor Update to spinpid2.sh

spinpid2.sh has a few minor changes. Again, if your current version is working, no need to change.

1. Changed the timing of the mismatch test (tests whether fan RPMs are way off from the current duty) so it now happens right after fan data are read. This avoids false mismatch findings that were due to things changing in the CPU cycles.
2. Rearranged some code so it's easier to maintain.
3. Edited some comments so they make more sense (at least to me).
4. The previous configuration...

Read the rest of this update entry...

 

[nasnice]

Hi Glorious,
Thank you for an excellent well documented script!!

After playing around and getting the params right it now runs from the console without a hitch!

Am trying to get it to start post init but have yet to succeed...

I have put the script (spinpid2.sh) in a seperate scripts directory in a shared mount (owner root) it has rwxrwxrwx permissions and fails to run post init.. No discernable errors in the log, in fact that is not created. I edited the config file to have a log file only, (how can you make that append? Tried -a but probably in the wrong place..)

When exactly are post init scripts supposed to start?

In short I am @ a loss. My system is just a big Multimedia file share and has no home directories... Have read through this discussion and found one more user with this problem but his solution did not work for me... Can you point me in the right direction?