ipmi.sel.info and ipmi.sel.elist jobs filling up Tasks, why and how to make it stop?

ibaggett

Cadet
Joined
Jul 14, 2023
Messages
2
Using TrueNAS-SCALE-22.12.3.2 build, the Jobs->History->Tasks page now shows jobs "ipmi.sel.info" and "ipmi.sel.elist" every 5 minutes, but no IPMI hardware is present. These IPMI jobs did not populate the Tasks page in the previous build. I'm guessing it some middleware that is doing this. It crowds out the history for actually important jobs like rsync and replication. If there is a setting to make these (phantom?) jobs go away, I'd like to know what it is. Or if it could get fixed in the next build. I saw a reddit post about this same issue, but no replies with any useful remedy. Thank you for your attention.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Hello @ibaggett

IPMI SEL is the "System Event Log" - normally the cause of logspew from this event source is the SEL being full. Even if you don't have a dedicated IPMI interface, your hardware might have a system event log, or something that TrueNAS thinks is one, and is trying to query from or write to it.

Can you post some details about your hardware configuration, specifically the motherboard?
 

ibaggett

Cadet
Joined
Jul 14, 2023
Messages
2
Hi @HoneyBadger

Thank you for your reply. I'm running 2 instances of TrueNAS-SCALE-22.12.3.2 as VM's on the same machine with Proxmox 7.4-3. One instance has the motherboard's PCIe HBA passed through, to 2 physical SATA hard disks in a mirror pool. The other instance has 2 VirtIO SCSI drives on the local-lvm storage, as 2 disks in a mirror pool. Yes, I know that using VirtIO drives for SCALE pools is not the way, but this is just a learning and test setup. Once I learn and get confident with SCALE, the second instance will be removed and installed on a remote machine with its own physical drives for the pool.

Both instances have been and still are working fine for 2 weeks, but after I updated each instance through the System Settings->Update page, they both started showing this IPMI "logspew". Each ipmi.sel.* job shows "Arguments: No arguments are passed", "Log Path: N/A", "Log Excerpt: No logs are available", and "Error: No errors".
 

opana

Cadet
Joined
Dec 11, 2022
Messages
9
Using TrueNAS-SCALE-22.12.3.2 build, the Jobs->History->Tasks page now shows jobs "ipmi.sel.info" and "ipmi.sel.elist" every 5 minutes, but no IPMI hardware is present
Similar situation. But ipmi hardware is present.
Every 5 minutes two entries in Jobs->History->Tasks
1689718710721.png
OS Version: TrueNAS-SCALE-22.12.3.2
Product: X10SLM-F
Model: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz
Memory: 31 GiB
SEL Information
Version : 1.5 (v1.5, v2 compliant)
Entries : 4
Free Space : 10160 bytes
Percent Used : 0%
Last Add Time : 01/02/2023 21:24:05
Last Del Time : Not Available
Overflow : false
Supported Cmds : 'Reserve' 'Get Alloc Info'
# of Alloc Units : 512
Alloc Unit Size : 20
# Free Units : 508
Largest Free Blk : 508
Max Record Size : 20
1 | 12/06/2022 | 01:09:25 | Unknown #0xff | | Asserted
2 | 12/06/2022 | 01:09:29 | Physical Security Chassis Intru | General Chassis intrusion () | Asserted
3 | 01/02/2023 | 21:19:09 | OS Boot | C: boot completed () | Asserted
4 | 01/02/2023 | 21:24:05 | OS Critical Stop | OS graceful shutdown () | Asserted
CPU Temp | 30.000 | degrees C | ok | 0.000 | 0.000 | 0.000 | 95.000 | 100.000 | 100.000
System Temp | 29.000 | degrees C | ok | -10.000 | -5.000 | 0.000 | 80.000 | 85.000 | 90.000
Peripheral Temp | 37.000 | degrees C | ok | -10.000 | -5.000 | 0.000 | 80.000 | 85.000 | 90.000
PCH Temp | 47.000 | degrees C | ok | 0.000 | 5.000 | 16.000 | 90.000 | 95.000 | 100.000
P1-DIMMA1 Temp | 30.000 | degrees C | ok | -5.000 | 0.000 | 5.000 | 80.000 | 85.000 | 90.000
P1-DIMMA2 Temp | 31.000 | degrees C | ok | -5.000 | 0.000 | 5.000 | 80.000 | 85.000 | 90.000
P1-DIMMB1 Temp | 31.000 | degrees C | ok | -5.000 | 0.000 | 5.000 | 80.000 | 85.000 | 90.000
P1-DIMMB2 Temp | 32.000 | degrees C | ok | -5.000 | 0.000 | 5.000 | 80.000 | 85.000 | 90.000
FAN1 | 800.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FAN2 | na | | na | na | na | na | na | na | na
FAN3 | na | | na | na | na | na | na | na | na
FAN4 | 1000.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
FANA | 1600.000 | RPM | ok | 300.000 | 500.000 | 700.000 | 25300.000 | 25400.000 | 25500.000
Vcpu | 1.800 | Volts | ok | 1.242 | 1.260 | 1.395 | 1.899 | 2.088 | 2.106
VDIMM | 1.488 | Volts | ok | 1.092 | 1.119 | 1.200 | 1.641 | 1.722 | 1.749
12V | 12.320 | Volts | ok | 10.144 | 10.272 | 10.784 | 12.960 | 13.280 | 13.408
5VCC | 5.027 | Volts | ok | 4.244 | 4.487 | 4.730 | 5.378 | 5.540 | 5.594
3.3VCC | 3.299 | Volts | ok | 2.789 | 2.823 | 2.959 | 3.554 | 3.656 | 3.690
VBAT | 2.944 | Volts | ok | 2.384 | 2.496 | 2.580 | 3.476 | 3.588 | 3.672
5V Dual | 5.135 | Volts | ok | 4.244 | 4.379 | 4.487 | 5.378 | 5.540 | 5.594
3.3V AUX | 3.333 | Volts | ok | 2.789 | 2.891 | 2.959 | 3.554 | 3.656 | 3.690
1.2V BMC | 1.269 | Volts | ok | 1.080 | 1.107 | 1.152 | 1.404 | 1.431 | 1.458
1.05V PCH | 1.059 | Volts | ok | 0.870 | 0.897 | 0.942 | 1.194 | 1.221 | 1.248
Chassis Intru | 0x1 | discrete | 0x0100| na | na | na | na | na | na
H/W path Device Class Description
============================================================
system X10SLM-F (To be filled by O.E.M.)
/0 bus X10SLM-F
/0/0 memory 64KiB BIOS
/0/45 processor Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz
/0/45/46 memory 256KiB L1 cache
/0/45/47 memory 1MiB L2 cache
/0/45/48 memory 8MiB L3 cache
/0/4a memory 32GiB System Memory
/0/4a/0 memory 8GiB DIMM DDR3 Synchronous 1600 MHz (0.6 ns)
/0/4a/1 memory 8GiB DIMM DDR3 Synchronous 1600 MHz (0.6 ns)
/0/4a/2 memory 8GiB DIMM DDR3 Synchronous 1600 MHz (0.6 ns)
/0/4a/3 memory 8GiB DIMM DDR3 Synchronous 1600 MHz (0.6 ns)
/0/100 bridge Xeon E3-1200 v3 Processor DRAM Controller
/0/100/14 bus 8 Series/C220 Series Chipset Family USB xHCI
/0/100/14/0 usb1 bus xHCI Host Controller
/0/100/14/0/3 bus Hub
/0/100/14/0/3/1 input Virtual mouse/keyboard device
/0/100/14/1 usb3 bus xHCI Host Controller
/0/100/16 communication 8 Series/C220 Series Chipset Family MEI Controller #1
/0/100/16.1 communication 8 Series/C220 Series Chipset Family MEI Controller #2
/0/100/19 eno2 network Ethernet Connection I217-LM
/0/100/1a bus 8 Series/C220 Series Chipset Family USB EHCI #2
/0/100/1a/1 usb2 bus EHCI Host Controller
/0/100/1a/1/1 bus Integrated Rate Matching Hub
/0/100/1c bridge 8 Series/C220 Series Chipset Family PCI Express Root Port #1
/0/100/1c/0 bridge AST1150 PCI-to-PCI Bridge
/0/100/1c/0/0 display ASPEED Graphics Family
/0/100/1c.1 bridge 8 Series/C220 Series Chipset Family PCI Express Root Port #2
/0/100/1c.1/0 eno1 network I210 Gigabit Network Connection
/0/100/1c.4 bridge 8 Series/C220 Series Chipset Family PCI Express Root Port #5
/0/100/1c.4/0 scsi0 storage SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
/0/100/1c.4/0/0.0.0 /dev/sda disk 2TB WDC WD2005FBYZ-0
/0/100/1c.4/0/0.0.0/1 /dev/sda1 volume 2047MiB swap partition
/0/100/1c.4/0/0.0.0/2 /dev/sda2 volume 1861GiB OS X ZFS partition or Solaris /usr partition
/0/100/1c.4/0/0.1.0 /dev/sdb disk 1TB ST1000DM003-1ER1
/0/100/1c.4/0/0.1.0/1 /dev/sdb1 volume 1023KiB BIOS Boot partition
/0/100/1c.4/0/0.1.0/2 /dev/sdb2 volume 511MiB Windows FAT volume
/0/100/1c.4/0/0.1.0/3 /dev/sdb3 volume 915GiB OS X ZFS partition or Solaris /usr partition
/0/100/1c.4/0/0.1.0/4 /dev/sdb4 volume 15GiB swap partition
/0/100/1c.4/0/0.2.0 /dev/sdd disk 1TB ST1000DM003-1ER1
/0/100/1c.4/0/0.2.0/1 /dev/sdd1 volume 1023KiB BIOS Boot partition
/0/100/1c.4/0/0.2.0/2 /dev/sdd2 volume 511MiB Windows FAT volume
/0/100/1c.4/0/0.2.0/3 /dev/sdd3 volume 915GiB OS X ZFS partition or Solaris /usr partition
/0/100/1c.4/0/0.2.0/4 /dev/sdd4 volume 15GiB swap partition
/0/100/1c.4/0/0.3.0 /dev/sdc disk 480GB KINGSTON SEDC450
/0/100/1c.4/0/0.3.0/1 /dev/sdc1 volume 2047MiB swap partition
/0/100/1c.4/0/0.3.0/2 /dev/sdc2 volume 445GiB OS X ZFS partition or Solaris /usr partition
/0/100/1c.4/0/0.4.0 /dev/sdf disk 2TB WDC WD2005FBYZ-0
/0/100/1c.4/0/0.4.0/1 /dev/sdf1 volume 2047MiB swap partition
/0/100/1c.4/0/0.4.0/2 /dev/sdf2 volume 1861GiB OS X ZFS partition or Solaris /usr partition
/0/100/1c.4/0/0.5.0 /dev/sde disk 2TB WDC WD2005FBYZ-0
/0/100/1c.4/0/0.5.0/1 /dev/sde1 volume 2047MiB swap partition
/0/100/1c.4/0/0.5.0/2 /dev/sde2 volume 1861GiB OS X ZFS partition or Solaris /usr partition
/0/100/1c.4/0/0.6.0 /dev/sdg disk 2TB WDC WD2005FBYZ-0
/0/100/1c.4/0/0.6.0/1 /dev/sdg1 volume 2047MiB swap partition
/0/100/1c.4/0/0.6.0/2 /dev/sdg2 volume 1861GiB OS X ZFS partition or Solaris /usr partition
/0/100/1d bus 8 Series/C220 Series Chipset Family USB EHCI #1
/0/100/1d/1 usb4 bus EHCI Host Controller
/0/100/1d/1/1 bus Integrated Rate Matching Hub
/0/100/1f bridge C224 Series Chipset Family Server Standard SKU LPC Controller
/0/100/1f.3 bus 8 Series/C220 Series Chipset Family SMBus Controller
/0/100/1f.6 generic 8 Series Chipset Family Thermal Management Controller
/0/1 system PnP device PNP0c01
/0/2 system PnP device PNP0c02
/0/3 system PnP device PNP0b00
/0/4 generic PnP device INT3f0d
/0/5 system PnP device PNP0c02
/0/6 communication PnP device PNP0501
/0/7 communication PnP device PNP0501
/0/8 system PnP device PNP0c02
/0/9 system PnP device PNP0c02
/1 power To Be Filled By O.E.M.
/2 br0 network Ethernet interface
/3 vnet0 network Ethernet interface
 
Last edited:

0xDEADBEEF

Cadet
Joined
Aug 7, 2022
Messages
5
I have the same issue on an HP Microserver N54L. Only that it doesn't have any IPMI sensors at all.

The hardware is old, based on an AMD Turion N54L processer, but the motherboard should be well-known. In any case, here's a list of HW.
It's running TrueNAS-SCALE-22.12.3.2. Not that I think it is related, but it boots off from a small SSD, has a single storage pool with a 2-wide mirror of 8TB spindles and has 16GB of RAM.
-Yes I know it is low end hardware, but it is only for off-site backup to be used for disaster recovery if my house burns or my primary instance is otherwise suddenly destroyed.

Code:
H/W path             Device         Class      Description
==========================================================
                                    system     ProLiant MicroServer (744900-421)
/0                                  bus        Motherboard
/0/0                                memory     64KiB BIOS
/0/3                                processor  AMD Turion(tm) II Neo N54L Dual-Core Processor
/0/3/4                              memory     256KiB L1 cache
/0/3/5                              memory     2MiB L2 cache
/0/14                               memory     16GiB System Memory
/0/14/0                             memory     8GiB DIMM Synchronous 1333 MHz (0.8 ns)
/0/14/1                             memory     8GiB DIMM Synchronous 1333 MHz (0.8 ns)
/0/100                              bridge     RS880 Host Bridge
/0/100/1                            bridge     AMD RS780/RS880 PCI to PCI bridge (int gfx)
/0/100/1/5                          display    RS880M [Mobility Radeon HD 4225/4250]
/0/100/6                            bridge     RS780 PCI to PCI bridge (PCIE port 2)
/0/100/6/0           enp2s0         network    NetXtreme BCM5723 Gigabit Ethernet PCIe
/0/100/11            scsi2          storage    SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]
/0/100/11/0          /dev/sdb       disk       8001GB ST8000DM004-2CX1
/0/100/11/0/1        /dev/sdb1      volume     2047MiB swap partition
/0/100/11/0/2        /dev/sdb2      volume     7450GiB OS X ZFS partition or Solaris /usr partition
/0/100/11/1          /dev/sdc       disk       8001GB ST8000DM004-2U91
/0/100/11/1/1        /dev/sdc1      volume     2047MiB swap partition
/0/100/11/1/2        /dev/sdc2      volume     7450GiB OS X ZFS partition or Solaris /usr partition
/0/100/12                           bus        SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/12/1          usb4           bus        OHCI PCI host controller
/0/100/12.2                         bus        SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/100/12.2/1        usb1           bus        EHCI Host Controller
/0/100/13                           bus        SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/13/1          usb5           bus        OHCI PCI host controller
/0/100/13.2                         bus        SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/100/13.2/1        usb2           bus        EHCI Host Controller
/0/100/14                           bus        SBx00 SMBus Controller
/0/100/14.1          scsi0          storage    SB7x0/SB8x0/SB9x0 IDE Controller
/0/100/14.1/0.1.0    /dev/sda       disk       32GB SSDSA2SH032G1GN
/0/100/14.1/0.1.0/1  /dev/sda1      volume     1023KiB BIOS Boot partition
/0/100/14.1/0.1.0/2  /dev/sda2      volume     511MiB Windows FAT volume
/0/100/14.1/0.1.0/3  /dev/sda3      volume     29GiB OS X ZFS partition or Solaris /usr partition
/0/100/14.3                         bridge     SB7x0/SB8x0/SB9x0 LPC host controller
/0/100/14.4                         bridge     SBx00 PCI to PCI Bridge
/0/100/16                           bus        SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/16/1          usb6           bus        OHCI PCI host controller
/0/100/16.2                         bus        SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/100/16.2/1        usb3           bus        EHCI Host Controller
/0/101                              bridge     Family 10h Processor HyperTransport Configuration
/0/102                              bridge     Family 10h Processor Address Map
/0/103                              bridge     Family 10h Processor DRAM Controller
/0/104                              bridge     Family 10h Processor Miscellaneous Control
/0/105                              bridge     Family 10h Processor Link Control
/0/1                                system     PnP device PNP0c02
/0/2                                system     PnP device PNP0b00
/0/4                                system     PnP device PNP0c02
/0/5                                system     PnP device PNP0c02
/0/6                                system     PnP device PNP0c02
/0/7                                system     PnP device PNP0c01
/1                   kube-bridge    network    Ethernet interface
/2                   veth17551355   network    Ethernet interface
/3                   vethf9e47a3b   network    Ethernet interface
/4                   veth1af1f01b   network    Ethernet interface
/5                   kube-dummy-if  network    Ethernet interface

Code:
admin@fjalar[~]$ ls /dev | grep "^i"
initctl
input
admin@fjalar[~]$ sudo ipmitool       
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
admin@fjalar[~]$ 



1690113980820.png
 

Bloty

Cadet
Joined
Jul 29, 2023
Messages
1
Having the same issue. Bump for visibility.

Every 5 minutes one entry each for "ipmi.sel.info" and "ipmi.sel.elist"
 

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
I noticed the same behavior today on both my TrueNAS SCALE boxes (recently updated to 22.12.3.3, if that matters). Both have ASRock Rack mobos with IPMI (one is E3C246D4M-4L and the other E3C246D2I). Full specs are in my sig.

Both have over 98% free space in the "IPMI Event Log" according to the IPMI Web interfaces, so I don't think the SEL is filling up.

Note that it's not the spammy tasks themselves that really bother me, but rather the fact they pushed tasks I actually care about out of the buffer (e.g., I can't even see this morning's replication and sync tasks).

Let me know if there's any other info I can supply to help debug this.
 
Last edited:

bcat

Explorer
Joined
Oct 20, 2022
Messages
84
Not 100% sure I'm doing this right (I'm still learning about IPMI), but it doesn't look like the SEL is full:

Code:
$ sudo ipmitool sel
[sudo] password for bcat:
SEL Information
Version          : 1.5 (v1.5, v2 compliant)
Entries          : 24
Free Space       : 65070 bytes
Percent Used     : 0%
Last Add Time    : 04/01/2023 02:19:42
Last Del Time    : 03/06/2023 02:04:31
Overflow         : false
Supported Cmds   : 'Delete' 'Partial Add' 'Reserve' 'Get Alloc Info'
# of Alloc Units : 3639
Alloc Unit Size  : 18
# Free Units     : 3615
Largest Free Blk : 3615
Max Record Size  : 1
 

bcat

Explorer
Joined
Oct 20, 2022
Messages
84

FrankWard

Explorer
Joined
Feb 13, 2023
Messages
71
bump because this just happened to my server.
 
Top