IPMI Related kernel log error messages - Is this a sensible course of action?

[Ceetan]

I have received this message several times from kerenel log messages for the past month. Last time I had it, a power cycle solved the problem then but not so this time. It There does not seem to impact the function much, but I am a tad worried, mainly because there seem to be a message about problematic USB ports, and the only peripherial connected to the server is the UPS (Server/UPS interaction is something you want to have functioning, especially with the thunder storm coming up)



My plan is to update bios and ipmi firmware to see it it removes the thing.

1.Is that a sensible course of action?

2. Should this be done in an particular order?

Note that I have posted about this before but I kind of messed that post up. I hope I've made this one a bit clearer and posted slightly less output so that it's at least nominally readable.

Code:

> ipmi0: KCS Error Data ff
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd0 on uhub1
> ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 45> on usbus0
> kbd0 at ukbd0
> ums0 on uhub1
> ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 45> on usbus0
> ums0: 3 buttons and [Z] coordinates ID=0
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0 (disconnected)
> ipmi0: uhub1: at uhub0, port 12, addr 43 (disconnected)
> KCS: Reply address mismatch
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
> ukbd0: at uhub1, port 1, addr 45 (disconnected)
> ukbd0: detached
> ums0: at uhub1, port 1, addr 45 (disconnected)
> ums0: detached
> uhub1: detached
> ipmi0: KCS Error Data ff
> ipmi0: KCS Error Data ff
> usbd_setup_device_desc: getting device descriptor at addr 46 failed, USB_ERR_SHORT_XFER
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0
> uhub1 on uhub0
> uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 46> on usbus0
> uhub1: 4 ports with 3 removable, self powered
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd0 on uhub1
> ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 47> on usbus0
> kbd0 at ukbd0
> ums0 on uhub1
> ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 47> on usbus0
> ums0: 3 buttons and [XYZ] coordinates ID=0
> ipmi0: KCS: Error retry exhausted
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
> ukbd0: at uhub1, port 1, addr 47 (disconnected)
> ukbd0: detached
> ums0: at uhub1, port 1, addr 47 (disconnected)
> ums0: detached
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd0 on uhub1
> ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 48> on usbus0
> kbd0 at ukbd0
> ums0 on uhub1

 

[Ericloewe]

Is this on an X11SSL-CF?

Something is indeed wrong there. Could be the PCH's USB controller, could be the traces to the BMC or it could be the BMC itself. The IPMI log may have more useful information.

 

[Jailer]

I used to get something similar with my X10SL7-F that was caused by IPMI crashing. A BIOS and BMC update fixed it for me.

Can you log in to IPMI?

 

[Ceetan]

Is this on an X11SSL-CF?

Something is indeed wrong there. Could be the PCH's USB controller, could be the traces to the BMC or it could be the BMC itself. The IPMI log may have more useful information.

Thank you for your time @Ericloewe, @Jailer

This what is could pilfer from the BMC/IPMI eventlog, if that is what you meant
https://pastebin.com/bCFMPJWF

I suspect that you will find the more intresting bit at the end.

It is probably worth noting that, at at least one point yesterday, I strongly suspect the BMC crashed and reset itself, because I was momentairly unable to log in (which i have always been able to do, and am currently able to do.)

 

[Ericloewe]

Nothing too interesting, unless the host system did not reboot. The memory errors are common on X11 boards, at least with early firmware versions.

I'd suggest updating the BIOS and the BMC firmware and keeping an eye on things.

 

[Ceetan]

Nothing too interesting, unless the host system did not reboot. The memory errors are common on X11 boards, at least with early firmware versions.

I'd suggest updating the BIOS and the BMC firmware and keeping an eye on things.

I will do that then.

As far as I could tell, the server itself did not reboot when the BMC (most likely) reset itself. The memoory errors inthe log happened when I attempted a reboot. I revolved that issue by turning the thing of via IPMI, then back on again.

 

[JRD]

I have been getting a similar error message:

Code:

NAS.local kernel log messages:
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0 (disconnected)
> uhub1: at uhub0, port 9, addr 2 (disconnected)
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
> ukbd1: at uhub1, port 1, addr 3 (disconnected)
> ukbd1: detached
> ums1: at uhub1, port 1, addr 3 (disconnected)
> ums1: detached
> uhub1: detached
> ipmi0: KCS: Failed to start write
> ipmi0: KCS Error Data ff
> usbd_setup_device_desc: getting device descriptor at addr 6 failed, USB_ERR_SHORT_XFER
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0
> uhub1 on uhub0
> uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 6> on usbus0
> uhub1: 4 ports with 3 removable, self powered
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd1 on uhub1
> ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 7> on usbus0
> kbd2 at ukbd1
> ums1 on uhub1
> ums1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 7> on usbus0
> ums1: 3 buttons and [XYZ] coordinates ID=0
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
> ukbd1: at uhub1, port 1, addr 7 (disconnected)
> ukbd1: detached
> ums1: at uhub1, port 1, addr 7 (disconnected)
> ums1: detached
> ipmi0: KCS Error Data ff
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd1 on uhub1
> ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 8> on usbus0
> kbd2 at ukbd1
> ums1 on uhub1
> ums1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 8> on usbus0
> ums1: 3 buttons and [Z] coordinates ID=0
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0 (disconnected)
> uhub1: at uhub0, port 9, addr 6 (disconnected)
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
> ukbd1: at uhub1, port 1, addr 8 (disconnected)
> ukbd1: detached
> ums1: at uhub1, port 1, addr 8 (disconnected)
> ums1: detached
> uhub1: detached
> ipmi0: KCS: Failed to start write
> ipmi0: KCS Error Data ff
> usbd_setup_device_desc: getting device descriptor at addr 9 failed, USB_ERR_SHORT_XFER
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0
> uhub1 on uhub0
> uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 9> on usbus0
> uhub1: 4 ports with 3 removable, self powered
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd1 on uhub1
> ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 10> on usbus0
> kbd2 at ukbd1
> ums1 on uhub1
> ums1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 10> on usbus0
> ums1: 3 buttons and [XYZ] coordinates ID=0
> ipmi0: KCS Error Data ff
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
> ukbd1: at uhub1, port 1, addr 10 (disconnected)
> ukbd1: detached
> ums1: at uhub1, port 1, addr 10 (disconnected)
> ums1: detached
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd1 on uhub1
> ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 11> on usbus0
> kbd2 at ukbd1
> ums1 on uhub1
> ums1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 11> on usbus0
> ums1: 3 buttons and [Z] coordinates ID=0
> ipmi0: KCS: Error retry exhausted
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0 (disconnected)
> uhub1: at uhub0, port 9, addr 9 (disconnected)
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
> ukbd1: at uhub1, port 1, addr 11 (disconnected)
> ukbd1: detached
> ums1: at uhub1, port 1, addr 11 (disconnected)
> ums1: detached
> uhub1: detached
> ipmi0: KCS: Failed to start write
> ipmi0: KCS Error Data ff
> usbd_setup_device_desc: getting device descriptor at addr 12 failed, USB_ERR_SHORT_XFER
> ugen0.3: <vendor 0x0557 product 0x7000> at usbus0
> uhub1 on uhub0
> uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 12> on usbus0
> uhub1: 4 ports with 3 removable, self powered
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd1 on uhub1
> ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 13> on usbus0
> kbd2 at ukbd1
> ums1 on uhub1
> ums1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 13> on usbus0
> ums1: 3 buttons and [XYZ] coordinates ID=0
> ipmi0: KCS Error Data ff
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
> ukbd1: at uhub1, port 1, addr 13 (disconnected)
> ukbd1: detached
> ums1: at uhub1, port 1, addr 13 (disconnected)
> ums1: detached
> ugen0.4: <vendor 0x0557 product 0x2419> at usbus0
> ukbd1 on uhub1
> ukbd1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 14> on usbus0
> kbd2 at ukbd1
> ums1 on uhub1
> ums1: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 14> on usbus0
> ums1: 3 buttons and [Z] coordinates ID=0
> ipmi0: KCS: Error retry exhausted

-- End of security output --

I replaced the motherboard only a couple of weeks ago. First, there were error messages about one or two drives. After detaching and re-connecting the data cables at both ends all seemed fine. I removed the flag with zpool clear.

I then got an error similar to the above report.
I upgraded FreeNAS to 11.1-U7
Re-flashed the Bios at the latest version
Upgraded the BMC to 1.48

The error message came again, above earlier this morning. Can anyone help, please?
Thanks

 

[JRD]

Is there anybody out there?

 

[Redcoat]

Looking for straws to clutch at here. I assume you've searched this forum and Google and seen the reports of these and similar messages.

Do you have any USB devices besides your boot drive(s) connected to your NAS? If so, I woud try selectively removing/reconnecting them in an attempt to isolate an issue.

 

[JRD]

Looking for straws to clutch at here. I assume you've searched this forum and Google and seen the reports of these and similar messages.

Do you have any USB devices besides your boot drive(s) connected to your NAS? If so, I woud try selectively removing/reconnecting them in an attempt to isolate an issue.

This was the only thread I could find, close to my error message.

I tried the suggestions to fix here but with no success.

However, I remembered I have a wireless usb transmitter plugged in for a keyboard. I powered down the system, removed the transmitter and booted up. Over night there has been no error message. Can it really be as simple as that?!

 

[JRD]

The error has returned, even without usb transmitter for the keyboard connected:

Code:

;
Jun  5 00:00:00 NAS syslog-ng[1714]: Configuration reload request received, reloading configuration;
Jun  5 03:45:00 NAS ZFS: vdev state changed, pool_guid=10701889971907797377 vdev_guid=6383874058473158766
Jun  5 03:45:00 NAS ZFS: vdev state changed, pool_guid=10701889971907797377 vdev_guid=9082450593141577428
Jun  5 11:06:44 NAS ugen0.2: <vendor 0x0557 product 0x7000> at usbus0 (disconnected)
Jun  5 11:06:44 NAS uhub1: at uhub0, port 9, addr 1 (disconnected)
Jun  5 11:06:44 NAS ugen0.3: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
Jun  5 11:06:44 NAS ukbd0: at uhub1, port 1, addr 2 (disconnected)
Jun  5 11:06:44 NAS ukbd0: detached
Jun  5 11:06:44 NAS ums0: at uhub1, port 1, addr 2 (disconnected)
Jun  5 11:06:44 NAS ums0: detached
Jun  5 11:06:44 NAS uhub1: detached
Jun  5 11:06:53 NAS ipmi0: KCS: Failed to start write
Jun  5 11:07:05 NAS ipmi0: KCS Error Data ff
Jun  5 11:07:16 NAS usbd_setup_device_desc: getting device descriptor at addr 5 failed, USB_ERR_SHORT_XFER
Jun  5 11:07:16 NAS ugen0.2: <vendor 0x0557 product 0x7000> at usbus0
Jun  5 11:07:16 NAS uhub1 on uhub0
Jun  5 11:07:16 NAS uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 5> on usbus0
Jun  5 11:07:16 NAS uhub1: 4 ports with 3 removable, self powered
Jun  5 11:07:17 NAS ugen0.3: <vendor 0x0557 product 0x2419> at usbus0
Jun  5 11:07:17 NAS ukbd0 on uhub1
Jun  5 11:07:17 NAS ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 6> on usbus0
Jun  5 11:07:17 NAS kbd0 at ukbd0
Jun  5 11:07:17 NAS ums0 on uhub1
Jun  5 11:07:17 NAS ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 6> on usbus0
Jun  5 11:07:17 NAS ums0: 3 buttons and [XYZ] coordinates ID=0
Jun  5 11:07:23 NAS ipmi0: KCS Error Data ff
Jun  5 11:07:25 NAS ugen0.3: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
Jun  5 11:07:25 NAS ukbd0: at uhub1, port 1, addr 6 (disconnected)
Jun  5 11:07:25 NAS ukbd0: detached
Jun  5 11:07:25 NAS ums0: at uhub1, port 1, addr 6 (disconnected)
Jun  5 11:07:25 NAS ums0: detached
Jun  5 11:07:27 NAS ugen0.3: <vendor 0x0557 product 0x2419> at usbus0
Jun  5 11:07:27 NAS ukbd0 on uhub1
Jun  5 11:07:27 NAS ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 7> on usbus0
Jun  5 11:07:27 NAS kbd0 at ukbd0
Jun  5 11:07:27 NAS ums0 on uhub1
Jun  5 11:07:27 NAS ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 7> on usbus0
Jun  5 11:07:27 NAS ums0: 3 buttons and [Z] coordinates ID=0
Jun  5 11:07:29 NAS ipmi0: KCS: Error retry exhausted
Stop refresh

I don't know what this means. Do I need a new motherboard?

 

[JRD]

I moved the boot usb sticks to the usb2 ports, error came back
Disabled XHCI handoff in BIOS ditto
Added a Tunable; xhci_load Value YES as Loader ditto
Updated os to 11.2-U4.1 quite a few days...... then the error message has come back again.

Can anyone help, please?

 

[JRD]

Bump

 

[thepixelgeek]

I'm seeing this issue on 112.-U7. It appears to reset IPMI.

Running the most recent BIOS (9.524) and IPMI (2.08) firmware.

Code:

NAS uhub1: at uhub0, port 9, addr 1 (disconnected)
NAS ugen0.3: <vendor 0x0557 product 0x2419> at usbus0 (disconnected)
NAS ukbd0: at uhub1, port 1, addr 2 (disconnected)
NAS ukbd0: detached
NAS ums0: at uhub1, port 1, addr 2 (disconnected)
NAS ums0: detached
NAS uhub1: detached
NAS ipmi0: KCS: Failed to start write
NAS ipmi0: KCS Error Data ff