Timer Interrupt and Hard Reset?

[Glorious1]

Last night I received four emails from my IPMI with extremely vague information about Watchdog and a non-recoverable error like this:
message
IP : 192.168.0.3
Hostname: Tabernacle_IPMI
SEL_TIME:"2015/01/01 23:21:47"
SENSOR_NUMBER:"ca"
SENSOR_TYPE:"Watchdog 2 "
SENSOR_ID:" "
EVENT_DESCRIPTION:" "
EVENT_DIRECTION:"Assertion "
EVENT SEVERITY:"non-recoverable"

This morning I checked the Event Log in IPMI and found this. Two "Timer Interrupt" followed immediately each time by "Hard Reset". It sounds serious. What the heck is going on?

Code:

    
Select an event log category:
    Event Log: Max= 512, Used= 4 (event entries)
  Event ID         Time Stamp         Sensor Name         Sensor Type         Description 
1    2015/01/01 21:14:20    Watchdog 2 #0xca    Watchdog 2    Timer Interrupt - Asserted
2    2015/01/01 21:14:21    Watchdog 2 #0xca    Watchdog 2    Hard Reset - Asserted
3    2015/01/01 23:21:47    Watchdog 2 #0xca    Watchdog 2    Timer Interrupt - Asserted
4    2015/01/01 23:21:48    Watchdog 2 #0xca    Watchdog 2    Hard Reset - Asserted

 

[zambanini]

that is an error regarding the kernel driver and your bmc watchdog.

 

[Glorious1]

What/where log might shed more light on the cause? I'm rather boggled by all the logs in this thing scattered all over the place.

 

[zambanini]

you may create a bug entry, devs might know more.

https://forums.freenas.org/index.php?posts/166689/

 

[aaronouthier]

I am having a similar issue. My system is resetting after 4 minutes. The strange thing, is that my hardware watchdog is DISABLED in BIOS, and works great with Linux and Windows. This issue is preventing me from even installing FreeNAS, since my system resets during the install process.

 

[Glorious1]

Yes my Watchdog is disabled too. I don't even know what Watchdog is. And I'm not sure if this is fundamentally a hardware issue or OS related. But I did submit it.
https://bugs.freenas.org/issues/7374#change-32814

 

[bkvamme]

The watchdog is intended to automatically reset the system if it detects that the OS has frozen/crashed. If you have a Supermicro board, please check if you have a jumper on the motherboard for the watchdog, this might affect it.

 

[Glorious1]

Yes, thank you. According to the motherboard manual, the jumper is set to reset the system by default if the OS crashes. And I never changed that. However, there is a setting in BIOS that must be on too, and it is disabled by default. I haven't changed that either. But Watch Dog seems to be working anyway. From the manual:

Watch Dog Timer Enable

Watch Dog (JWD1) is a system monitor that can be used to reboot the system when a software application hangs. Close pins 1-2 to re-set the system if an application hangs. Close pins 2-3 to generate a non-maskable interrupt signal for the application that hangs. See the Open table on the right for jumper settings. Watch Dog must also be enabled in the BIOS.

 

[zambanini]

you should have added your board to the bug entry.

 

[aaronouthier]

Just for reference, my other thread on the topic:
https://forums.freenas.org/index.php?threads/freenas-installer-rebooting-after-4-minutes.26320/

 

[aaronouthier]

I have both a pdsmi+ and a pdsmi (original, non-plus) board. I had previously retired the regular pdsmi board, since it maxes out at a Pentium-D. I was first trying on a PDSMI+, but when this problem started to arise, I dug out the older board, and set it up in an old desktop case. I am having the same issue there.

Last thing to note, is that I have removed the IPMI daughterboard and tested again on both systems, with no change as to this issue.

 

[aaronouthier]

Seems to be a bug in FreeBSD itself:
http://freebsd.1045724.n5.nabble.com/Supermicro-BIOS-s-watchdog-feature-td4041350.html

.diff patches provided, but they are just crude Hacks. I would be happy to beta-test a patched binary. I may also try to compile the fix myself, IF I get some time.

 

[cyberjock]

That link is from 2010.. you sure that hasn't been incorporated already?

 

[aaronouthier]

Don't know, but the bug mentioned is still a problem. On which version of FreeBSD is FreeNAS 9.3 based? Maybe the fix has been made upstream, but FreeNAS hasn't caught up?

I am using another FreeBSD project called pfSense. As of this writing, the latest stable release is 2.1.5, which uses FreeBSD 8.2 as the underlying OS. The 2.2 version is in RC status, but goes forward by leaps and bounds, with 10.1 stable as the OS upon which it is based. I realize that FreeNAS is a different project than pfSense, but perhaps FreeNAS isn't based on the latest FreeBSD? I'm not sure. Perhaps you could clarify?

Edit: first sentence is a statement, not a question. Changed ? -> .

 

[Ericloewe]

FreeNAS matches the major FreeBSD version it's named after. So, 9.3 is based on FreeBSD 9.3.

 

[aaronouthier]

Ok, so I have been doing some poking around the 'Net. It appears to be an issue with the ichwd driver. Unfortunately, on FreeNAS, it appears to be compiled as part of the kernel, instead of as a module, as it is in mainstream FreeBSD, so I cannot simply copy over and replace the module. The module ichwd.ko appears in /boot/kernel on the FreeBSD 9.3 installer, but there is no such file on the FreeNAS installer.

I am inferring from this, that the kernels themselves are not identical, so I dare not try to simply replace the kernel itself either.

Curses, foiled again...

 

[cyberjock]

Well, 9.3 came out in October 2014 or so. It is the latest 9.3 branch, so it's not exactly "outdated".

I'd be surprised if the fix isn't incorporated already and you are having the same problem with a different cause.

Considering the age of your board (PDSMI+) I'm not surprised that its not working. That board was discontinued 5 or so years ago. The latest BIOS is dated 2008 (who knows when initial release was). But I'm not the least bit surprised that the board doesn't work flawlessly. I'm somewhat doubtful that the BIOS support gpt booting flawlessly and such (which is something that's only really been popular the last 3-4 years. Considering that board maxes out at 8GB of RAM and has a front side bus (Intel ditched FSBs in Nov 2008 with the Nahelem CPU) your hardware is definitely outdated and not really recommended for a FreeNAS server.

 

[aaronouthier]

I suppose I can't really assume that this is the same issue as the OP, but if it were, I would point out that his board is less than a year old. SuperMicro AS1Ri-2758F. It appears to be a similar issue, given that it is the same manufacturer. That said, if there is one thing I've learned over the years, it is that nothing is simple when it comes to the inner-workings of computers.

Edit: Removed redundant statement.

 

[JayG30]

Should note that I just saw this as well on a X10SL7-F running latest 9.3 release. Watchdog is OFF.

Assertion: Watchdog 2| Event = Timer interrupt
Assertion: Watchdog 2| Event = Hard Reset

This is a FreeBSD issue so obviously effects FreeNAS. I'm surprised more people haven't seen this (maybe they don't bother checking IPMI or setup emailing so don't know.

At this point I've seen far to many bugs in the latest 9.3 release running "recommended server grade hardware".
Supermicro 826E1 chassis
Supermicro X10SL7-F
E3-1231V3
2 x 8GB Crucial RAM (the recommended model)
Enterprise HDD's

Unfortunately I think I'm going to have to move to a Solaris derivative even if I like what FreeNAS offers.

 

[Bidule0hm]

Well, I have a X10SL7-F with watchdog disabled both in the BIOS and with the jumper, and the latest 9.3 update with email correctly configured and I check regularly the IPMI log.

I don't have these emails/events so you must have something not correctly configured somewhere ;)