11.1 Message log now flooded when system is in warning status

[MasterTacoChief]

I upgraded from 11.0 on a test system which has a disconnected volume (on purpose). The message log is now flooded with the same message repeatedly:

Dec 14 13:05:24 nas2 daemon[2903]: 2017/12/14 13:05:24 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:07:30 nas2 daemon[2903]: 2017/12/14 13:07:30 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:09:36 nas2 daemon[2903]: 2017/12/14 13:09:36 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:11:41 nas2 daemon[2903]: 2017/12/14 13:11:41 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:13:47 nas2 daemon[2903]: 2017/12/14 13:13:47 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:15:52 nas2 daemon[2903]: 2017/12/14 13:15:52 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:17:58 nas2 daemon[2903]: 2017/12/14 13:17:58 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:20:04 nas2 daemon[2903]: 2017/12/14 13:20:04 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:22:09 nas2 daemon[2903]: 2017/12/14 13:22:09 [WARN] agent: Check 'service:nas-health' is now warning
Dec 14 13:24:15 nas2 daemon[2903]: 2017/12/14 13:24:15 [WARN] agent: Check 'service:nas-health' is now warning

I don't know if it's just a disconnected volume, or other warnings would result in this same behavior. I'd expect it to appear once, not repeatedly which makes it hard to find other messages in the log.

 

[dlavigne]

This has been reported and should be fixed for next update: https://redmine.ixsystems.com/issues/26355.

 

[fmdx]

Is there anything that can be done to stem the flood of spam until the next update (I'm assuming possibly months away)?

 

[dlavigne]

I'm not sure what the fix/workaround will be (it will be in the ticket once known), but the next update is definitely not months away (more like a few weeks).

 

[millst]

Just update freenas.json per the ticket.

The spamming stopped on its own on my system. Perhaps, because I upgraded my pool and cleared out an ignored alert.

-tm

 

[DiWa]

Hi
I'm just facing the same issue since my 11.0-U4 -> 11.1 upgrade.
I've updated my /usr/local/etc/consul.d/freenas.json accordingly but the warning is still there.

Is there a service to bounce or something else to do as obviously restarting Freenas just loose the updated file.

Cheers

 

[millst]

Try restarting consul?

-tm

 

[DiWa]

unfortunately even after the file update +
service consul stop
followed by
service consul start
The warning is still fired every 2 minutes.

So I just fired the service consul stop once more to make it quiet.
It should do the job until a proper release is out.

 

[nojohnny101]

I am dealing with this as well. I just stopped the consul as @DiWa indicated.

 

[jg3]

I updated my /usr/local/etc/consul.d/freenas.json according to the link from DiWa above, but now I get this when restarting consul:

Code:

root@FreeNAS:~ # service consul restart
Config validation failed: data_dir cannot be empty

Any ideas where this is coming from? I looked in the .sh that consul checks (freenas_health.sh) and the .py that it is a wrapper for (alertcli.py) to see if I could find a data_dir but haven't found it.

 

[jg3]

Also, if I run alertcli.py i get a WARN message.

Code:

root@fn:~ # /usr/local/bin/alertcli.py
WARN - ActiveDirectory did not bind to the domain

should I see this warning in my syslog, or is it supposed to show me the generic message "[WARN] agent: Check 'freenas_health' is now warning" ?

 

[paradoxiom]

Is there anything that can be done to stem the flood of spam until the next update (I'm assuming possibly months away)?

Ironically I just updated last night, never saw this error before and my log is flooded with it this morning after updating.

Code:

Jan 20 09:51:38 OrbitalHub daemon[3675]:	 2018/01/20 09:51:38 [ERR] agent: failed to sync remote state: No cluster leader
Jan 20 09:51:39 OrbitalHub daemon[3675]:	 2018/01/20 09:51:39 [WARN] raft: Heartbeat timeout from "" reached, starting election
Jan 20 09:52:37 OrbitalHub daemon[3675]:	 2018/01/20 09:52:37 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 09:54:37 OrbitalHub daemon[3675]:	 2018/01/20 09:54:37 [WARN] agent: Check 'freenas_health' is now warning
<LOADS MORE>
Jan 20 12:20:56 OrbitalHub daemon[3675]:	 2018/01/20 12:20:56 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:22:56 OrbitalHub daemon[3675]:	 2018/01/20 12:22:56 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:24:57 OrbitalHub daemon[3675]:	 2018/01/20 12:24:57 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:26:57 OrbitalHub daemon[3675]:	 2018/01/20 12:26:57 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:28:57 OrbitalHub daemon[3675]:	 2018/01/20 12:28:57 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:30:57 OrbitalHub daemon[3675]:	 2018/01/20 12:30:57 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:32:58 OrbitalHub daemon[3675]:	 2018/01/20 12:32:58 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:34:58 OrbitalHub daemon[3675]:	 2018/01/20 12:34:58 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:36:58 OrbitalHub daemon[3675]:	 2018/01/20 12:36:58 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:38:58 OrbitalHub daemon[3675]:	 2018/01/20 12:38:58 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:40:59 OrbitalHub daemon[3675]:	 2018/01/20 12:40:59 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:42:59 OrbitalHub daemon[3675]:	 2018/01/20 12:42:59 [WARN] agent: Check 'freenas_health' is now warning
Jan 20 12:44:59 OrbitalHub daemon[3675]:	 2018/01/20 12:44:59 [WARN] agent: Check 'freenas_health' is now warning

>All files verified successfully!

Current Train: FreeNAS-11-STABLE (...)

Code:

Build	FreeNAS-11.1-U1
Platform	Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
Memory	16311MB
System Time	Sat, 20 Jan 2018 12:48:31 +0000
Uptime	12:48PM up 2:59, 0 users
Load Average	0.82, 1.99, 2.15

is it safe to ignore or is my system imploding somewhere that i'm now only just being warned about after updating?

 

[Redcoat]

Safe to ignore, it's a yet-unresolved bug - see https://forums.freenas.org/index.php?threads/a-new-message-with-11-1-rc1.58844

 

[toolchain]

there was a fix for this issue that was merged recently, I found this solution to be a way to just pull the updated file down to use it to replace the existing freenas_health.sh file that has an error in the code which causes it to spam the same warning message

here's that commit on github
https://github.com/freenas/freenas/pull/723/files

in a root shell on freenas or via SSH as root on freenas, these commands get the job done

Code:

cd /usr/local/etc/consul-checks/
rm freenas*
wget https://raw.githubusercontent.com/freenas/freenas/8175ddd35c189c43c47a51e42a1353e96d9ac2cb/src/freenas/usr/local/etc/consul-checks/freenas_health.sh
chmod a+x freenas_health.sh

 

[nojohnny101]

Thanks @toolchain

This temporary fix survives a reboot?

 

[trentk10]

Thanks for this :)

 

[fmdx]

I haven't done the fix suggested above, but I did update to the 11.1-u2 update and the console spam is still present at this time.

 

[Redcoat]

The fix is now slated for 11.2.