Spencer V2.0-BETA 1- Now with Pineapple - An Email Alert Script for Potentially Hidden Problems

[NickF]

NickF submitted a new resource:

Spencer - An Email Alert Script for Potentially Hidden Problems - A Companion for Multi-Report by joeschmuck

View attachment 68056
Spencer v1.1
Not Quite Psychic...​

This script was written to identify issues which may otherwise be silent, as they are not...

Read more about this resource...

 

[joeschmuck]

I like this script although I have no clue how to code in Python. I wanted to use Python last year to make the Multi-Report script more portable but I just couldn't wrap my head around the structure so I gave up.

So your script only sends an email if an error message exists in the 'messages' file.

I have some feedback for you and I will send you a message to keep it out of this discussion thread. Nice work @NickF

 

[NickF]

I like this script although I have no clue how to code in Python. I wanted to use Python last year to make the Multi-Report script more portable but I just couldn't wrap my head around the structure so I gave up.

So your script only sends an email if an error message exists in the 'messages' file.

I have some feedback for you and I will send you a message to keep it out of this discussion thread. Nice work @NickF

I tried to take a look at your script on GitHub. Originally I was going to fork your script on GitHub and just add this in. Then I saw how many lines it was…

Feedback is always welcomed!

 

[NickF]

NickF updated Spencer - An Email Alert Script for Potentially Hidden Problems with a new update entry:

Feature Update v1.2

V.1.2 includes a change in order to properly handle and report errors which had already been reported.

Read the rest of this update entry...

 

[NickF]

NickF updated Spencer - An Email Alert Script for Potentially Hidden Problems with a new update entry:

V1.3 - Credit to JoeSchmuck

# Spencer is a basic Python script which is designed to run in conjunction with "multi_report" by Joe Schmuck
# https://github.com/JoeSchmuck/Multi-Report
# Spencer checks for additional errors which may appear in your logs that you otherwise may be unaware of.
# The initial version of this script is versioned v1.1 and was written by ChatGPT and NickF
# Verion 1.3 08/13/23
# Added a new feature for tracking and reporting previous errors differantly than new errors.

# This updated...

Read the rest of this update entry...

 

[joeschmuck]

I will post the new version of Multi-Report version 2.4.4 which integrates this script on 20 August. For now, spencer.py can run as a normal script.

 

[NickF]

No rush sir! Please don't worry on my regard. Get your other changes merged and done :)

 

[joeschmuck]

Multi-Report version 2.4.4 has been posted and supports Spencer. Please place the file 'spencer.py' in the same directory as 'multi_report.sh'. By default if spencer.py is detected, the script will be run. You do not need to configure 'spencer.py' at all when used in this manner.

Thanks @NickF for the script. I'm glad you saw a problem and provided a solution. Team Work!

 

[NickF]

NickF updated Spencer - An Email Alert Script for Potentially Hidden Problems with a new update entry:

v2.0 - BETA -1

Code:

# Spencer is a basic Python script which is designed to run in conjunction with "multi_report" by Joe Schmuck
# https://github.com/JoeSchmuck/Multi-Report
# Spencer checks for additional errors which may appear in your logs that you otherwise may be unaware of.
# The initial version of this script is versioned v1.1 and was written by ChatGPT and NickF
# -----------------------------------------------
# Version 2.0 08/22/23
# Refactored alot of code.
# Dynamically determine whether run...

Read the rest of this update entry...

 

[joeschmuck]

@NickF WOW! That was a huge rewrite! A lot of changes. Nice work!

 

[NickF]

Mostly still ChatGPT more than me. But I've started using https://refact.ai/ in Visual Studio Code to help me out alot too.
I'm trying to learn Python from ChatGPT while also solving problems...and I am introducing more tools into my workflow to help me become a better programmer.

At this stage in my development, I feel like I'm playing the Job's to "AI"'s Wozniak but way crappier than Apple. Pretty sure I will ever measure up to either lmao.

 

[jafin]

@NickF Do you have a github repo for this script?

Had the below issue with the 2.0 Beta script:

If i run python -m chardet.cli.chardetect /var/log/messages it response
msg: ISO-8859-1 with confidence 0.73

I believe I am running a standard install of truenas scale. (i.e. I don't believe I've adjusted file formats for log messages) The file may suppose to be UTF-8.
On a second Truenas core machine it reports back as 'ascii' , so TBH not sure what the default format of this file is. Anyhow forcing to ignore errors gets it going with the script ofr me.

The below change will resolve reading the log for 'invalid' content in the file encoding:

Code:

367: with open('/var/log/messages', 'r', encoding='UTF-8', errors='ignore') as file:

Or even removing encoding='UTF-8' and just set errors='ignore' Given I am not 100% confident the file is suppose to always be utf-8 encoded.


Error produced from script:

Code:

2023-09-11 08:09:37.764044 - Operating System version determined: TrueNAS SCALE, Version: 22.12.3.3 - Debian GNU/Linux 11 (bullseye) (TrueNAS SCALE)
Traceback (most recent call last):
  File "/mnt/vol1/appdata/scripts/spencer.py", line 496, in <module>
    matches, match_counts, repeat_errors = search_log_file(os_version)
  File "/mnt/vol1/appdata/scripts/spencer.py", line 368, in search_log_file
    for i, line in enumerate(file):
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd2 in position 3062: invalid continuation byte

 

[joeschmuck]

If i run python -m chardet.cli.chardetect /var/log/messages it response
msg: ISO-8859-1 with confidence 0.73

I get the same thing. When I run the script, no errors on Scale. Curious, can you explain where the file is located (I have an idea from your posting) and exactly how you run the script? Via CRONTAB, from the CLI? And the exact command/setup?

Please be descriptive and do not assume we understand what you are saying, treat us as not the sharpest tool in the shed. It helps to stop assumptions. Basically it would be nice to reproduce your situation to generate the failure. Also, how did you create the spencer.py file? Sometimes text editors can leave garbage behind. It happens.

I assume this is repeatable?

I'm sure @NickF will chime in.

 

[jafin]

@joeschmuck I don't believe it matters where it is run from, it's a parsing issue with python reading the file.

But for completeness, run from a vol /mnt/vol1/appdata/scripts. via a terminal.. It doesn't change if run from crontab.

Repeatable: Yes.

Run beta script: FAIL
Run script with mod suggested above: PASS

If I look at the contents of my /var/log/messages there are some characters in there that may be UTF-8 or not, running `python -m chardet.cli.chardetect /var/log/messages` tells you what python THINKS the file is.

I can also force it to 'ISO-8859-1' and it will read without error. I just don't know enough about truenas to determine what is the default encoding, for /var/log/messages hence the Errors='ignore' feels like the more correct choice.

 

[NickF]

I do have a GitHub setup. I haven’t encounter non utf8 messages myself but I have absolutely no objection to adding error handling to more gracefully handle them if they ooccur.

If you have a solution already feel free to issue a PR https://github.com/SCNickF1227/spencer

 

[joeschmuck]

running `python -m chardet.cli.chardetect /var/log/messages` tells you what python THINKS the file is.

I'm still new to Python so I'm fine learning something here. The command you entered is to check the '/var/log/messages' file, correct? The file TrueNAS creates, not Spencer.

The 'UTF-8' sounds familiar, I know I had to add it into my Multi-Report script to fix language differences that would cause a big fail.

I haven’t encounter non utf8 messages myself

I think someone using a different language may cause an issue, but maybe ignoring the errors helps solve that potential problem. I would recommend removing the UTF-8 portion of the code and then wait and see if someone complains. Maybe none complain.

 

[jafin]

I'm still new to Python so I'm fine learning something here. The command you entered is to check the '/var/log/messages' file, correct? The file TrueNAS creates, not Spencer.

Yes correct. I have 2 truenas here, on one it reported as 'ASCII' the other as ISO-8859-1 (when issuing the command python -m chardet.cli.chardetect /var/log/messages) . I tried to search what the correct encoding SHOULD BE, but couldn't find anything definitive.