[HOWTO] Manage Automatic Daily Email Reports

Status
Not open for further replies.

lurker

Cadet
Joined
Mar 19, 2013
Messages
8
NOTE: THIS THREAD DOES NOT APPLY TO FREENAS 9+.


Hi Everyone,

If you have configured the email settings, you probably have noticed that you get daily emails about zpool status, security etc.
While it is good to know about these things I didn't want to get them unless there is an issue.

I looked through the forums and have found a few people in the same boat, but no real answers on what to do - so I decided to make a script to sort this out for me.
There is a scheduled task that runs at 3am daily which runs through the configured daily scripts. I changed some settings so that instead of just emailing the results out to root it will output the results to a temporary file which then has some checks applied to it. If any errors are found then it will email, otherwise it won't.

This should then cut down on the spam but still send out emails if an issue occurs.
For the people who still want daily emails so they can archive/know it is working ok, there is an option to send a replacement email daily with the subject line "OK: Daily Output Report" and "OK: Daily Security Report" which can then be filtered.

Note:The changes needed are not persistent with upgrades so will need to be re-applied if FreeNAS is upgraded.

Steps:
1.
Enable SSH through the Services Tab and then SSH onto the FreeNAS box as root.
2. Run the command: 'mount -uw /' (don't copy the " ' " characters) - this enables the root filesystem to be written to and persist changes after a reboot
3. Use your favourite text editor (I like nano) to edit a file: 'nano /conf/base/etc/defaults/periodic.conf'
Update line 33
From: daily_output="root"
To: daily_output="/tmp/output.txt"

Update line 163
From: daily_status_security_output="root"
To: daily_status_security_output="/tmp/security.txt"
4. Copy the following code into a new script file at a persistent location e.g. I have created a "Scripts" dataset on my main zpool and have the script called daily-checks.sh
Code:
#!/bin/bash

######################################################################
# Enter in the name of the daily report file
OUTPUT="/tmp/output.txt"
# Enter in the name of the daily security file
SECURITY="/tmp/security.txt"
######################################################################

######################################################################
# Check if the daily output report file has been created
if [ -e "$OUTPUT" ]; then

        # Basic checks to see if any errors found
        GROUP=`cat $OUTPUT | grep "/etc/group is fine" | wc -l`
        POOL=`cat $OUTPUT | grep "all pools are healthy" | wc -l`
        ALARM=`cat $OUTPUT | grep "No new alarms." | wc -l`
        COUNTOUT=`echo "$GROUP"+"$POOL"+"$ALARM" | bc`
        # If any error found then email daily output report to root's email address
        if [ "$COUNTOUT" -lt 3 ]; then
                cat "$OUTPUT" | mail -s "Error: Daily Output Report" root
        # Else if no errors are found do nothing - uncomment second line to send daily email all OK
        else
                echo "OK: Daily Output Report"
#               echo "" | mail -s "OK: Daily Output Report" root
        fi

        # Remove daily report file
        rm "$OUTPUT"
else
        echo "No daily $OUTPUT file found!" | mail -s "Error: No daily $OUTPUT file produced" root
fi
######################################################################

######################################################################
# Check if the daily security report file has been created
if [ -e "$SECURITY" ]; then

    # Basic checks to see if any errors found
    FILESYS=`cat $SECURITY | grep "changes in mounted filesystems:" | wc -l`
    if [ "`cat $SECURITY | grep "root 0" | wc -l`" -eq "1" ]; then
        ROOT=0
    else
        ROOT=1
    fi
    if [ "`grep --after-context=1 "Checking for passwordless accounts:" "$SECURITY" | grep -v "Checking for passwordless accounts:"`" == "" ]; then
        NOPASS=0
    else
        NOPASS=1
    fi
    if [ "`grep --after-context=1 "Checking login.conf permissions:" "$SECURITY" | grep -v "Checking login.conf permissions:"`" = "" ]; then
        LOGINCONF=0
    else
        LOGINCONF=1
    fi
    if [ "`grep --after-context=1 "Checking for ports with mismatched checksums:" "$SECURITY" | grep -v "Checking for ports with mismatched checksums:"`" = "" ]; then
        PORTS=0
    else
        PORTS=1
    fi
    KERN=`cat $SECURITY | grep "kernel log messages:" | wc -l`
    if [ "`grep --after-context=1 "login failures:" "$SECURITY" | grep -v "login failures:"`" = "" ]; then
        LOGINFAIL=0
    else
        LOGINFAIL=1
    fi
    if [ "`grep --after-context=1 "refused connections:" "$SECURITY" | grep -v "refused connections:"`" = "" ]; then
        REFCONNS=0
    else
        REFCONNS=1
    fi
        # If any error found then email daily security report to root's email address
    COUNTSEC=`echo "$FILESYS"+"$ROOT"+"$NOPASS"+"$LOGINCONF"+"$PORTS"+"$KERN"+"$LOGINFAIL"+"$REFCONNS" | bc`
        # If any error found then email daily security report to root's email address
        if [ "$COUNTSEC" -gt 0 ]; then
                cat "$SECURITY" | mail -s "Error: Daily Security Report" root
        # Else if no errors are found do nothing - uncomment second line to send daily email all OK
        else
                echo "OK: Daily Security Report"
#               echo "" | mail -s "OK: Daily Security Report" root
        fi

        # Remove daily report file
        rm "$SECURITY"
else
        echo "No daily $SECURITY file found!" | mail -s "Error: No daily $SECURITY file produced" root
fi
######################################################################

5. Run the command: 'chown root:wheel ./daily-check.sh; chmod 660 ./daily-check.sh' (don't copy the " ' " characters) - this just updates the permissions on the file
6. In the web interface navigate to System -> Cron Jobs -> Add Cron Job
User: root
Command: '/bin/bash <path to script>/daily-check.sh' - update <path to script> to your path
Minute -> Each Selected Minute Tab: 10 - the system scripts run at 3am so this will run 10 minutes after, update if you want
Hour -> Each Selected Hour Tab: 03 - only run at 3am
Day of Month -> Every N Day of Month Tab: 1 - everyday
Month: Check every month
Day of Week: Check every day
Redirect Stdout: Leave checked
Redirect Stderr: Leave unchecked
Enabled: Checked
7. Reboot the system to update the changes or manually do the updates in Step 3. to the file "/etc/defaults/periodic.conf"

Note: If you want to be emailed daily saying all is OK then just uncomment line 25 for the normal output checks and line 80 for the security checks.

Feel free to share any updates you find that helps and I hope this helps some people out there.
 
Last edited by a moderator:

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I'm going to bookmark this. This is actually a commonly asked question so I feel it should be a sticky. Good job OP! Welcome to the forums!
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
I think all HowTo's should be treated equally, but Cyberjock is right. For now I created a copy and made it a Sticky in the Noob section, and Un-Sticky'd this one. Thanks for a great first post!
 

lurker

Cadet
Joined
Mar 19, 2013
Messages
8
Glad it will be a help.
I did quite a bit of testing before posting, but if anyone finds any issues let me know and I can update.

Cheers
 

mirrors

Cadet
Joined
Jun 2, 2013
Messages
2
Very useful! Here's my own version of your script. I don't like getting multiple emails so I combined both logs into a single report that only gets mailed to root if any issues are found. I cleaned it up a bit which should make it easier to maintain.

Code:
#!/bin/bash

OUTPUT="/var/log/output-$(date +%m%d%y).log"
SECURITY="/var/log/security-$(date +%m%d%y).log"

if [ -e "$OUTPUT" -a -e "$SECURITY" ]; then
  GROUP="`cat "$OUTPUT" | grep "/etc/group is fine" | wc -l`" #1
  POOLS="`cat "$OUTPUT" | grep "all pools are healthy" | wc -l`" #1
  ALARMS="`cat "$OUTPUT" | grep "No new alarms" | wc -l`" #1
  MOUNTS="`cat "$SECURITY" | grep "changes in mounted filesystems" | wc -l`" #0
  UIDS="`cat "$SECURITY" | grep "root 0" | wc -l`" #1
  [ -z "`grep -A 1 "passwordless accounts" "$SECURITY" | grep -v "passwordless accounts"`" ] && ACCOUNTS="0" || ACCOUNTS="1" #0
  [ -z "`grep -A 1 "login.conf permissions" "$SECURITY" | grep -v "login.conf permissions"`" ] && PERMS="0" || PERMS="1" #0
  [ -z "`grep -A 1 "ports with mismatched checksums" "$SECURITY" | grep -v "ports with mismatched checksums"`" ] && PORTS="0" || PORTS="1" #0
  MESSAGES="`cat $SECURITY | grep "kernel log messages" | wc -l`" #0
  [ -z "`grep -A 1 "login failures" "$SECURITY" | grep -v "login failures"`" ] && FAILURES="0" || FAILURES="1" #0
  [ -z "`grep -A 1 "refused connections" "$SECURITY" | grep -v "refused connections"`" ] && REFUSED="0" || REFUSED="1" #0
else
  echo "Output files(s) missing!" | /usr/bin/mail -s "FreeNAS Daily Run - Error!" "root"
  exit 1
fi

COUNT="`echo "$GROUP" + "$POOLS" + "$ALARMS" + "$MOUNTS" + "$UIDS" + "$ACCOUNTS" + "$PERMS" + "$PORTS" + "$MESSAGES" + "$FAILURES" + "$REFUSED" | bc`"
if [ "$COUNT" -ne "4" ]; then
  REPORT="`sed -e '/Security check/,+2d' "$OUTPUT" && cat "$SECURITY"`"
  echo "$REPORT" | /usr/bin/mail -s "FreeNAS Daily Run - Error!" "root"
  exit 1
fi

exit 0
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
@OP, Nice work. Could come in handy. And that was very professional of you to verify your work before posting it.
 

tanik1

Contributor
Joined
Mar 31, 2013
Messages
163
hello lurker, weird to say that. But I used your scripts but I am getting this error email but I don't know what the error is about maybe you or someone could help me. I only get that email, so it seems that there is no error with security report thank you.

subject: Error: Daily Output Report

Body:
Removing stale files from /var/preserve:

Cleaning out old system announcements:

Backup passwd and group files:
no /var/backups/master.passwd.bak
no /var/backups/group.bak

Verifying group file syntax:
/etc/group is fine

Backing up mail aliases:
no /var/backups/aliases.bak

Backing up package db directory:

Disk status:
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/FreeNASs1a 926M 573M 278M 67% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/md0 4.6M 3.8M 391k 91% /etc
/dev/md1 823k 3.0k 755k 0% /mnt
/dev/md2 149M 23M 114M 17% /var
/dev/ufs/FreeNASs4 19M 1.4M 16M 8% /data
Vol1 9.7T 405G 9.3T 4% /mnt/Vol1
Vol1/Family 9.4T 28G 9.3T 0% /mnt/Vol1/Family
Vol1/Media 10T 626G 9.3T 6% /mnt/Vol1/Media
/dev/ufs/temp 359G 14G 315G 4% /mnt/temp
/dev/ufs/Jail 224G 1.5G 205G 1% /mnt/Jail
devfs 1.0k 1.0k 0B 100% /mnt/Jail/bit_1/dev
procfs 4.0k 4.0k 0B 100% /mnt/Jail/bit_1/proc
/mnt/temp/downloads 359G 14G 315G 4% /mnt/Jail/bit_1/usr/downloads

Checking status of zfs pools:
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
Vol1 14.5T 1.43T 13.1T 9% 1.00x ONLINE /mnt

all pools are healthy

Checking status of ATA raid partitions:

Checking status of gmirror(8) devices:

Checking status of graid3(8) devices:

Checking status of gstripe(8) devices:

Network interface status:
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop
usbus 0 <Link#1> 0 0 0 0 0 0 0
usbus 0 <Link#2> 0 0 0 0 0 0 0
usbus 0 <Link#3> 0 0 0 0 0 0 0
usbus 0 <Link#4> 0 0 0 0 0 0 0
usbus 0 <Link#5> 0 0 0 0 0 0 0
usbus 0 <Link#6> 0 0 0 0 0 0 0
usbus 0 <Link#7> 0 0 0 0 0 0 0
re0 1500 <Link#8> b8:97:5a:3b:02:f9 449200 0 0 1337231 0 0 0
re0 1500 192.168.1.0 192.168.1.4 229038 - - 1176370 - - -
re0 1500 fe80::ba97:5a fe80::ba97:5aff:f 0 - - 17 - - -
ipfw0 65536 <Link#9> 0 0 0 0 0 0 0
lo0 16384 <Link#10> 86 0 0 86 0 0 0
lo0 16384 localhost ::1 0 - - 0 - - -
lo0 16384 fe80::1%lo0 fe80::1 0 - - 0 - - -
lo0 16384 your-net localhost 86 - - 86 - - -
bridg 1500 <Link#11> 02:fe:4a:c8:9c:00 370127 0 0 1547042 0 0 0
epair 1500 <Link#12> 02:b7:4f:00:0c:0a 160642 0 0 209876 1 0 0
epair 1500 fe80::b7:4fff fe80::b7:4fff:fe0 0 - - 4 - - -

Security check:
(output logged separately)

Checking status of 3ware RAID controllers:
Alarms (most recent first):
+++ /var/log/3ware_raid_alarms.today 2013-08-15 03:01:00.000000000 -0700
@@ -0,0 +1 @@
+

-- End of daily output --
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
@Tanik1

Does this look like a problem?


Code:
Checking status of 3ware RAID controllers:
Alarms (most recent first):
+++ /var/log/3ware_raid_alarms.today 2013-08-15 03:01:00.000000000 -0700
@@ -0,0 +1 @@
 

tanik1

Contributor
Joined
Mar 31, 2013
Messages
163
@Tanik1

Does this look like a problem?


Code:
Checking status of 3ware RAID controllers:
Alarms (most recent first):
+++ /var/log/3ware_raid_alarms.today 2013-08-15 03:01:00.000000000 -0700
@@ -0,0 +1 @@


hmm yes it does look like that can be an error. but i don't have a 3ware raid controller. my raid controller is a high point brand. or is this checking any controller but it outputs saying the 3ware?
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
Sorry, I don't know the details of how this error message is generated, it just looks like FreeNAS is telling you that it sees an error. I think you should investigate this further to determine if this is a real problem or not. You can adjust the script to ignore the problem, provided you are positive you want to do that.

Take a look at the log file this message is pointing to, see what it says. That might help out.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970

tanik1

Contributor
Joined
Mar 31, 2013
Messages
163
Ok, thank you.

I will need to check if my problem will fix itself. And if there isn't really an error I could also adjust the script like you said. I actually have my NAS off for awhile due to something is wrong with my battery backup. Might need to get a new one. :(
 

diedrichg

Wizard
Joined
Dec 4, 2012
Messages
1,319
noob errors that I made:
  • The OP named his file daily-check.sh but then gives the instructions (chown root:wheel ./daily-check.sh; chmod 660 ./daily-check.sh), just make sure your file name matches exactly. daily-check.sh vs daily-checks.sh
  • When you copy the long code in step 4 make sure you paste it into a text editor first and then copy the script from the text editor because if you don't; when you paste it into your ssh command line, formatting issues will occur and you'll get a few lines that wrap to the next line and the system takes that literally as two lines rather than a wrapped line.
 

Super Paul

Cadet
Joined
Aug 20, 2013
Messages
6
I realize this is an older post but I would love to achieve this functionality.

I've been trying to accomplish it in 9.1.1, but the files appear not to be created. The result is that I get a "file not found" error email for each of the reports. Has anyone got this working in 9.1.1 and can share the steps they took, or maybe comment on anything I can check to fix my issue?

I have double checked the script itself, and there appear to be no syntax errors. I never attempted this in 8.3, so I don't have an "it used to work" point of reference.

Thanks in advance.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,970
There are two scripts listed on this page, please post a copy of your script just so we can take a look at it. I don't have this script running yet, I haven't had 9.1.1 installed long enough to where I want to cut down the emails yet.
 

Super Paul

Cadet
Joined
Aug 20, 2013
Messages
6
@joeshmuck,

Sorry, I probably should have specified. I'm using the first script posted on this thread. The second, I noticed, makes reference to dated file names, but the author doesn't specify how those files would be created (and named). I believe the script is running correctly because I get an with the contents of the error message in the script. ie:

echo "No daily $OUTPUT file found!" | mail -s "Error: No daily $OUTPUT file produced" root

So, my assumption is that the system is not outputting the contents of the daily reports to the file (as specified by changing values in the periodic.conf file). Thus, since the two files are never created, the script can't find them, and errors. I have double and triple checked my periodic.conf (both), and restarted the system, so I believe that part to be correct.

Edit:

I think I'm an idiot. I just noticed he mentions in the OP that the "mount -uw" is intended to persist through reboots. I've been remounting the root FS read only after my changes, so I'm going to take a wild guess that it's not able to write the files necessary to get emailed via the scheduled job. I've changed the location of the files written to my storage area, and I'll know tonight if it works.
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
I think there is actually much easier way to accomplish this. Just add these lines to /etc/periodic.conf (or/conf/base/etc/periodic.conf to make it permanent):
daily_show_success="NO"
daily_show_info="NO"
security_show_success="NO"

Quoting the periodic documentation:

Each script is required to exit with one of the following values:

0 The script has produced nothing notable in its output. The
<basedir>_show_success variable controls the masking of this out-
put.

1 The script has produced some notable information in its output.
The <basedir>_show_info variable controls the masking of this out-
put.

2 The script has produced some warnings due to invalid configuration
settings. The <basedir>_show_badconfig variable controls the mask-
ing of this output.

>2 The script has produced output that must not be masked.
It works perfectly for me. I get no daily emails, but I do get email if something happens (e.g. if I plug/unplug an USB device I get the security mail showing the kernel messages).
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
The daily_show_info="NO" is only needed to mask the output of the 400.status-disks script (output of df). You can replace daily_show_info="NO" with daily_status_disks_enable="NO" to block just that one script and keep any possible info outputs enabled for other scripts.
 

Super Paul

Cadet
Joined
Aug 20, 2013
Messages
6
Wanted to chime in to verify what Dusan has posted. That method is (in my opinion) far easier, and works exactly as expected, as far as I can tell. I never got the scripts to correctly generate via the previous method, despite checking permissions 50 times. I suppose it's possible that it behaves differently in 9.1.1, but either way, that's neither here nor there since Dusan's suggestions are far simpler and do the same job.

Thanks for figuring this out.
 

mirrors

Cadet
Joined
Jun 2, 2013
Messages
2
hmm yes it does look like that can be an error. but i don't have a 3ware raid controller. my raid controller is a high point brand. or is this checking any controller but it outputs saying the 3ware?

Change /conf/base/etc/periodic.conf to: daily_status_3ware_raid_enable="NO"
 
Status
Not open for further replies.
Top