Nightly check of FreeNAS database!

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Note: This is a repost since I accidentally deleted the thread instead of a few posts.. whoops!

So I've noticed a lot of people having corruption problems with their FreeNAS configuration file as of late. And It gave me an idea. Since it's simply an sql database, why not do an integrity check as part of a daily routine? Who wants to use a corrupted configuration file for days or weeks only to learn their backups are corrupt. So here's how you do it!

Edit: If you want to run the command locally or from ssh, the command is (and the quotations are supposed to be there!)
sqlite3 /data/freenas-v1.db "pragma integrity_check"​
If it returns "ok" then all is good.

NOTE: You can even run this command against backup copies of your database file!



1. Create a .sh file, call it whatever you want, and put it wherever you want(preferably on your zpool in a safe place). Make sure it has execute permissions for root!!!
2. Create a cronjob that points to the file:

User: root
Command : /whereveryouputthescript/scriptname.sh
Short Description: Daily database check
All of the date/time options: However you want to set them. I do nightly at 2330.
Redirect Stdout: Checked
Redirect Stderr: Checked
Enabled: What do you think? If you don't know, stop using FreeNAS right now...

Here's the script:
Code:
#!/bin/sh
# This file runs a database integrity check and emails you in the event that
# your database is corrupt. This file requires you to have *properly* setup
# emailing from the FreeNAS GUI.  The following variables are available for
# you to use:
#
# YourEmail:  The email address you want to send the email to if your database
# is found to be corrupt.  Multiple email addresses are supported if separated
# by a space.
YourEmail="youremail@server.com"
# ServerName:  How you want your server's name to appear in the email in the
# event that the database is corrupt.
ServerName=YourServerNameHere
# TempLocation:  Location for the temp file for the email.  Default is /tmp
TempLocation=/tmp
# Scroll down to edit the email as you see fit.  The default setup is recommended
# since it works and conveys a simple email to let you know what the problem
# is and that you need to take action.
# If you want to test this to ensure it works, simply rename the line with
# freenas-v1.db to pointto a file that doesn't exist.  It will error, and you will get
# an email.
if [ -f "$TempLocation"/badconfig.txt ];
    then rm "$TempLocation"/badconfig.txt
fi
 
if [ "$( sqlite3 /data/freenas-v1.db "pragma integrity_check;" )" == "ok" ]; then
        #echo "Database is ok."
        #bail out with zero (all o.k.) status
        exit 0
    else
        # The following lines are the email that will be sent in the event that errors
        # are found.  Change it however you wish, just make sure the general format
        # is protected.
        echo "To: $YourEmail" >> $TempLocation/badconfig.txt
        echo "Subject: ERROR: Database corrupt on server $ServerName" >> $TempLocation/badconfig.txt
        echo "Your server, $ServerName, has been found to have a corrupt FreeNAS config." >> $TempLocation/badconfig.txt
        echo " " >> $TempLocation/badconfig.txt
        echo "It is recommended you troubleshoot and correct the problem as soon as possible.  Just because the server is operating fine does not mean you can ignore this message." >> $TempLocation/badconfig.txt
        echo "$TempLocation"/badconfig.txt
        sendmail -t < "$TempLocation"/badconfig.txt
        rm "$TempLocation"/badconfig.txt
    fi
exit 1
 


If you like this idea, don't forget to read my thread on how to backup your config file to your pool automatically every night here.

If you want to do backups AND config checks, you can change the cronjob to instead be something like: /mnt/tank/chkconfig.sh && /mnt/tank/bkpconfig.sh

With that command, if the chkconfig fails then the backup doesn't happen.(Thanks Neil Whitworth for the idea!)

Good luck and happy NASing!
 

dwhacks

Dabbler
Joined
Mar 27, 2014
Messages
18
If you want to do backups AND config checks, you can change the cronjob to instead be something like: /mnt/tank/chkconfig.sh && /mnt/tank/bkpconfig.sh

Do we have to add sh in front of the command like you stated in the backupconfig thread?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Nope. You set it as executable.
 

erturne

Dabbler
Joined
Sep 5, 2013
Messages
19
I created a script called /mnt/nas/diffconfig.sh that returns 0 if a backup doesn't exist, or if the most recent backup is different than the current freenas config. That way I only create a backup when the config has changed.

Note my volume is mounted under /mnt/nas so you'll want to change that if you decide to use this.

Code:
#!/bin/sh
 
CFGFILE=/data/freenas-v1.db
 
# Find the latest backup.
BKPFILE=""
BKPFILES=$(ls /mnt/nas/*.db 2> /dev/null | wc -l)
if [ "$BKPFILES" != "0" ]
then
  BKPFILE=`ls -t /mnt/nas/*.db | head -1`
fi
 
# Bail out with zero (all o.k.) if a backup file does not exist
# or if the current configuration is different than the most
# recent backup.
if [ -z "$BKPFILE" ]
then
  exit 0
else
  DIFF=`diff $BKPFILE $CFGFILE`
  if [ ! -z "$DIFF" ]
  then
      exit 0
  fi
fi
 
exit 1


Then my cron looks like: /mnt/nas/diffconfig.sh && /mnt/nas/chkconfig.sh && /mnt/nas/bkpconfig.sh
 

mute

Dabbler
Joined
Dec 8, 2013
Messages
19
Food for thought. A friend of mine who introduced me to FreeNAS wrote a script (that I modified) that copies the running FreeNAS database (if it has changed) to a config dataset that's snapshotted. Makes it pretty easy to restore from a backup in the event of corruption or if you really screwed up your settings. :)
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
That's basically what I'm doing, but without a script to check if anything has changed. It's just an rsync task.
 

dwhacks

Dabbler
Joined
Mar 27, 2014
Messages
18
Im doing this with back ups nightly, but even if my database is ok, I get emails, why is that?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
You tell us? You must have done something wrong. Not sure what, but something.
 

dwhacks

Dabbler
Joined
Mar 27, 2014
Messages
18
The email subject line is:
Cron <root@freenas> PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" /mnt/dwhacks1/chkconfig.sh && /mnt/dwhacks1/bkpconfig.sh > /dev/null 2> /dev/null‏

And the email says "Database is ok"

chkconfig,sh:
Code:
#!/bin/sh
# This file runs a database integrity check and emails you in the event that
# your database is corrupt. This file requires you to have *properly* setup
# emailing from the FreeNAS GUI.  The following variables are available for
# you to use:
#
# YourEmail:  The email address you want to send the email to if your database
# is found to be corrupt.  Multiple email addresses are supported if separated
# by a space.
YourEmail="forsakenrider@hotmail.com"
# ServerName:  How you want your server's name to appear in the email in the
# event that the database is corrupt.
ServerName=dwhacksFreeNAS
# TempLocation:  Location for the temp file for the email.  Default is /tmp
TempLocation=/tmp
# Scroll down to edit the email as you see fit.  The default setup is recommended
# since it works and conveys a simple email to let you know what the problem
# is and that you need to take action.
# If you want to test this to ensure it works, simply rename the line with
# freenas-v1.db to pointto a file that doesn't exist.  It will error, and you will get
# an email.
if [ -f "$TempLocation"/badconfig.txt ];
    then rm "$TempLocation"/badconfig.txt
fi
 
if [ "$( sqlite3 /data/freenas-v1.db "pragma integrity_check;" )" == "ok" ]; then
        echo "Database is ok."
        #bail out with zero (all o.k.) status
        exit 0
    else
        # The following lines are the email that will be sent in the event that errors
        # are found.  Change it however you wish, just make sure the general format
        # is protected.
        echo "To: $YourEmail" >> $TempLocation/badconfig.txt
        echo "Subject: ERROR: Database corrupt on server $ServerName" >> $TempLocation/badconfig.txt
        echo "Your server, $ServerName, has been found to have a corrupt FreeNAS config." >> $TempLocation/badconfig.txt
        echo " " >> $TempLocation/badconfig.txt
        echo "It is recommended you troubleshoot and correct the problem as soon as possible.  Just because the server is operating fine does not mean you can ignore this message." >> $TempLocation/badconfig.txt
        echo " " >> $TempLocation"/badconfig.txt
        sendmail -t < "$TempLocation"/badconfig.txt
        rm "$TempLocation"/badconfig.txt
    fi
exit 1


and bkconfig.sh:

Code:
cp /data/freenas-v1.db /mnt/dwhacks1/backupconfig/`date +%Y%m%d`.db


I cant see the problem.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Did you do...

2. Create a cronjob that points to the file:

User: root
Command : /whereveryouputthescript/scriptname.sh
Short Description: Daily database check
All of the date/time options: However you want to set them. I do nightly at 2330.
Redirect Stdout: Checked
Redirect Stderr: Checked
Enabled: What do you think? If you don't know, stop using FreeNAS right now...

correctly? Money says "no". ;)
 

dwhacks

Dabbler
Joined
Mar 27, 2014
Messages
18
Sure did.

User: root
Command: /mnt/dwhacks1/chkconfig.sh && /mnt/dwhacks1/bkpconfig.sh
short description: Daily check and Backup of Config
Minute: 55
Hour: 23
day of month, every n day: 1
All months checked
all days checked
redirect stdout: check
redirect stderr: check
enabled: check

where else could my error be?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Actually, now I am stumped. Does it actually make the backup?

If you point the config checking script to a file that isn't a valid sql database, does the backup still happen?

Basically we need to boil down if both scripts are actually being run through to conclusion.
 

erturne

Dabbler
Joined
Sep 5, 2013
Messages
19
where else could my error be?

I think the problem is that
Code:
> /dev/null
that the cron definition puts on the end based on redirect stdout and stderr being checked is only redirecting the output of the second shell script. You have three options:
  1. Remove the
    Code:
    echo "Database is ok."
    from chkconfig.sh
  2. Make the command:
    Code:
    /mnt/dwhacks1/chkconfig.sh > /dev/null 2>/dev/null && /mnt/dwhacks1/bkpconfig.sh
  3. Make the command:
    Code:
    /bin/sh -c "/mnt/dwhacks1/chkconfig.sh && /mnt/dwhacks1/bkpconfig.sh"
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Haha.. erturne probably just hit the nail on the head. I didn't even think about that. I'm not sure if #3 will work. Since you are creating a blank environment from /bin/sh it may or may not work.
 

dwhacks

Dabbler
Joined
Mar 27, 2014
Messages
18
Ohhhhhh I think I will try number two. Because yes, my backups are getting made.

If i put that in the shell:
Code:
[root@freenas] /mnt/dwhacks1/backupconfig# /mnt/dwhacks1/chkconfig.sh > /dev/null 2>/dev/null && /mnt/dwhacks1/bkpconfig.sh
Ambiguous output redirect.
 


does that seem correct? Because it doesnt create a backup if I run that
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yeah.. I'll remove the echo from my original post. ;)
 

alexg

Contributor
Joined
Nov 29, 2013
Messages
197
Cyberjock, there is a typo at the end line 38 in chkconfig.sh. I think you probably meant to end it with "it" and ending double quote.
 

alexg

Contributor
Joined
Nov 29, 2013
Messages
197
Actually, your line 39 in chkconfig is really should be part of line 38 as
Code:
echo "It is recommended you troubleshoot and correct the problem as soon as possible.  Just b
ecause the server is operating fine does not mean you can ignore it" >> $TempLocation/badconfig.txt
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Thanks alexg. Somehow that line got fubared. Not sure if it is/was a copy/paste error or what. But my script on my server doesn't have the error. /shrug
 
Top