Scripts to report SMART, ZPool and UPS status, HDD/CPU T°, HDD identification and backup the config

[Johannez]

Bidule0hm, on both of my machines i cancelled a scrub and the new Zpool script does work however when a scrub is canceled it also creates another email.

Content of the email:

Code:

<br />date: illegal option -- -<br />usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...<br />            [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]<br />

Subject of the email:

Code:

<br />Cron &lt;root@freenas&gt; PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" sh /root/scripts/zpool.sh &gt; /dev/null

Afcorse this ain't a problem cause the script works perfect but I wanted to share this with you.
Thank you for your wonderful scripts.

 

[Bidule0hm]

Which version of FreeNAS?

Thanks for reporting the problem ;)

 

[Johannez]

FreeNAS-9.10.1 (d989edd) and FreeNAS-9.10-STABLE-201606072003 (696eba7)

 

[Stux]

Here's a present ;)

I made a new script I wanted to make for a long time: it checks the config file integrity and sends a copy of it if everything is ok or an error message otherwise: http://pastebin.com/syF2JeAU :)

But before I put it in the first post can someone with another client than gmail checks that it works ok? (the way the file is attached is more or less a hack but as we can't add apps to the system it's not like if I have the choice...) NB: the mail can take some time to be received so wait a bit before checking the inbox.

I wasn't very satisfied with the way the email attachment was handled in the email config script...

./email_attachment.sh from@email.com to@email.com "Subject Of Email" "The body of the email" attachment1.tar.gz attachment2.jpg attachment3.pdf

Code:

#!/bin/bash

# script to send zero or more attachments with mime-encoding from any email address to any email address.

# based on:
# http://backreference.org/2013/05/22/send-email-with-attachments-from-script-or-command-line/

if [ "$#" -lt 4 ]; then
  echo "Usage: ${0} from@email to@email \"subject of email\" \"body of email\" [attachment1 [...]]"
  exit
fi

from="$1"
to="$2"
subject="$3"
boundary="ZZ_/afg6432dfgkl.94531q"
body="$4"
declare -a attachments

shift 4

attachments=($@)

get_mimetype(){
  # warning: assumes that the passed file exists
  file --mime-type "$1" | sed 's/.*: //'
}

# Build headers
{

printf '%s\n' "From: $from
To: $to
Subject: $subject
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary=\"$boundary\"

--${boundary}
Content-Type: text/plain; charset=\"US-ASCII\"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

$body
"

# now loop over the attachments, guess the type
# and produce the corresponding part, encoded base64
for file in "${attachments[@]}"; do

  [ ! -f "$file" ] && echo "Warning: attachment $file not found, skipping" >&2 && continue

  mimetype=$(get_mimetype "$file")

  printf '%s\n' "--${boundary}
Content-Type: $mimetype
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=\"$file\"
"

  base64 "$file"
  echo
done

# print last boundary with closing --
printf '%s\n' "--${boundary}--"

} | sendmail -t -oi   # one may also use -f here to set the envelope-from

 

[Stux]

Here's something. I wasn't very satisfied with the config attachment in the email configs script...

Modified version which uses the email_attachment.sh script

Code:

#!/bin/sh

### Parameters ###
email_script="/root/email_attachment.sh"
logfile="/tmp/config_backup_error.tmp"
tarfile="/tmp/config_backup.tar.gz"
filename="$(date "+FreeNAS_Config_%Y-%m-%d_%H-%M-%S")"
to_email="to@email.com"
from_email="from@email.com"
subject="Config Backup for FreeNAS"

if [ "$(sqlite3 /data/freenas-v1.db "pragma integrity_check;")" == "ok" ]
then
### Send config backup ###
	cp /data/freenas-v1.db "/tmp/${filename}.db"
	md5 "/tmp/${filename}.db" > /tmp/config_backup.md5
	sha256 "/tmp/${filename}.db" > /tmp/config_backup.sha256
	cd "/tmp/"; tar -czf "${tarfile}" "./${filename}.db" ./config_backup.md5 ./config_backup.sha256; cd -

	${email_script} "${from_email}" "${to_email}" "${subject}" "Config Backup for FreeNAS : ${filename}" "${tarfile}"

	rm "/tmp/${filename}.db"
	rm /tmp/config_backup.md5
	rm /tmp/config_backup.sha256
	rm "${tarfile}"
else
### Send error message ###
	(
		echo "To: ${email}"
		echo "Subject: ${subject} : FAILED"
		echo "Content-Type: text/html"
		echo "MIME-Version: 1.0"
		echo -e "\r\n"
		echo "<pre style=\"font-size:14px\">"
		echo ""
		echo "Automatic backup of FreeNAS config failed."
		echo ""
		echo "The config file is corrupted!"
		echo ""
		echo "You should correct this problem as soon as possible."
		echo ""
		echo "</pre>"
	) >> "${logfile}"
	sendmail -t < "${logfile}"
	rm "${logfile}"
fi

As a bonus is gzips the tar file, which reduces it down to 54KB from 352KB in my test.

If you wanted to, you could directly attach the md5 and sha256 without actually using tar with the email_attachment.sh script

Thankyou for your scripts :)

 

[Bidule0hm]

Thanks :D

I'll read and test that asap, seems very interesting ;)

 

[skyline65]

I tried using this... but as you know Im a total noob at this. The only way I could get it to send was modifying

from_email="from.email.com"

to

from_email="from@email.com"

is that right?

 

[Stux]

I tried using this... but as you know Im a total noob at this. The only way I could get it to send was modifying

from_email="from.email.com"

to

from_email="from@email.com"

is that right?

Yes.

It's the email address the email will appear to be from. Make it whatever you want.

Silly mistake on my behalf not using the @ sign when I edited out my servers email.

Ie on my setup my servers email me from servername@mydomain.com, that way I can easily see which server is sending emails

 

[skyline65]

OK thats great. I appreciate all the work you have done.:)

On another subject are there any books you would recommend to learn the code for scripting? As I spend hours a day staring at a display designing layouts sometimes it is nice to look at a book.

 

[cryptyk]

This is so awesome. One quick question on the sendmail of the drive test script:
I'd like to only send the email when there was at least one warning. I planned on setting a variable in the awk script section but then realized that you can't set a shell script variable from within there.
Have you tried to do that before in other scripts? Is there a pattern you already have for that?

Thanks so much for the scripts!!!

--EDIT--
Nevermind. I hacked it together by grepping for the warning and critical symbols in the logfile and only sending it if it found them :)

Thanks again!

 

[Dice]

This is so awesome. One quick question on the sendmail of the drive test script:
I'd like to only send the email when there was at least one warning.

Would you mind posting a versions with some sort of instruction for us less script savvy guys to take part in?
My main reason I've not used the test script is ...what you just fixed.

Cheers, Dice

 

[Stux]

OK thats great. I appreciate all the work you have done.:)

On another subject are there any books you would recommend to learn the code for scripting? As I spend hours a day staring at a display designing layouts sometimes it is nice to look at a book.

Sorry, can't recommend any books. You might not realise it, but the Hybrid CPU controller was pretty much the first perl script I've ever written. I just pieced it together with googling what I wanted to do and previous coding experience. This perhaps explains why its style is not exactly perly ;)

 

[kspare]

Hey Guys, when I run the zpoolreport on two of my servers I get the following error:


date: illegal option -- -

usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...

[-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]

/usr/bin/zpoolreport: arithmetic expression: expecting primary: "((1478696400 - ) + 43200) / 86400"


it does this on two of my three boxes.

 

[Bidule0hm]

That's probably because at least one of the pools haven't been scrubbed.

 

[dcevansiii]

All these scripts are indispensable. Can't thank you enough.

I've altered the ability for the drive_information.sh script to grab the drive size too. I thought people might find that useful.

It consists of extending the header/footer , adding a new variable: size, and altering the printf line.

Cheers.

Code:

#!/bin/sh

drives=$(for drive in $(sysctl -n kern.disks); do \
if [ "$(smartctl -i /dev/${drive} | grep "Serial Number" | awk '{print $3}')" ]
then printf ${drive}" "; fi done | awk '{for (i=NF; i!=0 ; i--) print $i }')

echo ""
echo "+========+============================================+=================+=================+"
echo "| Device | GPTID									  | Serial		  | Size			|"
echo "+========+============================================+=================+=================+"
for drive in $drives
do
	gptid=`glabel status -s "${drive}p2" | awk '{print $1}'`
	serial=`smartctl -i /dev/${drive} | grep "Serial Number" | awk '{print $3}'`
	size=`smartctl -i /dev/${drive} | grep "User Capacity" | awk '{print $5$6}'`
	printf "| %-6s | %-42s | %-15s | %-15s \n" "$drive" "$gptid" "$serial" "$size"
	echo "+--------+--------------------------------------------+-----------------+-----------------+"
done
echo ""

 

[GBillR]

@Bidule0hm - Thank you for your work on these... simply awesome effort.

I am having a minor formatting issue with the smart script. my SSD reports POH in xxxh+00m+00.00s. As a result, the emailed table formatting is a bit off:

Code:

########## SMART status report summary for all drives ##########

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|Device|Serial  |Temp|Power|Start|Spin |ReAlloc|Current|Offline |UDMA  |Seek  |High  |Command|Last|
|  |  |  |On  |Stop |Retry|Sectors|Pending|Uncorrec|CRC  |Errors|Fly  |Timeout|Test|
|  |  |  |Hours|Count|Count|  |Sectors|Sectors |Errors|  |Writes|Count  |Age |
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|ada0 ?|163603456807  | 25 |161h+00m+00.000s|  |  |  |  |  |  |  N/A|  N/A|  0|  6|
|ada1 ?|WD-WXG1A259PP95| 18 |  83|  4|  0|  0|  0|  0|  0|  N/A|  N/A|  N/A|  3|
|ada2  |WD-WCC4N4KY2PFX| 19 |  115|  9|  0|  0|  0|  0|  0|  N/A|  N/A|  N/A|  1|
|ada3  |WD-WCC4N2KNSYVF| 20 |  115|  9|  0|  0|  0|  0|  0|  N/A|  N/A|  N/A|  1|
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+



########## SMART status report for ada0 drive (SandForce Driven SSDs: 163603456807) ##########
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  5 Retired_Block_Count  0x0032  100  100  000  Old_age  Always  -  0
  9 Power_On_Hours_and_Msec 0x0032  000  100  000  Old_age  Always  -  161h+00m+00.000s
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  10
165 Unknown_Attribute  0x0032  100  100  000  Old_age  Always   -  65536
166 Unknown_Attribute  0x0032  100  100  000  Old_age  Always  -  0
167 Unknown_Attribute  0x0032  100  100  000  Old_age  Always  -  0
168 Unknown_Attribute  0x0032  100  100  000  Old_age  Always  -  0
169 Unknown_Attribute  0x0032  100  100  000  Old_age  Always  -  0
170 Reserve_Block_Count  0x0032  100  100  000  Old_age  Always  -  0
171 Program_Fail_Count  0x0032  100  100   000  Old_age  Always  -  0
172 Erase_Fail_Count  0x0032  100  100  000  Old_age  Always  -  0
173 Unknown_SandForce_Attr  0x0032  100  100  000  Old_age  Always  -  0
174 Unexpect_Power_Loss_Ct  0x0032  100  100  000  Old_age  Always  -  1
187 Reported_Uncorrect  0x0032  100  100  000  Old_age  Always  -  0
188 Command_Timeout  0x0032  100  100  000  Old_age  Always  -  0
194 Temperature_Celsius   0x0022  075  057  000  Old_age  Always  -  25 (Min/Max 0/43)
199 SATA_CRC_Error_Count  0x0032  100  100  000  Old_age  Always  -  0
230 Life_Curve_Status  0x0032  100  100  000  Old_age  Always  -  0
232 Available_Reservd_Space 0x0033  100  100  004  Pre-fail  Always  -  100
233 SandForce_Internal  0x0032  100  100  000  Old_age  Always  -  0
234 SandForce_Internal  0x0032  100  100  000  Old_age  Always   -  2
241 Lifetime_Writes_GiB  0x0030  253  253  000  Old_age  Offline  -  1
242 Lifetime_Reads_GiB  0x0030  253  253  000  Old_age  Offline  -  1
244 Unknown_Attribute  0x0032  000  100  000  Old_age  Always  -  0

No Errors Logged

Test_Description  Status  Remaining  LifeTime(hours)  LBA_of_first_error
Short offline  Completed without error  00%  8  -

I'm no Bash expert (barely know enough to be dangerous)... Any suggestions on how best to modify the following line to achieve the desired POH output for my SSD:

/Power_On_Hours/{onHours=$10} \

Thank you in advance.

 

[joeschmuck]

Use a monospaced font might help.

 

[GBillR]

Use a monospaced font might help.

Thank you for your reply Joe. I may have not been clear enough in my initial post.... and the copy-paste job I did altered the "header" lines slightly (which I assume is the basis for your response).

My specific formatting issue is due to the length of the Power On Hours for the Boot SSD (ada0). It reports POH to the millisecond, and therefore kicks that one line out of alignment with the rest of the table:

Code:

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|ada0 ?|163603456807   | 25 |161h+00m+00.000s| | | | | | | N/A| N/A| 0| 6|
|ada1 ?|WD-WXG1A259PP95| 18 | 83| 4| 0| 0| 0| 0| 0| N/A| N/A| N/A| 3|
|ada2  |WD-WCC4N4KY2PFX| 19 | 115| 9| 0| 0| 0| 0| 0| N/A| N/A| N/A| 1|
|ada3  |WD-WCC4N2KNSYVF| 20 | 115| 9| 0| 0| 0| 0| 0| N/A| N/A| N/A| 1|
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+

My question is: Is there a change I can make to the script that will truncate the POH for that drive such that it rounds to whole hours? I have gone back and modified the script to poll only my other 3 drives for now, but doing so has removed the "self-adaptive" functionality (which I technically can live without).

Another alternative would be to simply change the self-adaptive section to remove ada0 from the list without removing the functionality... but again, I am no Bash guru, and would need to research that. That said, I recall reading one of the posts in this thread that noted that the script and associated SMART data was never really intended to be used on a SSD. Perhaps an enhancement to the script could be to alter the drives section to filter out SSDs (similar to the USB filtering that was added)?

Thanks for your assistance.

 

[joeschmuck]

Simple answer is YES, you can make a change to the script to truncate the text, not really round it. I don't have the time to dive into the actual solution but there are commands to manipulate the text data, but you need to ensure it manipulates everything properly. But it also looks like you will have alignment issues still because you have fields with no data, where the spinning rust has data. So unless you are willing to learn how to write a script, you might be best to just live with it.

 

[GBillR]

Simple answer is YES, you can make a change to the script to truncate the text, not really round it. I don't have the time to dive into the actual solution but there are commands to manipulate the text data, but you need to ensure it manipulates everything properly. But it also looks like you will have alignment issues still because you have fields with no data, where the spinning rust has data. So unless you are willing to learn how to write a script, you might be best to just live with it.

The actual email I receive without including ada0 looks decent, and is aligned. Since I can't figure out how to insert the aligned text into this message, I've attached an image.

Thank you for pointing me in the right direction. I am very willing to learn how to modify this script (that's part of the reason I selected FreeNAS over other solutions), so I will do some more research. I'd be very appreciative of any advice or references that you care to offer.

Bill