Scripts to report SMART, ZPool and UPS status, HDD/CPU T°, HDD identification and backup the config

[Darren Myers]

I finally got those to work! Huge thanks for those - they are a life saver.

The CPU/HDD temp output works great, I'm having some errors with the e-mail the SMART report one.

Here's the error I got & then the output it gives me - I have 6 drives. I get the full report for the detailed SMART status, but the grid doesn't fill out ada1 to ada4 - ada6 and ada7 are not in use at the moment, but will be in a few days.

Any ideas?

Code:

[root@freenas] /scripts# ./smart_hdd.sh
awk: can't open file 81
source line number 1
awk: can't open file 21
source line number 1
awk: can't open file 81
source line number 1
awk: can't open file 21
source line number 1

Code:

########## SMART status report summary for all drives ##########


+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+

|Device|Serial         |Temp|Power|Start|Spin |ReAlloc|Current|Offline |UDMA  |Seek  |High  |Command|Last|

|      |               |    |On   |Stop |Retry|Sectors|Pending|Uncorrec|CRC   |Errors|Fly   |Timeout|Test|

|      |               |    |Hours|Count|Count|       |Sectors|Sectors |Errors|      |Writes|Count  |Age |

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+

|ada0  |W1F4TVXX       | 35 | 9953| 1079|    0|      0|      0|       0|     0|     0|     0|      0| 415|

|ada5  |Z1F4Z3EB       | 34 | 1321|  539|    0|      0|      0|       0|     0|     0|     0|      0|  55|

|ada6  |               |  |     |     |     |       |       |        |      |   N/A|   N/A|    N/A|   0|

|ada7  |               |  |     |     |     |       |       |        |      |   N/A|   N/A|    N/A|   0|

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+




########## SMART status report for ada0 drive (Seagate Barracuda 7200.14: W1F4TVXX) ##########


SMART overall-health self-assessment test result: PASSED...

Are you sure the drives are adaX and not daX? 2 of my drives (the boot drives) come up as adaX and all my hdds come up as daX

 

[shnurov]

Yes - I mean, I see the full SMART report, I just don't see the data for my RED's, only for the two 3TB's

 

[Darren Myers]

Yes - I mean, I see the full SMART report, I just don't see the data for my RED's, only for the two 3TB's

If you goto FreeNAS>Storage>View Disks are all your disks labeled as adaX or daX?

 

[shnurov]

If you goto FreeNAS>Storage>View Disks are all your disks labeled as adaX or daX?

ada0 (3tb), ada1 (4tb), ada2 (4tb), ada3 (4tb), ada4 (4tb), ada5 (3tb)
The two 3tb are not part of any ZFS volume as I am waiting to take out the other 3TB's and put them in together.

 

[Bidule0hm]

You're welcome ;)

Ok, first I need to know the output of camcontrol devlist and I also need the output of cat -e the_script_name_here.sh

 

[Darren Myers]

and your script is told to do ada0 ada1 etc ? Just like in the example

Code:

drives="da0 da1 da2 da3 da4 da5 da6 da7"

 

[shnurov]

Yes my script is all 'ada0....ada7'

Camcontrol devlist:

Code:

<ST3000DM001-1CH166 CC27>          at scbus1 target 0 lun 0 (pass0,ada0)
<Marvell Console 1.01>             at scbus9 target 0 lun 0 (pass1)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus10 target 0 lun 0 (pass2,ada1)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus11 target 0 lun 0 (pass3,ada2)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus12 target 0 lun 0 (pass4,ada3)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus13 target 0 lun 0 (pass5,ada4)
<ST3000DM001-1CH166 CC27>          at scbus14 target 0 lun 0 (pass6,ada5)
<Kingston DataTraveler 2.0 PMAP>   at scbus17 target 0 lun 0 (pass7,da0)

Cat -e gives:

Code:

#!/bin/sh$
$
### Parameters ###$
logfile="/tmp/smart_report.tmp"$
email="phil@lavimage.com"$
subject="SMART Status Report for FreeNAS"$
drives="ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7"$
tempWarn=40$
tempCrit=45$
sectorsCrit=10$
warnSymbol="?"$
critSymbol="!"$
$
### Set email headers ###$
($
    echo "To: ${email}"$
    echo "Subject: ${subject}"$
    echo "Content-Type: text/html"$
    echo "MIME-Version: 1.0"$
    echo -e "\r\n"$
) > ${logfile}$
$
### Set email body ###$
echo "<pre style=\"font-size:14px\">" >> ${logfile}$
$
###### summary ######$
($
    echo ""$
    echo "########## SMART status report summary for all drives ##########"$
    echo ""$
    echo "+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+"$
    echo "|Device|Serial         |Temp|Power|Start|Spin |ReAlloc|Current|Offline |UDMA  |Seek  |High  |Command|Last|"$
    echo "|      |               |    |On   |Stop |Retry|Sectors|Pending|Uncorrec|CRC   |Errors|Fly   |Timeout|Test|"$
    echo "|      |               |    |Hours|Count|Count|       |Sectors|Sectors |Errors|      |Writes|Count  |Age |"$
    echo "+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+"$
) >> ${logfile}$
for drive in $drives$
do$
    ($
        smartctl -A -i -v 7,hex48 /dev/${drive} | \$
        awk -v device=${drive} -v tempWarn=${tempWarn} -v tempCrit=${tempCrit} -v sectorsCrit=${sectorsCrit} \$
        -v warnSymbol=${warnSymbol} -v critSymbol=${critSymbol} -v lastTestHours=`smartctl -l selftest /dev/${drive} | grep "# $
1" | awk '{print $9}'` '\$
        /Serial Number:/{serial=$3} \$
        /Temperature_Celsius/{temp=$10} \$
        /Power_On_Hours/{onHours=$10} \$
        /Start_Stop_Count/{startStop=$10} \$
        /Spin_Retry_Count/{spinRetry=$10} \$
        /Reallocated_Sector/{reAlloc=$10} \$
        /Current_Pending_Sector/{pending=$10} \$
        /Offline_Uncorrectable/{offlineUnc=$10} \$
        /UDMA_CRC_Error_Count/{crcErrors=$10} \$
        /Seek_Error_Rate/{seekErrors=("0x" substr($10,3,4));totalSeeks=("0x" substr($10,7))} \$
        /High_Fly_Writes/{hiFlyWr=$10} \$
        /Command_Timeout/{cmdTimeout=$10} \$
        END {$
            if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)$
                device=device " " critSymbol;$
            else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0)$
                device=device " " warnSymbol;$
            seekErrors=sprintf("%d", seekErrors);$
            totalSeeks=sprintf("%d", totalSeeks);$
            if (totalSeeks == "0") {$
                seekErrors="N/A";$
                totalSeeks="N/A";$
            }$
            if (hiFlyWr == "") hiFlyWr="N/A";$
            if (cmdTimeout == "") cmdTimeout="N/A";$
            testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);$
            printf "|%-6s|%-15s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%6s|%6s|%7s|%4s|\n",$
            device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc, \$
            crcErrors, seekErrors, hiFlyWr, cmdTimeout, testAge;$
        }'$
    ) >> ${logfile}$
done$
($
    echo "+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+"$
    echo ""$
    echo ""$
) >> ${logfile}$
$
###### for each drive ######$
for drive in $drives$
do$
    brand=`smartctl -i /dev/${drive} | grep "Model Family" | awk '{print $3, $4, $5}'`$
    serial=`smartctl -i /dev/${drive} | grep "Serial Number" | awk '{print $3}'`$
    ($
        echo ""$
        echo "########## SMART status report for ${drive} drive (${brand}: ${serial}) ##########"$
        smartctl -H -A -l error /dev/${drive}$
        smartctl -l selftest /dev/${drive} | grep "# 1 \|Num" | cut -c6-$
        echo ""$
        echo ""$
    ) >> ${logfile}$
done$
sed -i '' -e '/smartctl 6.3/d' ${logfile}$
sed -i '' -e '/Copyright/d' ${logfile}$
sed -i '' -e '/=== START OF READ/d' ${logfile}$
sed -i '' -e '/SMART Attributes Data/d' ${logfile}$
sed -i '' -e '/Vendor Specific SMART/d' ${logfile}$
sed -i '' -e '/SMART Error Log Version/d' ${logfile}$
echo "</pre>" >> ${logfile}$
$
### Send report ###$
sendmail -t < ${logfile}$
rm ${logfile}$

 

[Bidule0hm]

You should delete your email address from the output ;)

Ok, I see the problem.

drives="ada0 ada1 ada2 ada3 ada4 ada5 ada6 ada7" should be drives="ada0 ada1 ada2 ada3 ada4 ada5" because you only have 6 drives, not 8.

Then line 43 is broken at the end :

Code:

-v warnSymbol=${warnSymbol} -v critSymbol=${critSymbol} -v lastTestHours=`smartctl -l selftest /dev/${drive} | grep "# $
1" | awk '{print $9}'` '\$

It should be

Code:

-v warnSymbol=${warnSymbol} -v critSymbol=${critSymbol} -v lastTestHours=`smartctl -l selftest /dev/${drive} | grep "# 1" | awk '{print $9}'` '\$

I'll edit the script to shorten the line, in the mean time you can replace the line by this this in your script :

Code:

-v warnSymbol=${warnSymbol} -v critSymbol=${critSymbol} -v lastTestHours=`smartctl -l selftest /dev/${drive} | \
grep "# 1" | awk '{print $9}'` '\$

Edit: I've edited the script.

 

[shnurov]

Woohoo! Great :) it works, thanks.
I had my terminal large, but not THAT large.
I'll keep the other 2 drives listed there as I'm going to be adding them in a few days and I don't want to go back into the configs to change stuff up.

 

[Bidule0hm]

Ok, great ;)

You can also use the self-adaptive script: http://pastebin.com/7RFkrxJU it'll list all the drives (but the USB sticks) without any human tweaking :)

 

[okgunguy]

The Self-Adaptive works great. No worries about how many drives, or how they're listed. Even when I add more, they'll be covered.

 

[zoomzoom]

I've modified the temperatures script to be more user friendly ;)

@zoomzoom Have you tested the pop3 protocol?

I haven't, but I should have time tomorrow to do so

 

[Bidule0hm]

Ok, thanks ;)

 

[djakdarippa]

This is most likely not the place to ask, but has anyone lately been getting python/sendmail errors? Using smart_report by hand works like a charm every time. The same script when scheduled often just sends me the following:

Code:

Traceback (most recent call last):
  File "/etc/find_alias_for_smtplib.py", line 127, in <module>
    main()
  File "/etc/find_alias_for_smtplib.py", line 124, in main
    do_sendmail(msg, to_addrs=to, parse_recipients=args.parse_recipients)
  File "/etc/find_alias_for_smtplib.py", line 40, in do_sendmail
    to_addrs = map(str.strip, em.get('To').split(','))
AttributeError: 'NoneType' object has no attribute 'split'

I can't remember when this worked properly last, 2 or 3 weeks back probably. The zpool report seems to have no issues...
From what I can understand it objects to being told to split recipients. Such an attribute couldn't have vanished overnight, right?

 

[zoomzoom]

@zoomzoom Have you tested the pop3 protocol?

It doesn't work when hotmail.com/outlook.com/live.com emails are configured with POP3 either... however there is something odd that occurs that seems to indicate the config backup is actually attached, but inaccessible to the user.

I say this because of email sizes:

• Gmail [attachment accessible]: 692KB
• Hotmail/Live/Outlook.com [inaccessible]: 1,523KB
  • Both output the tar as text, both outputs equal 8343 lines of text

 

[zoomzoom]

This is most likely not the place to ask, but has anyone lately been getting python/sendmail errors? Using smart_report by hand works like a charm every time. The same script when scheduled often just sends me the following:

Code:

Traceback (most recent call last):
  File "/etc/find_alias_for_smtplib.py", line 127, in <module>
    main()
  File "/etc/find_alias_for_smtplib.py", line 124, in main
    do_sendmail(msg, to_addrs=to, parse_recipients=args.parse_recipients)
  File "/etc/find_alias_for_smtplib.py", line 40, in do_sendmail
    to_addrs = map(str.strip, em.get('To').split(','))
AttributeError: 'NoneType' object has no attribute 'split'

I can't remember when this worked properly last, 2 or 3 weeks back probably. The zpool report seems to have no issues...
From what I can understand it objects to being told to split recipients. Such an attribute couldn't have vanished overnight, right?

Reviewing the last email for the cron smart script, I receive:

• Syntax error: "(" unexpected

I assumed I forget to close a bracket when I was editing the script, however I just ran it manually and it ran fine. Like you as well, all other cron run scripts (zpool, config backup, etc.) run fine.

• I'm unsure what logs/information is needed to troubleshoot where the error is occuring, so please let me know what information is needed.

On an unrelated note, is there a way to force the SMART script to report a "0" if no temp is garnished from a drive? For example, my Samsung 850 evo doesn't output a temp reading, which causes a two character offset in the output for the drive.

• This isn't an issue, I was simply curious =] (This shows a 5 character offset for some reason)

|ada5 || 33 | 1343| 9890| 0| 0| 0| 0| 0| 0| 0| 0| 5|
|ada6 || | 4839| | | 0| | | | N/A| N/A| N/A| 10|
|ada7 || 27 | 7159| 1793| 0| 0| 0| 0| 0| 1| 0| 0| 5|
​

 

[Bidule0hm]

@djakdarippa I have no idea, sorry.

@zoomzoom Ok, thanks for the tests. The file is here, it's just outputed as text in the mail instead of being downloadable as an attachement, that's why the mail is still big.

On an unrelated note, is there a way to force the SMART script to report a "0" if no temp is garnished from a drive?

Yes, you can. Just add the line if (temp == "") temp="00"; before the line seekErrors=sprintf("%d", seekErrors); ;)

NB: I haven't tested the scripts on FreeNAS 9.3.1 because I'm still on 9.3.0 and I've not planned to upgrade before a long time given the problems the 9.3.1 has so I'll not be able to test them if there's some problems related to the FreeNAS version.

 

[zoomzoom]

@zoomzoom The file is here, it's just outputed as text in the mail instead of being downloadable as an attachement, that's why the mail is still big.

Yes, you can. Just add the line if (temp == "") temp="00"; before the line seekErrors=sprintf("%d", seekErrors); ;)

Awesome, thanks =]

How do I take the text output of the file and use it to create the tar file?

 

[Bidule0hm]

In theory you can echo it to a file but I've not tested this so it's maybe more complex than that (and I'm tired today so...)

 

[zoomzoom]

In theory you can echo it to a file but I've not tested this so it's maybe more complex than that (and I'm tired today so...)

I'll give that a try =]