Scripts to report SMART, ZPool and UPS status, HDD/CPU T°, HDD identification and backup the config

[Darren Myers]

Is it possible to add fragmentation to the zpool status script?

 

[Spearfoot]

Is it possible to add fragmentation to the zpool status script?

Yes, and done!

I added it to my version of zpool_report.sh, available on GitHub: github.com/Spearfoot/FreeNAS-scripts.

The new header looks like this:

 

[Darren Myers]

Yes, and done!

I added it to my version of zpool_report.sh, available on GitHub: github.com/Spearfoot/FreeNAS-scripts.

The new header looks like this:
View attachment 17492

That is awesome! Thanks so much!

 

[Xyrgh]

Baffling!
But you really ought to start a new thread, perhaps in the networking subforum. This thread is too long already, and we're distracting from the intent of the original poster.

Just FYI, I've resolved this issue. I ended up adding 127.0.0.1 as a bind address in the System>WebUI bind address settings and it resolved the issue. Not sure if this was a default value, but if it was, it wasn't there and didn't migrate with the upgrade.

/edit Just a question with regards to the smart script. Is there any way to extend the 'serial' column by a few spaces? My SSD serial numbers are quite lengthy and makes the table look all askew. Adding a few extra 'columns' would be beneficial.

 

[Darren Myers]

Yes, and done!

I added it to my version of zpool_report.sh, available on GitHub: github.com/Spearfoot/FreeNAS-scripts.

The new header looks like this:
View attachment 17492

Quick note ; line 34 you did a " } " instead of " | " just an FYI incase you're a neat freak like myself haha

 

[Spearfoot]

Quick note ; line 34 you did a " } " instead of " | " just an FYI incase you're a neat freak like myself haha

Dooooh!
Fixed it. Thanks!

 

[zoomzoom]

I just updated to the new version of your smart report script and noticed Last Test Age was missing. I added the code block [below] from the older version, however it's not correctly calculating so I'm assuming something has changed in how the value is polled/calculated? How would I go about trying to calculate the last test age to incorporate into the newer version of your script?

Code:

# Old Script Version:
	if (cmdTimeout == "") cmdTimeout="N/A";
			testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
			printf "|%-6s|%-15s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%6s|%6s|%7s|%4s|\n",
			device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc, \
			crcErrors, seekErrors, hiFlyWr, cmdTimeout, testAge;

# New Script Version:
	  if (cmdTimeout == "") cmdTimeout="N/A";
	  testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
	  printf "|%-6s  |%-18s| %s | %5s | %5s |%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
	  device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
	  seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;

  # Test Age specific code:
   testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
   %4s|
   testAge;

 

[Spearfoot]

I just updated to the new version of your smart report script and noticed Last Test Age was missing. I added the code block [below] from the older version, however it's not a correctly calculating so I'm assuming something has changed in how the value is polled/calculated? How would I go about trying to calculate the last test age to incorporate into the newer version of your script?

Code:

# Old Script Version:
	if (cmdTimeout == "") cmdTimeout="N/A";
			testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
			printf "|%-6s|%-15s| %s |%5s|%5s|%5s|%7s|%7s|%8s|%6s|%6s|%6s|%7s|%4s|\n",
			device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc, \
			crcErrors, seekErrors, hiFlyWr, cmdTimeout, testAge;

# New Script Version:
	  if (cmdTimeout == "") cmdTimeout="N/A";
	  testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
	  printf "|%-6s  |%-18s| %s | %5s | %5s |%5s|%7s|%7s|%8s|%6s|%10s|%6s|%7s|%4s|\n",
	  device, serial, temp, onHours, startStop, spinRetry, reAlloc, pending, offlineUnc,
	  seekErrors, totalSeeks, hiFlyWr, cmdTimeout, testAge;

  # Test Age specific code:
   testAge=sprintf("%.0f", (onHours - lastTestHours) / 24);
   %4s|
   testAge;

Hmmm... I must have started with a really old version of the script that didn't have the 'Last Test Age' information...

If you refer to @Bidule0hm's original script on pastebin, you'll see there's additional code required to generate this value. You have to make a call to smartctl to get the hours of the last test, pass the value to awk as a variable, etc.

I've modified my version to include this feature; it's available on GitHub here: github.com/Spearfoot/FreeNAS-scripts.

Thanks for pointing this out!

 

[zoomzoom]

@Spearfoot awesome =] Thanks!

 

[Xyrgh]

Hey @Spearfoot, I posted a question above but you might have missed it. Is there any way to extend the 'serial' column by a few spaces? My SSD serial numbers are quite lengthy and makes the table look all askew. Adding a few extra 'columns' would be beneficial.

 

[Spearfoot]

Hey @Spearfoot, I posted a question above but you might have missed it. Is there any way to extend the 'serial' column by a few spaces? My SSD serial numbers are quite lengthy and makes the table look all askew. Adding a few extra 'columns' would be beneficial.

The version on GitHub supports serial number up to 18 characters long. How long are your serial numbers?

 

[zoomzoom]

@Xyrgh If you need to extend past the 18, and you haven't customized the script, you'll need to adjust following lines equally 68 - 72 & 116, as well as 108, specifically |%-18s|

@Spearfoot Is it no longer possible to have the config backup tar emailed in Corral, or is it that if the script is configured to copy the config backup to a local directory, it doesn't send a tarball? I'm currently getting the following emailed, but without any attachment:

Configuration file saved successfully on Sun Apr 9 01:01:46 CDT 2017

FreeNAS:
Server: Fractals
Version: FreeNAS-Corral-10.0.3 (4845fa74c)
File: /mnt/nas-system/config/Fractals-FreeNAS-Corral-10.0.3-4845fa74c-20170409010146.db
​

 

[Spearfoot]

@Xyrgh If you need to extend past the 18, and you haven't customized the script, you'll need to adjust following lines equally 68 - 72 & 116, as well as 108, specifically |%-18s|

@Spearfoot Is it no longer possible to have the config backup tar emailed in Corral, or is it that if the script is configured to copy the config backup to a local directory, it doesn't send a tarball? I'm currently getting the following emailed, but without any attachment:

Configuration file saved successfully on Sun Apr 9 01:01:46 CDT 2017

FreeNAS:
Server: Fractals
Version: FreeNAS-Corral-10.0.3 (4845fa74c)
File: /mnt/nas-system/config/Fractals-FreeNAS-Corral-10.0.3-4845fa74c-20170409010146.db
​

The script has never created a tarball -- it basically just copies the FreeNAS configuration file to the directory you specify.

For FreeNAS 9.x it just copies the *.db file to the directory. Corral is different, it uses Mongo instead of sqlite, so for Corral the script makes a command-line call to download the configuration to the directory.

 

[zoomzoom]

@Spearfoot I must have added that bit then lol

 

[Spearfoot]

@Spearfoot I must have added that bit then lol

It would be a nice feature. All you've have to do, I suppose, is create the tarball, MIME-encode it, and add it to the mail message body.

 

[zoomzoom]

I finally found the old script and adjusted the config_backup.sh script to make it work. I'm not familiar with using cli mail programs so some of the edits, or cuts, are likely unnecessary, such as some of the ${logfile} output, as I couldn't figure out how to make mail utilize the formatting to get monospaced output.

I always customize the formatting of others' scripts I use, so I'll post the script in whole form, as well as with the necessary parts for emailed tarballs. Since MS Outlook doesn't display tarballs as a physical attachment, and instead as the raw tarball code, I added single line separators to separate the logfile output from the tarball raw code.

• I haven't had a chance to test the the backup tarballs yet, so before anyone relies on them, please test.

• Since I'm not aware of how things are stored in the db files, someone will need to chime in on whether the tarballs should be encrypted (i.e. are passwords easy to decrypt in the db files) before being emailed.

Emailed Tarballs

Code:

fnconfig="${freenashost}"-"${fnconfigdest_version}"-"${fnconfigdest_date}".db

  (
	echo "To: ${email}"
	echo "Subject: ${subject}"
	echo "Content-Type: text/html"
	echo "MIME-Version: 1.0"
	echo ""
	echo "--------------------------------------------------------------------------------------------------------------"
	echo ""
	if [ $l_status -eq 0 ]; then
	  echo "Configuration file saved successfully on ${rundate}"
	else
	  echo "Configuration backup failed with status=${l_status} on ${rundate}"
	fi
	echo "FreeNAS"
	echo "Server: ${freenashostname}"
	echo "Version: ${freenasversion}"
	echo "File: ${fnconfigdest}"
	echo ""
	echo "--------------------------------------------------------------------------------------------------------------"
	echo ""
	echo ""
  ) > ${logfile}
   if [ "$(sqlite3 /data/freenas-v1.db "pragma integrity_check;")" == "ok" ]; then
	tarball=/tmp/"${fnconfig}".tar
	tarlog=/tmp/config_tarball_error.tmp
	if [ ! -z "${iscorral}" ]; then
	  # Corral:
	  cp "${fnconfigdest}" /tmp/"${fnconfig}"
	else
	  # 9.x:
	  cp /data/freenas-v1.db /tmp/"${fnconfig}"
	fi
	md5 /tmp/"${fnconfig}" > /tmp/"${fnconfig}".md5
	sha256 /tmp/"${fnconfig}" > /tmp/"${fnconfig}".sha256
	cd "/tmp/" ; tar -cf "${tarball}" ./"${fnconfig}" ./"${fnconfig}".md5 ./"${fnconfig}".sha256; cd -
	( cat "${logfile}" ; uuencode "${tarball}" "${fnconfig}".tar ) | mail -s "${subject}" "${email}"
	rm /tmp/"${fnconfig}" /tmp/"${fnconfig}".md5 /tmp/"${fnconfig}".sha256 "${tarball}"
  else
	(
	  echo "To: ${email}"
	  echo "Subject: ${subject}"
	  echo "Content-Type: text/html"
	  echo "MIME-Version: 1.0"
	  echo -e "\r\n"
	  echo "<pre style=\"font-size:14px\">"
	  echo ""
	  echo "Automatic backup of FreeNAS config failed."
	  echo ""
	  echo "The config file is corrupted!"
	  echo ""
	  echo "You should correct this problem as soon as possible."
	  echo ""
	  echo "</pre>"
	) >> "${tarlog}"
	sendmail -t < "${tarlog}" & rm "${tarlog}"
  fi

Full Script

Code:

#!/bin/sh

			 ##::[[-----> FreeNAS Config Backup Script <-----]]::##



	# Variables #
#--------------------------------------


# Date:
rundate=$(date)


# Email:
email="jw0914+FreeNAS_Backup@gmail.com"


# ESXi:
esxihost=""


# FreeNAS:
freenashost=$(hostname -s)

configdir="/mnt/nas-system/config"

fnconfigdest_version=$(< /etc/version sed -e 's/)//;s/(//;s/ /-/' | tr -d '\n')
fnconfigdest_date=$(date +%Y.%m.%d_%H:%M:%S)
fnconfigdest="${configdir}"/"${freenashost}"-"${fnconfigdest_version}"-"${fnconfigdest_date}".db

fnconfig="${freenashost}"-"${fnconfigdest_version}"-"${fnconfigdest_date}".db


	# Backup #
#--------------------------------------


echo "Backup Corral database Config: ${fnconfigdest}"

iscorral=$(< /etc/version grep "Corral" | awk {'print $1'})

if [ ! -z "${iscorral}" ]; then
  # Corral:
  cli -e "system config download path=${fnconfigdest}"
else
  # 9.x:
  cp /data/freenas-v1.db "${fnconfigdest}"
fi

l_status=$?


	# ESXi Host #
#--------------------------------------


if [ ! -z "${esxihost}" ]; then
  esxihostname=$(ssh root@"${esxihost}" hostname)
  esxiversion=$(ssh root@"${esxihost}" uname -a | sed -e "s|VMkernel ||;s|$esxihostname ||")
  esxiconfig_url=$(ssh root@"${esxihost}" vim-cmd hostsvc/firmware/backup_config | awk '{print $7}' | sed -e "s|*|$esxihostname|")
  esxiconfig_date=$(date +%Y%m%d%H%M%S)
  esxiconfig_file="${configdir}"/"${esxihost}"-configBundle-"${esxiconfig_date}".tgz

  echo "Downloading $esxiconfig_url to $esxiconfig_file"

  wget --no-check-certificate --output-document="${esxiconfig_file}" "${esxiconfig_url}"
fi


	# Email Content #
#--------------------------------------


if [ ! -z "${email}" ]; then
  freenashostuc=$(hostname -s | tr '[:lower:]' '[:upper:]')
  freenashostname=$(hostname)
  freenasversion=$(cat /etc/version)

  logfile="/tmp/save_config.tmp"

  if [ $l_status -eq 0 ]; then
	subject="FreeNAS configuration saved on server ${freenashostuc}"
  else
	subject="FreeNAS configuration backup failed on server ${freenashostuc}"
  fi

  # Header:
  #--------------------------------------

  (
	echo "To: ${email}"
	echo "Subject: ${subject}"
	echo "Content-Type: text/html"
	echo "MIME-Version: 1.0"
	echo ""
	echo "--------------------------------------------------------------------------------------------------------------"
	echo ""

	# Status:
	#--------------------------------------

	if [ $l_status -eq 0 ]; then
	  echo "Configuration file saved successfully on ${rundate}"
	else
	  echo "Configuration backup failed with status=${l_status} on ${rundate}"
	fi

	# FreeNAS:
	#--------------------------------------

	echo "FreeNAS"
	echo "Server: ${freenashostname}"
	echo "Version: ${freenasversion}"
	echo "File: ${fnconfigdest}"
	echo ""
	echo "--------------------------------------------------------------------------------------------------------------"
	echo ""
	echo ""

	# ESXi:
	#--------------------------------------

	if [ ! -z "${esxihost}" ]; then
	  echo "ESXi"
	  echo "Server: ${esxihostname}"
	  echo "Version: ${esxiversion}"
	  echo "File: ${esxiconfig_file}"
	  echo ""
	  echo "--------------------------------------------------------------------------------------------------------------"
	  echo ""
	  echo ""
	fi
  ) > ${logfile}

  if [ "$(sqlite3 /data/freenas-v1.db "pragma integrity_check;")" == "ok" ]; then

	# Tarball:
	#--------------------------------------

	tarball=/tmp/"${fnconfig}".tar
	tarlog=/tmp/config_tarball_error.tmp

	if [ ! -z "${iscorral}" ]; then
	  # Corral:
	  cp "${fnconfigdest}" /tmp/"${fnconfig}"
	else
	  # 9.x:
	  cp /data/freenas-v1.db /tmp/"${fnconfig}"
	fi

	md5 /tmp/"${fnconfig}" > /tmp/"${fnconfig}".md5
	sha256 /tmp/"${fnconfig}" > /tmp/"${fnconfig}".sha256

	cd "/tmp/" ; tar -cf "${tarball}" ./"${fnconfig}" ./"${fnconfig}".md5 ./"${fnconfig}".sha256; cd -
	( cat "${logfile}" ; uuencode "${tarball}" "${fnconfig}".tar ) | mail -s "${subject}" "${email}"

	rm /tmp/"${fnconfig}" /tmp/"${fnconfig}".md5 /tmp/"${fnconfig}".sha256 "${tarball}"

  else

	# Send:
	#--------------------------------------

	(
	  echo "To: ${email}"
	  echo "Subject: ${subject}"
	  echo "Content-Type: text/html"
	  echo "MIME-Version: 1.0"
	  echo -e "\r\n"
	  echo "<pre style=\"font-size:14px\">"
	  echo ""
	  echo "Automatic backup of FreeNAS config failed."
	  echo ""
	  echo "The config file is corrupted!"
	  echo ""
	  echo "You should correct this problem as soon as possible."
	  echo ""
	  echo "</pre>"
	) >> "${tarlog}"
	sendmail -t < "${tarlog}" & rm "${tarlog}"
  fi

fi

EDIT
I've been playing around with differing options for mail to get the text output to be a monospaced font, however every way I 've tried ends up applying the same to the uuencode command, of which corrupts the binary data of the tar file.

I've been trying to apply the Content-Type: text/html only to the text from the logfile, however it's applying it to everything, corrupting the tar binary data.

• I'm currently using cat to get the logfile output in the email... is there a different command and code block that could be utilized to get the text output to be monospaced, since configuring the email for text/html content doesn't work when attaching the tar file?

 

[Xyrgh]

The version on GitHub supports serial number up to 18 characters long. How long are your serial numbers?

I've got an (old) Corsair SSD that has a serial number that is 20 digits. Don't worry too much about it, it's an edge case, happy to try @zoomzoom's fix below, rather than changing the script for everyone. Not sure what new Corsair SSD's are like though.

@Xyrgh If you need to extend past the 18, and you haven't customized the script, you'll need to adjust following lines equally 68 - 72 & 116, as well as 108, specifically |%-18s|

Thanks mate, will try this fix and see what happens.

 

[Raphaël Bleuse]

This code returns just the hour portion of strings of the form "Xh+XXm+XX.XXXs"... but I haven't figured out how to make it work in the script, yet.

Code:

onHours=$(echo ${onHours} | rev | sed 's/.*+//' | rev | sed 's/\h//g')

EDIT: My awk skills are a little rusty, but I've figured out a patch. Replace this line of code:

Code:

/Power_On_Hours/{onHours=$10} \

with this code:

Code:

 /Power_On_Hours/{split($10,a,"+");sub(/h/,"",a[1]);onHours=a[1];} \

This leaves the standard integer 'Power_On_Hours' unchanged, so it won't adversely affect anyone else's scripts. It will split your SSD's 'Power_on_Hours_and_MSec' string into substrings as delimited by the '+' character, discarding all but the first substring, which gives us the hours with an 'h' suffix. It strips off this extraneous 'h', and - voila! - we have the simple integer value we desire!

Have you tried forcing the formatting of the raw value?
I'm asking this because the option -v 9,msec24hour32 provides this exact format on my server. Maybe trying -v 9,raw48 would force the format?

 

[Raphaël Bleuse]

Very nice thread, I found a lot of useful information here :)
And, of course, I did my own version of some scripts: I notably modified the S.M.A.R.T. one to mark the field(s) triggering the warning/critical status. One may find it here: https://gitlab.com/snippets/1660607
As I changed the structure a lot (removed the emailing part, created functions,…), if you only are interested by the marking part you can replace the

Code:

if (temp > tempCrit || reAlloc > sectorsCrit || pending > sectorsCrit || offlineUnc > sectorsCrit)
	device=device " " critSymbol;
else if (temp > tempWarn || reAlloc > 0 || pending > 0 || offlineUnc > 0 || testAge > testAgeWarn)
	device=device " " warnSymbol;

by the content of the snippet between lines 98 and 135 (look for # set warning/critical flags).

 

[tfast500]

Spearfoot is there anyway you can easily fix the get_hdd_temps.sh to support more than 10cpus my output looks like:

Code:

=== CPU (24) ===
CPU 0:  40C
CPU 1:  43C
CPU 2:  40C
CPU 3:  39C
CPU 4:  39C
CPU 5:  40C
CPU 6:  35C
CPU 7:  33C
CPU 8:  34C
CPU 9:  34C
CPU 10:   3C
CPU 11:   3C
CPU 12:   3C
CPU 13:   4C
CPU 14:   3C
CPU 15:   3C
CPU 16:   3C
CPU 17:   3C
CPU 18:   4C
CPU 19:   3C
CPU 20:   3C
CPU 21:   3C
CPU 22:   3C
CPU 23:   3C

=== DRIVES ===
  da0:  28C Western Digital Red WD-WMC4N0325860
  da1:  27C Western Digital Red WD-WMC4N0477619
  da2:  29C Western Digital Red WD-WCC4N4HUC0P9
  da3:  28C Western Digital Red WD-WCC4N4HUCJ2L
  da4:  27C Western Digital Red WD-WCC4N0YU03CV
  da5:  29C Western Digital Red WD-WMC4N0442863
  da6:  27C ST3000VN007-2E4166   W6A26NWE
  da7:  26C ST3000VN007-2E4166   W6A269EA

Thanks!