Github repository for FreeNAS scripts, including disk burnin and rsync support

Github repository for FreeNAS scripts, including disk burnin and rsync support

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Spearfoot submitted a new resource:

Github repository for FreeNAS scripts, including disk burnin - Updated versions of BiduleOhm's handy scripts

I've created two repositories on GitHub: one containing FreeNAS-related scripts and another for a disk burnin script. These are all POSIX-compliant shell scripts that I've 'linted' at www.shellcheck.net, so they're free of syntax errors and all seem to work well. I use slightly modified version on my systems.

FreeNAS-scripts includes my versions of @Bidule0hm's code from his...

Read more about this resource...
 

melloa

Wizard
Joined
May 22, 2016
Messages
1,749
Thank you sir.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I've added some optional refinements to save_config.sh. These options allow you to:
  • Receive an email whenever the script runs
  • Backup the host ESXi server's configuration, if you're running FreeNAS as a VM
To enable the email option, edit the script and specify your email address:
Code:
email=your_account@your_mail_server
To back up the ESXi host's configuration, edit the script and specify its short hostname:
Code:
esxihost=felix
This ESXi host doesn't have to be the host where your FreeNAS VM runs - it could be any ESXi host on your network -- but I wrote this specifically for use on All-In-One systems where that's the case. The ESXi short and long hostnames should resolve on your network; in my case these are 'felix' and 'felix.spearfoot.net', which both resolve to the same IP.

The email status message looks like this:

save-config-status-email.jpg
 

melloa

Wizard
Joined
May 22, 2016
Messages
1,749
Great job sir :)
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Protip: If you use the update feature in the Resources section, updates will be posted here, too, and there's a nice changelog that people can review in one go.

Though I guess the latter is somewhat redundant in this case, with the git repo and all...
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478

Vito Reiter

Wise in the Ways of Science
Joined
Jan 18, 2017
Messages
232
Good stuff man, this should come in handy for the future :)
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Spearfoot updated Github repository for FreeNAS scripts, including disk burnin with a new update entry:

get_hdd_temp.sh displays drive capacity and fomatting handles 10 or more CPUs

I've posted an updated script with a few formatting changes. It displays the drive temperature, capacity, serial number, and model/family. The CPU temperature output now accomodates systems with 10 or more CPUs.

The output looks like this (note that I'm running FreeNAS as a VM and the actual CPU temperatures aren't available, as shown):
Code:
[root@bandit] ~/scripts/git/FreeNAS-scripts# ./get_hdd_temp.sh
=== CPU (4) ===
CPU  0: --n/a--
CPU  1: --n/a--
CPU  2: --n/a--
CPU  3: --n/a--

===...

Read the rest of this update entry...
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Spearfoot updated Github repository for FreeNAS scripts, including disk burnin with a new update entry:

Disk burnin script modified to poll for actual SMART test completion

I've discovered that disks don't always report accurate values for the short and extended test duration, often by a large factor. This meant that, in some cases, the burnin script would finish even though the SMART test(s) hadn't run to completion.

I've modified disk-burnin.sh so that it now sleeps for the test duration specified by the disk, after which - instead of blithely assuming the test finished - it then polls the disk, checking for either failure or actual completion...

Read the rest of this update entry...
 

Mugiwara

Dabbler
Joined
Apr 16, 2014
Messages
36
Hey Spearfoot,

Huge fan of your scripts, they are great. When I am trying to run the burnin script, however, I am getting a werid error about unit converion. Any ideas? I am running
FreeNAS-11.0-RC4 (ccc181927) with a striaght clone from github. Looking at the code, Im not sure why its throwing the exception.


Code:
root@freenas:~/disk-burnin-and-testing # ls -ltra
total 16
drwxr--r--  7 root  wheel	 18 Jun  7 17:06 ..
-rw-r--r--  1 root  wheel   5759 Jun  7 17:06 README.md
-rwxr--r--  1 root  wheel  11660 Jun  7 17:06 disk-burnin.sh
-rw-r--r--  1 root  wheel   1070 Jun  7 17:06 license.txt
drwxr-xr-x  8 root  wheel	 13 Jun  7 17:06 .git
drwxr-xr-x  3 root  wheel	  6 Jun  7 17:15 .
root@freenas:~/disk-burnin-and-testing # ./disk-burnin.sh da0
./disk-burnin.sh: arithmetic expression: variable conversion error: "Extended_Test_Minutes*60"
root@andrewserver:~/disk-burnin-and-testing # ./disk-burnin.sh da1
./disk-burnin.sh: arithmetic expression: variable conversion error: "Extended_Test_Minutes*60"

These are western digital 8TB drives connected to LSI HBAs through a supermicro backplane
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Hey Spearfoot,

Huge fan of your scripts, they are great. When I am trying to run the burnin script, however, I am getting a werid error about unit converion. Any ideas? I am running
FreeNAS-11.0-RC4 (ccc181927) with a striaght clone from github. Looking at the code, Im not sure why its throwing the exception.


Code:
root@freenas:~/disk-burnin-and-testing # ls -ltra
total 16
drwxr--r--  7 root  wheel	 18 Jun  7 17:06 ..
-rw-r--r--  1 root  wheel   5759 Jun  7 17:06 README.md
-rwxr--r--  1 root  wheel  11660 Jun  7 17:06 disk-burnin.sh
-rw-r--r--  1 root  wheel   1070 Jun  7 17:06 license.txt
drwxr-xr-x  8 root  wheel	 13 Jun  7 17:06 .git
drwxr-xr-x  3 root  wheel	  6 Jun  7 17:15 .
root@freenas:~/disk-burnin-and-testing # ./disk-burnin.sh da0
./disk-burnin.sh: arithmetic expression: variable conversion error: "Extended_Test_Minutes*60"
root@andrewserver:~/disk-burnin-and-testing # ./disk-burnin.sh da1
./disk-burnin.sh: arithmetic expression: variable conversion error: "Extended_Test_Minutes*60"

These are western digital 8TB drives connected to LSI HBAs through a supermicro backplane
Thank you, sir.

Would you please post the output of smartctl -c /dev/da0 between code tags?
 

Mugiwara

Dabbler
Joined
Apr 16, 2014
Messages
36
Sure, thanks for taking a look!
Code:
root@andrewserver:~ # smartctl -c /dev/da0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
										was never started.
										Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0) The previous self-test routine completed
										without error or no self-test has ever
										been run.
Total time to complete Offline
data collection:				(  101) seconds.
Offline data collection
capabilities:					(0x5b) SMART execute Offline immediate.
										Auto Offline data collection on/off support.
										Suspend Offline collection upon new
										command.
										Offline surface scan supported.
										Self-test supported.
										No Conveyance Self-test supported.
										Selective Self-test supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x01) Error logging supported.
										General Purpose Logging supported.
Short self-test routine
recommended polling time:		(   2) minutes.
Extended self-test routine
recommended polling time:		(1242) minutes.
SCT capabilities:			  (0x003d) SCT Status supported.
										SCT Error Recovery Control supported.
										SCT Feature Control supported.
										SCT Data Table supported.



And for a list of drives:

Code:
root@freenas:~ # sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 15.00.00.00 (2016.11.21)
Copyright (c) 2009-2016 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type						 : SAS3008
  BIOS version							: 0.00.00.00
  Firmware version						: 14.00.00.00
  Channel description					 : 1 Serial Attached SCSI
  Initiator ID							: 0
  Maximum physical devices				: 1023
  Concurrent commands supported		   : 10240
  Slot									: Unknown
  Segment								 : 0
  Bus									 : 1
  Device								  : 0
  Function								: 0
  RAID Support							: No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 0
  SAS Address							 : 4433221-1-0200-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 1
  SAS Address							 : 4433221-1-0300-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 2
  SAS Address							 : 4433221-1-0100-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 3
  SAS Address							 : 4433221-1-0000-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 4
  SAS Address							 : 4433221-1-0600-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 5
  SAS Address							 : 4433221-1-0700-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 6
  SAS Address							 : 4433221-1-0500-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 7
  SAS Address							 : 4433221-1-0400-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#							  : 1
  Logical ID							  : 50030480:1dea8901
  Numslots								: 8
  StartSlot							   : 0
------------------------------------------------------------------------


root@freenas:~ # sas2ircu 0 display
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type						 : SAS2008
  BIOS version							: 0.00.00.00
  Firmware version						: 20.00.07.00
  Channel description					 : 1 Serial Attached SCSI
  Initiator ID							: 0
  Maximum physical devices				: 255
  Concurrent commands supported		   : 3432
  Slot									: Unknown
  Segment								 : 0
  Bus									 : 2
  Device								  : 0
  Function								: 0
  RAID Support							: No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 0
  SAS Address							 : 4433221-1-0300-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 1
  SAS Address							 : 4433221-1-0200-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 2
  SAS Address							 : 4433221-1-0100-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 3
  SAS Address							 : 4433221-1-0000-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 4
  SAS Address							 : 4433221-1-0700-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 5
  SAS Address							 : 4433221-1-0600-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 6
  SAS Address							 : 4433221-1-0500-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 1
  Slot #								  : 7
  SAS Address							 : 4433221-1-0400-0000
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 7630885/15628053167
  Manufacturer							: ATA
  Model Number							: WDC WD80EFZX-68U
  Firmware Revision					   : 0A83
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#							  : 1
  Logical ID							  : 500605b0:013ca580
  Numslots								: 8
  StartSlot							   : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.

root@freenas:~ # ls -ltr /dev/*da* | sort
crw-r-----  1 root  operator  0x60 Jun  7 16:12 /dev/ada0
crw-r-----  1 root  operator  0x75 Jun  7 16:12 /dev/ada1
crw-r-----  1 root  operator  0x7c Jun  7 16:12 /dev/da0
crw-r-----  1 root  operator  0x7d Jun  7 16:12 /dev/da1
crw-r-----  1 root  operator  0x7e Jun  7 16:12 /dev/da2
crw-r-----  1 root  operator  0x7f Jun  7 16:12 /dev/da3
crw-r-----  1 root  operator  0x80 Jun  7 16:12 /dev/da4
crw-r-----  1 root  operator  0x81 Jun  7 16:12 /dev/da5
crw-r-----  1 root  operator  0x82 Jun  7 16:12 /dev/da6
crw-r-----  1 root  operator  0x83 Jun  7 16:12 /dev/da7
crw-r-----  1 root  operator  0x84 Jun  7 16:12 /dev/da8
crw-r-----  1 root  operator  0x85 Jun  7 16:12 /dev/da9
crw-r-----  1 root  operator  0x86 Jun  7 16:12 /dev/da10
crw-r-----  1 root  operator  0x87 Jun  7 16:12 /dev/da11
crw-r-----  1 root  operator  0x88 Jun  7 16:12 /dev/da12
crw-r-----  1 root  operator  0x89 Jun  7 16:12 /dev/da13
crw-r-----  1 root  operator  0x8a Jun  7 16:12 /dev/da14
crw-r-----  1 root  operator  0x8b Jun  7 16:12 /dev/da15
crw-r-----  1 root  operator  0x8c Jun  7 16:12 /dev/da16



The ada devices are two samsung SSDs for jails and such, the das are the actual WDC red storage drives
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I believe I see the problem... Busy this morning, will investigate later today.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
@Mugiwara : I've uploaded a modified disk burn-in script to the Github repository that should correctly parse the extended test duration for your (huge!) drive. Let me know if you still have problems with it.
 

Mugiwara

Dabbler
Joined
Apr 16, 2014
Messages
36
Awesome, thanks a ton, running like a charm now!

Code:
root@freenas:~/disk-burnin-and-testing # ./disk-burnin.sh da0
+-----------------------------------------------------------------------------
+ Started burn-in of /dev/da0 : Thu Jun  8 11:07:07 PDT 2017
+-----------------------------------------------------------------------------
Host: freenas
Drive Model: WDC_WD80EFZX-68UW8N0
Serial Number: VK0W2LRY
Short test duration: 2 minutes
Short test sleep duration: 120 seconds
Extended test duration: 1242 minutes
Extended test sleep duration: 74520 seconds
Log file: ./burnin-WDC_WD80EFZX-68UW8N0_VK0W2LRY.log
Bad blocks file: ./burnin-WDC_WD80EFZX-68UW8N0_VK0W2LRY.bb
+-----------------------------------------------------------------------------
+ Run SMART short test on drive /dev/da0: Thu Jun  8 11:07:07 PDT 2017
+-----------------------------------------------------------------------------
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Thu Jun  8 11:09:07 2017

Use smartctl -X to abort test.
Short test started, sleeping 120 seconds until it finishes



Is there a recommend way to run all 16 drives in parallel, or should I just open a bunch of SSH sessions and kick them off? I'm not well versed enough in shell scripting, a for loop would wait for each one to finish before beginning the enxt one Im assuming, which I probably don't want to do or it will take a month.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,455
Top