multi_report.sh version for Core and Scale 3.0

[NugentS]

smartctl -x /dev/sde

attached
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.81+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: SEAGATE Product: ST3000NM0023 Revision: 0006 Compliance: SPC-4 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Logical block size: 512 bytes LU is fully provisioned Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x5000c500625880ab Serial number: Z1Z5JDWE0000C507EFJZ Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Thu Feb 10 03:58:37 2022 GMT SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled Read Cache is: Enabled Writeback Cache is: Disabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Current Drive Temperature: 26 C Drive Trip Temperature: 60 C Manufactured in week 36 of year 2014 Specified cycle count over device lifetime: 10000 Accumulated start-stop cycles: 350 Specified load-unload count over device lifetime: 300000 Accumulated load-unload cycles: 2470 Elements in grown defect list: 0 Vendor (Seagate Cache) information Blocks sent to initiator = 3384446139 Blocks received from initiator = 3874042717 Blocks read from cache and sent to initiator = 1217256065 Number of read and write commands whose size <= segment size = 221550761 Number of read and write commands whose size > segment size = 0 Vendor (Seagate/Hitachi) factory information number of hours powered up = 52227.53 number of minutes until next internal SMART test = 26 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 1038243508 2 0 1038243510 92 41315.255 90 write: 0 0 0 0 0 32778.956 0 verify: 4981 0 0 4981 0 0.000 0 Non-medium error count: 6 [GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on'] SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background long Aborted (by user command) - 52197 - [- - -] # 2 Background short Completed - 52131 - [- - -] # 3 Background long Failed in segment --> - 52080 531 [0x3 0x11 0x0] # 4 Background short Failed in segment --> - 52079 1550955 [0x3 0x11 0x0] # 5 Background long Failed in segment --> - 52078 531 [0x3 0x11 0x0] # 6 Background long Completed - 52067 - [- - -] # 7 Background short Completed - 52062 - [- - -] # 8 Background short Completed - 52035 - [- - -] # 9 Background short Completed - 52010 - [- - -] #10 Background short Completed - 51986 - [- - -] #11 Background short Completed - 51962 - [- - -] #12 Background short Completed - 51938 - [- - -] #13 Background short Completed - 51914 - [- - -] #14 Background short Completed - 51890 - [- - -] #15 Background long Completed - 51870 - [- - -] #16 Background short Completed - 51842 - [- - -] #17 Background short Completed - 51818 - [- - -] #18 Background short Completed - 51794 - [- - -] #19 Background short Completed - 51770 - [- - -] #20 Background short Completed - 51746 - [- - -] Long (extended) Self-test duration: 26000 seconds [433.3 minutes] Background scan results log Status: scan is active Accumulated power on time, hours:minutes 52227:32 [3133652 minutes] Number of background scans performed: 737, scan progress: 0.00% Number of background medium scans performed: 737 # when lba(hex) [sk,asc,ascq] reassign_status 1 52050:25 0000000000c3299e [3,11,0] Recovered via rewrite in-place 2 52050:25 0000000000c3299f [3,11,0] Recovered via rewrite in-place 3 52050:25 0000000000c32ed2 [3,11,0] Recovered via rewrite in-place 4 52050:25 0000000000c32ed3 [3,11,0] Recovered via rewrite in-place 5 52050:26 0000000000c32ed4 [3,11,0] Recovered via rewrite in-place 6 52050:26 0000000000c32ed5 [3,11,0] Recovered via rewrite in-place 7 52050:26 0000000000c332e4 [1,18,1] Recovered via rewrite in-place 8 52050:26 0000000000c33817 [1,18,1] Recovered via rewrite in-place 9 52050:26 0000000000c33818 [3,16,0] Recovered via rewrite in-place 10 52050:26 0000000000c33819 [3,11,0] Recovered via rewrite in-place 11 52050:26 0000000000c3381a [3,11,0] Recovered via rewrite in-place 12 52050:26 0000000000c3415d [3,16,0] Recovered via rewrite in-place 13 52050:26 0000000000c3415e [3,16,0] Recovered via rewrite in-place 14 52050:26 0000000000c3415f [3,11,0] Recovered via rewrite in-place 15 52050:26 0000000000c34aa2 [3,11,0] Recovered via rewrite in-place 16 52050:26 0000000000c34aa3 [3,11,0] Recovered via rewrite in-place 17 52050:26 0000000000c34aa4 [3,11,0] Recovered via rewrite in-place Protocol Specific port log page for SAS SSP relative target port id = 1 generation code = 0 number of phys = 1 phy identifier = 0 attached device type: expander device attached reason: unknown reason: power on negotiated logical link rate: phy enabled; 6 Gbps attached initiator port: ssp=0 stp=0 smp=1 attached target port: ssp=0 stp=0 smp=1 SAS address = 0x5000c500625880a9 attached SAS address = 0x5005076028c343e0 attached phy identifier = 13 Invalid DWORD count = 0 Running disparity error count = 0 Loss of DWORD synchronization = 0 Phy reset problem = 0 Phy event descriptors: Invalid word count: 0 Running disparity error count: 0 Loss of dword synchronization count: 0 Phy reset problem count: 0 relative target port id = 2 generation code = 0 number of phys = 1 phy identifier = 1 attached device type: no device attached attached reason: unknown reason: unknown negotiated logical link rate: phy enabled; unknown attached initiator port: ssp=0 stp=0 smp=0 attached target port: ssp=0 stp=0 smp=0 SAS address = 0x5000c500625880aa attached SAS address = 0x0 attached phy identifier = 0 Invalid DWORD count = 0 Running disparity error count = 0 Loss of DWORD synchronization = 0 Phy reset problem = 0 Phy event descriptors: Invalid word count: 0 Running disparity error count: 0 Loss of dword synchronization count: 0 Phy reset problem count: 0

 

[joeschmuck]

LOL, that was for the SAS drive, I was looking for the DOM drive data. Specifically looking to see if it would report the true power on hours for the SMART Short/Long tests. The SAS drive issue was just the first line being overcome stating Available. I could work through that I'm sure.

 

[NugentS]

Sorry. I have logged a ticket with SMC re what the stats actually mean

root@scalenas[~]# smartctl -x /dev/sdq smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.81+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Supermicro SATA DOM (SuperDOM) Device Model: SuperMicro SSD Serial Number: SMC0515D93419AI31160 LU WWN Device Id: 5 15d934 19a000160 Firmware Version: SOB20R User Capacity: 15,837,691,904 bytes [15.8 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device TRIM Command: Available, deterministic, zeroed Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Feb 11 14:55:10 2022 GMT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Disabled APM feature is: Unavailable Rd look-ahead is: Enabled Write cache is: Enabled DSN feature is: Unavailable ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Unavailable === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x02) Offline data collection activity was completed without error. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 0) seconds. Offline data collection capabilities: (0x71) SMART execute Offline immediate. No Auto Offline data collection support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0002) Does not save SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 10) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SMART Attributes Data Structure revision number: 1 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate ------ 100 100 000 - 0 5 Reallocated_Sector_Ct ------ 100 100 000 - 0 9 Power_On_Hours ------ 100 100 000 - 12734 12 Power_Cycle_Count ------ 100 100 000 - 149 15 User_Cpcty_Sector_Cnt ------ 100 100 000 - 30932992 160 Not_In_Use ------ 100 100 000 - 0 161 Not_In_Use ------ 100 100 000 - 156 163 Not_In_Use ------ 100 100 000 - 10 164 Not_In_Use ------ 100 100 000 - 22123 165 Not_In_Use ------ 100 100 000 - 45 166 Minimum_PE_Cycles_TLC ------ 100 100 000 - 1 167 Not_In_Use ------ 100 100 000 - 10 168 Maximum_PE_Cycles_TLC ------ 100 100 000 - 3000 172 Unknown_Attribute ------ 100 100 000 - 0 175 Program_Fail_Count_Chip ------ 100 100 000 - 10 177 Wear_Leveling_Count ------ 100 100 000 - 0 181 Program_Fail_Cnt_Total ------ 100 100 000 - 0 187 Reported_Uncorrect ------ 100 100 000 - 0 192 Power-Off_Retract_Count ------ 100 100 000 - 39 194 Temperature_Celsius ------ 100 100 000 - 28 (Min/Max 0/56) 195 Hardware_ECC_Recovered ------ 100 100 000 - 87336 197 Current_Pending_Sector ------ 100 100 000 - 0 198 Offline_Uncorrectable ------ 100 100 000 - 0 199 UDMA_CRC_Error_Count ------ 100 100 000 - 0 231 SSD_Life_Left ------ 100 100 000 - 100 232 Available_Reservd_Space ------ 100 100 000 - 100 233 NAND_Writes_1GiB ------ 100 100 000 - 172 241 Lifetime_Writes_GiB ------ 100 100 000 - 70 242 Lifetime_Reads_GiB ------ 100 100 000 - 354 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 GPL,SL R/O 1 Summary SMART error log 0x02 GPL,SL R/O 1 Comprehensive SMART error log 0x03 GPL,SL R/O 1 Ext. Comprehensive SMART error log 0x04 GPL,SL R/O 8 Device Statistics log 0x06 GPL,SL R/O 1 SMART self-test log 0x07 GPL,SL R/O 1 Extended self-test log 0x09 GPL,SL R/W 1 Selective self-test log 0x10 GPL,SL R/O 1 NCQ Command Error log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log SMART Extended Comprehensive Error Log Version: 1 (1 sectors) No Errors Logged SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 12694 - # 2 Extended offline Completed without error 00% 12647 - # 3 Extended offline Completed without error 00% 12623 - # 4 Short offline Completed without error 00% 12608 - # 5 Short offline Completed without error 00% 12599 - # 6 Offline Completed without error 00% 12552 - # 7 Offline Completed without error 00% 12551 - # 8 Extended offline Completed without error 00% 12535 - # 9 Extended offline Completed without error 00% 12535 - #10 Short offline Completed without error 00% 12534 - #11 Short offline Completed without error 00% 12506 - #12 Short offline Completed without error 00% 12481 - #13 Offline Completed without error 00% 12457 - #14 Offline Completed without error 00% 12288 - #15 Short offline Completed without error 00% 0 - #16 Short offline Completed without error 00% 97 - #17 Short offline Completed without error 00% 12361 - #18 Short offline Completed without error 00% 12337 - #19 Extended offline Completed without error 00% 12335 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Commands not supported Device Statistics (GP Log 0x04) Page Offset Size Value Flags Description 0x01 ===== = = === == General Statistics (rev 2) == 0x01 0x008 4 149 --- Lifetime Power-On Resets 0x01 0x010 4 12734 --- Power-on Hours 0x01 0x018 6 147450403 --- Logical Sectors Written 0x01 0x020 6 4306703 --- Number of Write Commands 0x01 0x028 6 743366082 --- Logical Sectors Read 0x01 0x030 6 7340200 --- Number of Read Commands 0x02 ===== = = === == Free-Fall Statistics (empty) == 0x03 ===== = = === == Rotating Media Statistics (empty) == 0x04 ===== = = === == General Errors Statistics (rev 1) == 0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors 0x04 0x010 4 39 --- Resets Between Cmd Acceptance and Completion 0x05 ===== = = === == Temperature Statistics (empty) == 0x06 ===== = = === == Transport Statistics (rev 1) == 0x06 0x008 4 1446 --- Number of Hardware Resets 0x06 0x018 4 0 --- Number of Interface CRC Errors 0x07 ===== = = === == Solid State Device Statistics (rev 1) == 0x07 0x008 1 0 --- Percentage Used Endurance Indicator |||_ C monitored condition met ||__ D supports DSN |___ N normalized value Pending Defects log (GP Log 0x0c) not supported SATA Phy Event Counters (GP Log 0x11) not supported

 

[joeschmuck]

@NugentS
Well that was enlightening. Right now I'm not sure if the issue is the DOM or smartctl definition file. Using smartctl -x provided the correct data. You could send an email to Bruce or Christian with both complete reports of -a and -x and see what they have to say about it. They do accept emails, I have chatted with Bruce many years ago and it was good. I'm leaning towards smartctl being the issue so with that said, I do not want to modify the script for a one off case. I have created a personalized version for you (using -x vice -l) that "should" fix the SAS drive issue and the DOM issue but I'm curious if I broke something else in the process. I "may" have fixed the UDMA_CRC_Errors too, I found two typos (one formatting error, one true typo) but not certain they fixed it, you tell me.

Cheers,
-Joe

 

[joeschmuck]

Multi Report Version 1.4c is a greatly enhanced version of the previous versions of Multi Report. New features include:

-Improvements to run on FreeNAS/TrueNAS Core (FreeBSD) and TrueNAS Scale (Debian).
-Fully Customizable Reports in Email.
-Creates a report for SMART and Non-SMART drives.
-Updated Configuration Backup to select day the backup is sent vice every run of the script.
-Added Statistical Data Collection (in CSV) of the drives to be used for drive trend analysis. Also delivered via email as the user configures the system.
-Added “-s” switch to aid in statistical data collection without sending any email. This is great for high frequency data collection.
-Added corrections for UDMA CRC Errors.
-Added corrections for Multi Zone Errors.
-Added Drive Ignore List.

In the zip file is the script and a PDF of the instructions. I think anyone who uses the older scripts will like the changes. Please test it out and provide me feedback. If there are any changes to be made, let me know sooner than later while the coding is fresh in my mind. But please understand that I cannot test every variant, that is why I need people to tell me what works and more importantly what doesn't work. And of course, if you can improve the script, please do so. In my mind this is a community effort.

Cheers.
-Joe

EDIT: Removed attachment, need to make a few changes that got dropped. Be back later today.

 

[TooMuchData]

TooMuchData updated multi_report.sh versions for Core and Scale with a new update entry:

Extensive Improvements and fixes by joeschmuck

Joeschmuck for President!

Read the rest of this update entry...

 

[sretalla]

Great that there's a new version... can't wait to try it out... but 382 bytes seems far too few to make a good script, no?

 

[mistermanko]

Great that there's a new version... can't wait to try it out... but 382 bytes seems far too few to make a good script, no?

Yep....

Code:

SSD_Device_ID="true"
SSD_Serial_Number="true"
SSD_Model_Number="true"
SSD_Capacity="true"
SSD_SMART_Status="true"
SSD_Drive_Temp="true"
SSD_Power_On_Hours="true"
SSD_Wear_Level="true"
SSD_Reallocated_Sectors="true"
SSD_Reallocated_Events="true"
SSD_Pending_Sectors="true"
SSD_Offline_Uncorrectable="true"
SSD_UDMA_CRC_Errors="true"
SSD_Last_Test_Age="true"
SSD_Last_Test_Type="true"

 

[joeschmuck]

Great that there's a new version... can't wait to try it out... but 382 bytes seems far too few to make a good script, no?

That was the Stealth Version. I will get @TooMuchData to repost the code. Here is a copy for you to try out. You may massage it however you desire and if you find problems, just PM me and if you fix a problem, PM me with the problem and fix please. I added an updated columns mod version to make a minor change in the width of the columns, it looks a little more condensed (thanks @TooMuchData )

 

[TooMuchData]

TooMuchData updated multi_report.sh versions for Core and Scale with a new update entry:

Sorry. Bad file previously posted.

Just a wee bit of dain bramage.

Read the rest of this update entry...

 

[sretalla]

Worked so far on my first test on CORE.

Great work to have made it so detailed and configurable!

 

[mistermanko]

Seconded. Is working smoothly so far on my CORE install. Though I had to throw out the SSD reporting because my SSDs seem to have weird SMART errors that don't make any sense. Maybe also because they are attached to my HBA, I don't know. How useful is SSD SMART-data anyway?

 

[joeschmuck]

How useful is SSD SMART-data anyway?

Well... Guess that depends on the end user. If you want me to see what is going on with the SMART data and if I can fix it up for you just PM me with the output of your results and the full output of smartctl -a for the problem drive(s). It might be a simple fix that others could benefit from. If it is, I will post the update in this discussion thread and eventually have another formal revision. The fix could be also just for you if it would break the normal script for others. My only limitation is not having the hardware you may have to test the fix and sometimes it's a guess which column the data we want is in. I could make a script to dump the data by column but that sounds like work.

Great work to have made it so detailed and configurable!

I'm all ears if someone has a way to simplify/improve it. I can't say that I have a lot of free time but sometimes it's nice to learn a bit more about stupid AWK that I really dislike due to my own ignorance, but learning is learning.

 

[sretalla]

I'm getting this now after a few successful runs from the CLI, but with cron:

/mnt/vol6/scripts/multi_report.sh: 402: Syntax error: redirection unexpected (expecting word)
/mnt/vol6/scripts/multi_report.sh: 395: Syntax error: Error in command substitution

 

[joeschmuck]

I'm getting this now after a few successful runs from the CLI, but with cron:

/mnt/vol6/scripts/multi_report.sh: 402: Syntax error: redirection unexpected (expecting word)
/mnt/vol6/scripts/multi_report.sh: 395: Syntax error: Error in command substitution

Very odd, you would think that if it worked fine the first few times, it would continue working. What is odd is that line 395 checks if we are running Linux or not, I do not see any redirection. And line 402 is in the Ignore List section for FreeBSD (Core). So I'm making an assumption that you made some changes to the code, meaning you added/removed a line or two so that the line numbers I'm looking at do not correlate to the line numbers of your running script. That isn't really a big issue except I do not know what your line 402 and 395 are.

I would recommend rolling back to the original script and make sure it's working. Then add any configuration changes and retest one or two items at a time to identify the issue. And if you recently made a change to the script, go back and examine the script changes, ensure you used proper punctuation, it is the quickest way to mess up a BASH script. I really hope you find the issue. If not, you can PM me your entire script and I will run it on my machine and debug it, just change your email address so i don't start sending you massive emails .

 

[sretalla]

Very odd, you would think that if it worked fine the first few times, it would continue working

Perhaps even more odd... it still runs fine when I launch it from CLI directly... seems it's just the cron task that fails like that.

395 is:
smartdrives=$(for drive in $(sysctl -n kern.disks); do

402 is:
IFS=',' read -ra ADDR <<< "$IGNOREDRIVES"


I guess I removed some of the lines about ignoring some drives at the beginning.

 

[sretalla]

OK, so it's my fault... I had been running the previous version with "/bin/sh /mnt/path/script.sh" under cron due to something historic that had caused problems with zsh default shell.

Removing that (/bin/sh), it's fine now.

 

[joeschmuck]

OK, so it's my fault... I had been running the previous version with "/bin/sh /mnt/path/script.sh" under cron due to something historic that had caused problems with zsh default shell.

Removing that (/bin/sh), it's fine now.

Glad you found the issue and it was not me .

I guess I removed some of the lines about ignoring some drives at the beginning.

That is fine, it just makes is a challenge to troubleshoot.

 

[mistermanko]

I've stumbled over an interesting failure. The cronjob runs fine but the logs show trouble with the date conversion:

Code:

Failed conversion of ``19-on-Sat_Mar'' using format ``%Y-%b-%e_%H:%M:%S''
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
            [-I[date | hours | minutes | seconds]]
            [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
Executed CronTask - /mnt/tank/scripts/Multi_Report_Script/run_multi_report.sh > /dev/null: Failed conversion of ``19-on-Sat_Mar'' using format ``%Y-%b-%e_%H:%M:%S''
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds|file] [-t west] [-v[+|-]val[ymwdHMS]]
            [-I[date | hours | minutes | seconds]]
            [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]

The mail looks like that. I figure the scrub on tank wasn't finished before sending.

Pool Name	Status	Read Errors	Write Errors	Cksum Errors	Used %	Scrub Repaired Bytes	Scrub Errors	Last Scrub Age	Last Scrub Duration
ZPool Status Report Summary
boot-pool	ONLINE	0	0	0	3%	0	0	4	00:00:07
jails	ONLINE	0	0	0	10%	0	0	2	00:04:00
meta_files	ONLINE	0	0	0	1%	0	0	14	00:00:05
tank	ONLINE	0	0	0	64%	0	00:16:33	19073	1

 

[joeschmuck]

I've stumbled over an interesting failure. The cronjob runs fine but the logs show trouble with the date conversion:

Interesting. Out of curiosity, did you change anything like the date format of the system and language? Anything which might change the way the system handles date/time functions? I'm not sure why that would cause this, I would expect the date function to just work properly regardless. i do not recall using the underscore in the date section. And while it looks like you are running TrueNAS 12.0-U8 (Core), I do want to verify you are running Core. The date function quoted above is for Core and I would hate it if the code failed to recognize you were really running Scale.