multi_report.sh version for Core and Scale 3.0

[awasb]

IIRC zpool list shows raw space (including parity), while zfs list gives usable space (i.e. post parity calculation).

That's why I prefer zfs list -o space $POOLNAME, since it includes a reasonable multitude of attributes and types.

Code:

root@truenas: ~ # man zfs-list
[...]
     -o property
         A comma-separated list of properties to display.  The property must
         be:
         •   One of the properties described in the Native Properties section
             of zfsprops(7)
         •   A user property
         •   The value name to display the dataset name
         •   The value space to display space usage properties on file systems
             and volumes.  This is a shortcut for specifying
             -o name,avail,used,usedsnap,usedds,usedrefreserv,usedchild -t
             filesystem,volume.

+1 for the zfs list version, Joe!

 

[Ericloewe]

Sure, but they should show the same value for mirrors. We're not even talking about free space, which is always going to be an estimate, we're talking about used space, which has a relatively clear answer.

 

[awasb]

Sorry. Missed the detail. Could it be some "reserved space for this dataset"-tuning?

 

[joeschmuck]

That's why I prefer zfs list -o space $POOLNAME, since it includes a reasonable multitude of attributes and types.

This is my results, they show the same values except zpool provides a percentage value.
farm2=Raidz2 four 6TB drives.

Code:

root@freenas:~ # zfs list -o space farm2
NAME   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
farm2  7.52T  2.89T        0B   14.2M             0B      2.89T
root@freenas:~ # zfs list farm2
NAME    USED  AVAIL     REFER  MOUNTPOINT
farm2  2.89T  7.52T     14.2M  /mnt/farm2
root@freenas:~ # zpool list farm2
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
farm2  21.8T  5.96T  15.8T        -         -     1%    27%  1.00x    ONLINE  /mnt
root@freenas:~ #

So how to display the actual used space is the trick when using mirrors, and only displaying the new calculations when mirrors are assigned. If I need to figure out the configuration of a pool, it is getting very complicated. I'm fairly sure I have an idea on how to do it but that is a lot of work trying to test it.

So, I'm still open to suggestions on how to calculate used space.

 

[joeschmuck]

I'm running the 2.5.3.1 Beta and after updating SCALE from 22.12.4.2 to 23.10.2 multi_report.sh appears to have stopped running daily self tests on NVMe drives. I noticed this because "Last Test Age" kept ticking up until multi_report started showing a warning about it.

When I designed this portion I made an assumption that once smartmontools 7.4 was included, iXsystems would allow setting up smart tests in the GUI. I was wrong.

I just rework this section but I'm tired so I will wait until I get some sleep and test tomorrow. I will send it to you in a PM once it has been tested. If all seems to work please let me know,. I have a few other things to work on and then I will finalize it as version 3.0. (Maybe this weekend) I'm still working on the new GUI configuration and it will not be ready to be released at the same time but oh well, I can get the main program out the door to fix some of the small problems folks are finding.

 

[kae8]

When I designed this portion I made an assumption that once smartmontools 7.4 was included, iXsystems would allow setting up smart tests in the GUI. I was wrong.

I just rework this section but I'm tired so I will wait until I get some sleep and test tomorrow. I will send it to you in a PM once it has been tested. If all seems to work please let me know,. I have a few other things to work on and then I will finalize it as version 3.0. (Maybe this weekend) I'm still working on the new GUI configuration and it will not be ready to be released at the same time but oh well, I can get the main program out the door to fix some of the small problems folks are finding.

Hello, I have the same problem with nvme and Last Test Age. Is it possible to get a fix too?

 

[joeschmuck]

Hello, I have the same problem with nvme and Last Test Age. Is it possible to get a fix too?

I will send you a PM.

 

[Bloodpack]

If anyone is using SCALE 24.x (Dragonfish currently in Beta), and is encrypting email attachments, 7zip will not automatically install as it previously did, at least in the Beta.1 version. I have an updated script but it's in my Beta version right now and IF you are encrypting, ONLY then will 7zip be installed. Contact me and I will forward you a copy. The next formal release will be v3.0 with a few changes, hopefully things will be easier to setup and it will result in more accurate data being reported.

Encrypting your email attachments is not really required since the TrueNAS password file is encrypted by TrueNAS. But I know some people prefer to encrypt everything so 7zip is required for my version (I like it better).

Hello, can you please send me the BETA to get rid of the script trying to install 7zip on Dragonfish-24.04-RC.1

Thank you

Code:

Multi-Report v2.5 dtd:2023-11-26 (TrueNAS Scale 24.04-RC.1)
Installing 7-Zip...
cp: cannot create regular file '/usr/local/bin/7zzs': Read-only file system
ln: failed to create symbolic link '/usr/local/bin/7z': Read-only file system

 

[joeschmuck]

Hello, can you please send me the BETA to get rid of the script trying to install 7zip on Dragonfish-24.04-RC.1

That must be annoying. I am uploading version 3.0 to github, finished it last night. If you do not have Automatic Updates, please use the '-update' switch. I will tell all once it's posted. I need to generate the checksum values and test that it works first. But I've attached a copy here.

 

[joeschmuck]

joeschmuck updated multi_report.sh version for Core and Scale with a new update entry:

Multi-Report v3.0 - Posted

### Changelog:
# V3.0 (30 March 2024)

# Notable issues:
# NVMe - Last SMART Short and Long tests not displaying in text area. It will be solved when Smartmontools 7.4 is installed.
# -- This is not an issue in SCALE 23.10.2 as it has Smartmontools 7.4.
#
# - Fixed checking NVMe drives for if they support Self-tests.
# - Added NVME Short and Long Self-test for smartctl 7.3 and below. Monday through Saturday a Short Test, Sunday a Long Test.
# --- a Long Test, you may disable...

Read the rest of this update entry...

 

[Bloodpack]

joeschmuck updated multi_report.sh version for Core and Scale with a new update entry:

Multi-Report v3.0 - Posted



Read the rest of this update entry...

Thank you very much, i am going to test in a bit.....

 

[joeschmuck]

I have not completed the GUI for the configuration, I know folks will like that if I ever get it completed. Python just bores me so no energy to put into it, even though it feel like I'm almost done, I know I really am not.

 

[Bloodpack]

@joeschmuck

i have tested verion3 and it works fine. Thank you for your work

i have a combination of SATA and SAS drives, for the SAS drives there is not much data in my email, (i was the same on the last version)
the only change from the previous version to v3 is that for the SAS drives it shows now the model number

here is an example of a drive that has no data shown on the chart in the mail ( this data in the code is from the mail Multi-Report Text Section )

Code:

########## NON-SMART status report for sdl drive (N/A : WD-WCAY01422751) ##########
SMARTCTL DATA

=== START OF INFORMATION SECTION ===
Vendor:               WDC
Product:              WD2003FYYS-05TSM
Revision:             NA09
Compliance:           SPC-3
User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Rotation Rate:        7200 rpm
Logical Unit id:      0x500605ba00be9e20
Serial number:        WD-WCAY01422751
Device type:          disk
Transport protocol:   SAS (SPL-4)
Local Time is:        Sun Mar 31 17:59:43 2024 CEST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported

SMART Health Status: OK

Current Drive Temperature:     33 C
Drive Trip Temperature:        60 C

Accumulated power on time, hours:minutes 0:00
Manufactured in week    of year    
Specified cycle count over device lifetime:  65536
Accumulated start-stop cycles:  0
Read defect list: asked for grown list but didn't get it
Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0         0          0          0.000           0
write:         0        0         0         0          0          0.000           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Default           Completed                   -   22056                 - [-   -    -]
# 2  Default           Completed                   -   21889                 - [-   -    -]
# 3  Default           Completed                   -   21721                 - [-   -    -]
# 4  Default           Completed                   -   21553                 - [-   -    -]
# 5  Default           Completed                   -   21414                 - [-   -    -]
# 6  Default           Completed                   -   21385                 - [-   -    -]
# 7  Default           Completed                   -   21217                 - [-   -    -]

Long (extended) Self-test duration: 19140 seconds [5.3 hours]

 

[joeschmuck]

@Bloodpack
Would you please run the script using the switch '-dump emailextra' so I may collect that data and then see what I'm missing? I can obviously see I am missing some data. If I had a SCSI drive (and the controller of course), I would have all that figured out already.

Thank you.

 

[Bloodpack]

@Bloodpack
Would you please run the script using the switch '-dump emailextra' so I may collect that data and then see what I'm missing? I can obviously see I am missing some data. If I had a SCSI drive (and the controller of course), I would have all that figured out already.

Thank you.

Hello,

the SAS drives are in an

Code:

NetAPP DS4246

enclosure and there is an interposer that converts to SATA

Code:

NetApp LSI SAS SATA Interposer P003463-01A P003464-01B FCZBRG457-N145C3GG V291

where should i send the email?

EDIT: just saw the -dump emailextra sends the data automatically to you as well

Thank you

 

[joeschmuck]

Got the data. Thanks.

 

[BarefootWoodworker]

Tested this on TrueNAS core with the expected results that SMART tests for NVME haven't been run.

So, stupid question. . .is one able to update the SMART tools on Core? Or a better question would be, how does one *properly* upgrade the SMART tools on Core?

I know that you can enable the tests in the script, but I'm always curious about other options.

 

[joeschmuck]

So, stupid question. . .is one able to update the SMART tools on Core?

Yes, however TrueNAS does not support the ability to run a Self-test via the GUI. Multi-Report has the ability to run both Short and Long test, assuming your NVMe supports SMART Self-test, they all do not.

How to set it up: Use '-config' and then press the following keys 'A -> N' Now read the questions. Crap, found an error in the questions, left one out (setting Longtest to enable). Until I fix it, you can just forget the Long test or you can edit the multi_report_config.txt file and search for '#SSD/NVMe' and you should see the following bit of configuration data:

Code:

###### SSD/NVMe Specific Settings
WearLevelCrit=9                # Wear Level Alarm Setpoint for WARNING message, 9% is the default.
NVM_Low_Power="false"            # Set the NVMe power level to the minimum setting. This does not mean the NVMe will remain at this power level.
NVM_Daily_Short_Selftest="true"        # This will run a short test on all NVMe drives everyday of the week, IF smartmontools is not 7.4 or greater.
NVM_Weekly_Long_Selftest="false"        # This will run a long test on all NVMe drives on Sunday and ignore the short test run if set, IF smartmontools is not 7.4 or greater.
NVM_Weekly_Long_Selftest_Day="Sun"    # Value must be "Sun, Mon, Tue, Wed, Thu, Fri, Sat".  Default is Sun.
NVM_Smartmontools_74_Override="enable"    # Set to "enable" to override the check for smartmontools 7.4 and force the script to run the SMART tests. Default "disable".
Wait_For_SMART="true"            # "true" will wait for the SMART Self-test timer to complete, the goal being the test completes before the report is generated. Default = "true"
Wait_For_SMART_Short_Duration=130    # Set how long to wait for an NVMe SMART Short Test to pause the script, waiting on the test to complete. Default = 2 minutes 10 seconds (130 seconds)
Wait_For_SMART_Long_Duration=1    # Set how long to wait for an NVMe SMART Long Test to pause the script, waiting on the test to complete. Default = 20 minutes 10 seconds (1200 seconds)
NVMe_Ignore_Invalid_Errors=disable    # Set to "enable" to ignore "Invalid Field in Command" messages.  Google this message to see if you are comfortable ignoring it.

Now change the these values below:

Code:

NVM_Daily_Short_Selftest="true"
NVM_Weekly_Long_Selftest="true"
NVM_Weekly_Long_Selftest_Day="Sun"
NVM_Smartmontools_74_Override="enable"
Wait_For_SMART="true"
Wait_For_SMART_Short_Duration=130
Wait_For_SMART_Long_Duration=1

Save the file and run the script normally from the CLI.

Please let me know if your NVMe actually runs the test.

 

[BarefootWoodworker]

Oh yeah. . .I saw how to do it in the config file and already set that up.

I was just curious about the tools upgrade so that I don't inadvertently leave an option checked when things *do* get upgraded and get run over by a snowball I didn't see coming.

Before I forget. . .awesome script and appreciate the hard work you've put into it.

 

[awasb]

Works great out of the box, Joe! Thanks again!