FreeNas reports is giving me errors: what does it mean and how to fix it?

[philjans]

Hi,
I have a freenas which emails me reports everyday.
I just received this one: what does it mean and what the actions to take?

Code:

_________________
Checking status of zfs pools:
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
Tank          19.9T  8.84T  11.1T        -         -    44%    44%  1.00x  DEGRADED  /mnt
freenas-boot   136G  2.07G   134G        -         -      -     1%  1.00x  ONLINE  -

  pool: Tank
state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
                attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
                using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 938G in 0 days 12:37:58 with 0 errors on Sun Sep 19 15:16:05 2021
config:

                NAME                                              STATE     READ WRITE CKSUM
                Tank                                              DEGRADED     0     0     0
                  mirror-0                                        ONLINE       0     0     0
                    gptid/8ea73aa3-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                    gptid/8fecdbcb-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                  mirror-1                                        ONLINE       0     0     0
                    gptid/9179f265-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                    gptid/92c84653-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                  mirror-2                                        ONLINE       0     0     0
                    gptid/947550c1-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                    gptid/9622714e-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                  mirror-3                                        ONLINE       0     0     0
                    gptid/97be8f2f-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                    gptid/996a9367-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                  mirror-4                                        ONLINE       0     0     0
                    gptid/9c288892-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                    gptid/9ebf5f86-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                  mirror-5                                        DEGRADED     0     0     0
                    gptid/a0b7b6ff-b5d2-11e8-9075-0015174ac980    ONLINE       0     0     0
                    spare-1                                       DEGRADED     0     0    76
                      gptid/a22da98e-b5d2-11e8-9075-0015174ac980  DEGRADED     0     2   137  too many errors
                      gptid/6a9dedd7-c03b-11e8-b67e-0015174ac980  ONLINE       0     0   924
                  mirror-6                                        ONLINE       0     0     0
                    gptid/cc1d7dbd-c023-11e8-8aa7-0015174ac980    ONLINE       0     0     0
                    gptid/cfb3c468-c023-11e8-8aa7-0015174ac980    ONLINE       0     0     0
                  mirror-7                                        ONLINE       0     0     0
                    gptid/d6ec8577-c023-11e8-8aa7-0015174ac980    ONLINE       0     0     0
                    gptid/da7734e1-c023-11e8-8aa7-0015174ac980    ONLINE       0     0     0
                  mirror-8                                        ONLINE       0     0     0
                    gptid/e2086c0e-c023-11e8-8aa7-0015174ac980    ONLINE       0     0     0
                    gptid/e5980f42-c023-11e8-8aa7-0015174ac980    ONLINE       0     0     0
                  mirror-9                                        ONLINE       0     0     0
                    gptid/4c2f9998-c03b-11e8-b67e-0015174ac980    ONLINE       0     0     0
                    gptid/502f9324-c03b-11e8-b67e-0015174ac980    ONLINE       0     0     0
                  mirror-10                                       ONLINE       0     0     0
                    gptid/58e679ec-c03b-11e8-b67e-0015174ac980    ONLINE       0     0     0
                    gptid/6024bad9-c03b-11e8-b67e-0015174ac980    ONLINE       0     0     0
                spares
                  4051645707849655980                             INUSE     was /dev/gptid/6a9dedd7-c03b-11e8-b67e-0015174ac980
                  gptid/74f9933b-c03b-11e8-b67e-0015174ac980      AVAIL 

errors: No known data errors

-- End of daily output --

I found where I can see the pool status.
I have only 1 pool and it seems to be split in 11 Mirroir... which is weird to start with ...
And I see one Mirroir containing the da25p2 degraded but instead of having 2 hd in it, the 2nd hd says it is the "group SPARE" with da23p2 in it. why's that?


All morroir have 2 disks except this one which has 1 disc da25p2 and then a kind of group called SPARE which has 2 hd: da23p2 and da2p2...
What is that?

And how to find which one is defect physically?

They are in 2 Dell powervault md1200 and there doesn't seems to be any "orange" light that shows the defective one like my HP would have shown.
It is one of those 24 ones but without the orange light: how can I locate it!?

Code:

___________________________________
I also had this previously:
titan.xxx.com kernel log messages:
> (da2:mpr1:0:8:0): READ(10). CDB: 28 00 18 39 d7 a0 00 00 68 00
> (da2:mpr1:0:8:0): CAM status: SCSI Status Error
> (da2:mpr1:0:8:0): SCSI status: Check Condition
> (da2:mpr1:0:8:0): SCSI sense: HARDWARE FAILURE asc:32,0 (No defect spare location available)
> (da2:mpr1:0:8:0): Info: 0x1839d7c9
> (da2:mpr1:0:8:0): Field Replaceable Unit: 157
> (da2:mpr1:0:8:0): Actual Retry Count: 157
> (da2:mpr1:0:8:0): Retrying command (per sense data)

-- End of security output --

__________________
New alerts:
* The volume Tank state is DEGRADED: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

Gone alerts:
* The volume Tank state is DEGRADED: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.

Alerts:
* The volume Tank state is DEGRADED: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

 

[philjans]

I just did a smartctl -a on all HD and it is weird because I see with da2 a smart health status FAILURE PREDICTION TRESHOLD EXCEEDED but nothing for the real one that freenas tells me is dead

Is freenas working!?

Code:

root@titi[~]# smartctl -a /dev/da2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST32000444SS
Revision: KS69
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50040ca227b
Serial number: 9WM84EF3
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 29 16:41:18 2021 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Disabled or Not Supported

=== START OF READ SMART DATA SECTION ===
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED: ascq=0x5 [asc=5d, ascq=5]

Current Drive Temperature: 27 C
Drive Trip Temperature: 68 C

Manufactured in week 46 of year 2011
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 49
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 49
Elements in grown defect list: 4089

Vendor (Seagate) cache information
Blocks sent to initiator = 920210870
Blocks received from initiator = 3416573004
Blocks read from cache and sent to initiator = 2972203977
Number of read and write commands whose size <= segment size = 742506700
Number of read and write commands whose size > segment size = 0

Vendor (Seagate/Hitachi) factory information
number of hours powered up = 60035.77
number of minutes until next internal SMART test = 36

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 203945595 470 0 203946065 203946066 205050.056 1
write: 0 0 0 0 0 19593.574 0
verify: 1598533865 911 0 1598534776 1598534776 295372.465 0

Non-medium error count: 12

SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Completed 32 16 - [- - -]
# 2 Background short Completed 32 13 - [- - -]

Long (extended) Self Test duration: 18500 seconds [308.3 minutes]

_________________________________________________________________________________________________________________
da24 (which is really what freenas sees as da23

Code:

root@titi[~]# smartctl -a /dev/da24
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: HGST HUS724020ALA640
Serial Number: PN2161P5G9RNDX
LU WWN Device Id: 5 000cca 24ec46c65
Add. Product Id: DELL▒ EQ
Firmware Version: MF6OAAU0
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Sep 29 16:44:14 2021 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED


General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 28) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 322) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0004 136 136 000 Old_age Offline - 82
3 Spin_Up_Time 0x0007 142 142 024 Pre-fail Always - 387 (Average 489)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 107
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000a 100 100 000 Old_age Always - 0
8 Seek_Time_Performance 0x0004 142 142 000 Old_age Offline - 25
9 Power_On_Hours 0x0012 094 094 000 Old_age Always - 43671
10 Spin_Retry_Count 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 17
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 763
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 763
194 Temperature_Celsius 0x0002 206 206 000 Old_age Always - 29 (Min/Max 18/41)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
223 Load_Retry_Count 0x000a 100 100 000 Old_age Always - 0
241 Total_LBAs_Written 0x0012 100 100 000 Old_age Always - 476438298406
242 Total_LBAs_Read 0x0012 100 100 000 Old_age Always - 317809732030

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short captive Completed without error 00% 0 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Spearfoot]

I just did a smartctl -a on all HD and it is weird because I see with da2 a smart health status FAILURE PREDICTION TRESHOLD EXCEEDED but nothing for the real one that freenas tells me is dead

Is freenas working!?

Code:

root@titi[~]# smartctl -a /dev/da2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST32000444SS
Revision: KS69
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c50040ca227b
Serial number: 9WM84EF3
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 29 16:41:18 2021 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Disabled or Not Supported

=== START OF READ SMART DATA SECTION ===
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED: ascq=0x5 [asc=5d, ascq=5]

Current Drive Temperature: 27 C
Drive Trip Temperature: 68 C

Manufactured in week 46 of year 2011
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 49
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 49
Elements in grown defect list: 4089

Vendor (Seagate) cache information
Blocks sent to initiator = 920210870
Blocks received from initiator = 3416573004
Blocks read from cache and sent to initiator = 2972203977
Number of read and write commands whose size <= segment size = 742506700
Number of read and write commands whose size > segment size = 0

Vendor (Seagate/Hitachi) factory information
number of hours powered up = 60035.77
number of minutes until next internal SMART test = 36

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 203945595 470 0 203946065 203946066 205050.056 1
write: 0 0 0 0 0 19593.574 0
verify: 1598533865 911 0 1598534776 1598534776 295372.465 0

Non-medium error count: 12

SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Completed 32 16 - [- - -]
# 2 Background short Completed 32 13 - [- - -]

Long (extended) Self Test duration: 18500 seconds [308.3 minutes]

_________________________________________________________________________________________________________________
da24 (which is really what freenas sees as da23

Code:

root@titi[~]# smartctl -a /dev/da24
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: HGST HUS724020ALA640
Serial Number: PN2161P5G9RNDX
LU WWN Device Id: 5 000cca 24ec46c65
Add. Product Id: DELL▒ EQ
Firmware Version: MF6OAAU0
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Sep 29 16:44:14 2021 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED


General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 28) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 322) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0004 136 136 000 Old_age Offline - 82
3 Spin_Up_Time 0x0007 142 142 024 Pre-fail Always - 387 (Average 489)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 107
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000a 100 100 000 Old_age Always - 0
8 Seek_Time_Performance 0x0004 142 142 000 Old_age Offline - 25
9 Power_On_Hours 0x0012 094 094 000 Old_age Always - 43671
10 Spin_Retry_Count 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 17
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 763
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 763
194 Temperature_Celsius 0x0002 206 206 000 Old_age Always - 29 (Min/Max 18/41)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
223 Load_Retry_Count 0x000a 100 100 000 Old_age Always - 0
241 Total_LBAs_Written 0x0012 100 100 000 Old_age Always - 476438298406
242 Total_LBAs_Read 0x0012 100 100 000 Old_age Always - 317809732030

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short captive Completed without error 00% 0 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Disk da2 shows a read error and 4089 elements in the grown defects lists.

Disk da23/24 doesn't show anything readily apparent -- but the SMART data don't always give warning when a disk is about to fail.

Have you been running smart tests on these disks? It doesn't look like you have in a very long time, but I could be mistaken.

 

[philjans]

Disk da2 shows a read error and 4089 elements in the grown defects lists.

Disk da23/24 doesn't show anything readily apparent -- but the SMART data don't always give warning when a disk is about to fail.

Have you been running smart tests on these disks? It doesn't look like you have in a very long time, but I could be mistaken.

Hi, thanks for your quick answer,
No I have not run any smart test on any of them.
But :
1- do I have currently a harddrive failure? (does degraded means a HD currently has failed or does it mean it will fail
2- Is it da23 or da2 that are problematic?
3- Why would da2 shows "Online" in the middle of my printscreen but "unavail" at the buttom?
4- Why would those hd shows physically activity lights?
5- Why would this 6th mirroir not be like the others which are 2 hd and instead be 1 regular hd (da25p2) and for the 2nd one the group "SPARE" which included da23p2 and da2p2? I did change a harddrive a while back, I don't remember which it was but could it be the 6th mirroir 2nd drive and it associated SPARE because of that... because have not do any command like "offline" and/or "replace" so many freenas took the first available hd by itself?
6- The only HD that has no greenlight ON is this one

Thanks for you answers. I search every forum I could and it seems no one has any knowledge about freenas so I greatly appreciate any help.
tx

 

[Spearfoot]

Hi, thanks for your quick answer,
No I have not run any smart test on any of them.
But :
1- do I have currently a harddrive failure? (does degraded means a HD currently has failed or does it mean it will fail
2- Is it da23 or da2 that are problematic?
3- Why would da2 shows "Online" in the middle of my printscreen but "unavail" at the buttom?
4- Why would those hd shows physically activity lights?
5- Why would this 6th mirroir not be like the others which are 2 hd and instead be 1 regular hd (da25p2) and for the 2nd one the group "SPARE" which included da23p2 and da2p2? I did change a harddrive a while back, I don't remember which it was but could it be the 6th mirroir 2nd drive and it associated SPARE because of that... because have not do any command like "offline" and/or "replace" so many freenas took the first available hd by itself?
6- The only HD that has no greenlight ON is this one
View attachment 49667
Thanks for you answers. I search every forum I could and it seems no one has any knowledge about freenas so I greatly appreciate any help.
tx

Yes, you have currently failed disks: da2 and da23. It looks like FreeNAS tried to replace one of the failing disk in that mirrored pair with a SPARE -- but the SPARE disk was also bad. But I'm not sure about that. The good news is that one of the mirror disks (da25) is okay, so all you should have to do is replace one of the bad disks; FreeNAS will then resilver the mirrored pair.

After the dust settles, you really ought to set up regular SMART tests: I run short tests once a day and extended tests once a week.

Good luck!

 

[philjans]

Can you explain what you mean by "After the dust settles, you really ought to set up regular SMART tests: I run short tests once a day and extended tests once a week. "
You mean there is no default process that does that already?
I tought SMART was always active and when it finds something: it let the system knows... if not: there is nothing smart in SMART lol
I have a lot of servers here and in none of them anyone setup something manually about SMART.

 

[Samuel Tai]

You forget FreeNAS/TrueNAS operates more than just SATA drives. SAS drives don't grok SMART. As a result, SMART defaults to off out of the box in FreeNAS/TrueNAS. You need to enable the SMART daemon. You need to enroll SMART-capable drives into monitoring. You need to set up periodic jobs to poll SMART data. You need to set the alert level for SMART errors. You need to specify the e-mail address for alerts.

 

[Spearfoot]

Can you explain what you mean by "After the dust settles, you really ought to set up regular SMART tests: I run short tests once a day and extended tests once a week. "
You mean there is no default process that does that already?
I tought SMART was always active and when it finds something: it let the system knows... if not: there is nothing smart in SMART lol
I have a lot of servers here and in none of them anyone setup something manually about SMART.

What I meant was that you should schedule regular SMART tasks to run short & extended tests on all of your SATA & SAS drives.
You can then use something like my smart_report.sh script to track the health of your drives:

GitHub - Spearfoot/FreeNAS-scripts: Handy shell scripts for use on FreeNAS servers

Handy shell scripts for use on FreeNAS servers. Contribute to Spearfoot/FreeNAS-scripts development by creating an account on GitHub.

github.com

The goal is to have advance warning about failing drives, instead of being caught by surprise.

 

[philjans]

You forget FreeNAS/TrueNAS operates more than just SATA drives. SAS drives don't grok SMART. As a result, SMART defaults to off out of the box in FreeNAS/TrueNAS. You need to enable the SMART daemon. You need to enroll SMART-capable drives into monitoring. You need to set up periodic jobs to poll SMART data. You need to set the alert level for SMART errors. You need to specify the e-mail address for alerts.

Thanks for the clarifications: do you have a link that explain how to do that by any chance?
tx!

 

[Etorix]

Yes, you have currently failed disks: da2 and da23. It looks like FreeNAS tried to replace one of the failing disk in that mirrored pair with a SPARE -- but the SPARE disk was also bad. But I'm not sure about that.

That's my interpretation as well. da2 may be "online", but it had over 900 checksum errors so it's really no better than da23.

@philjans If all your drives are ten years old, you should prepare for them to fail in droves in a not-too-distant future. These two are probably just the beginning of your troubles.

Thanks for the clarifications: do you have a link that explain how to do that by any chance?

See section 8.4 of the FreeNAS manual or the corresponding TrueNAS documentation.

 

[philjans]

We were able to turn manually one of the 2 lights per Hd using this command
sesutil locate -u /dev/ses0 dax on or off and then once the defective hd was found: I removed it and replaced it and fix the pool and the spares.
Thank you all for your assistance