RAID5 pool degraded, SMB share visible but empty

[ThaDoctor72]

Hardware is a repurposed DATTO 4 bay with H97N motherboard
4x WD Red (WD60EFAX ) 6TB SATA
1x Samsung 850 Pro 512GB SSD
FreeNAS 11.3 Release P7 (I think, it won't boot so can't verify)

At one time I had a functional ZFS pool but one of the drives "disappeared" for whatever reason and shortly after that another drive started throwing codes and the whole SMB storage became almost unresponsive or massive delays so I rebooted it. The console shows continuously:

Code:

(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 60 08 18 33 68 40 29 01 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: ATA Status Error
(ada2:ahcich2:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
(ada2:ahcich2:0:0:0): RES: 41 10 18 33 68 00 29 01 00 00 00
(ada2:ahcich2:0:0:0): Retrying command
slow spa_sync: started 780 seconds ago, calls 37

The pool is broken up into a few iSCSI stores and a big fat SMB share. (That's where my wife's audio books get backed up to) along with my other systems backups. Obviously I'm more concerned about her stuff because... reasons.

Now when the share does actually show up (if I control C the console to continue boot) the share is visible but there is no content. Windows drive properties shows 13.8TB to be used but the share folder is empty.
Is this a lost cause? Do I need to go explain to her that I lost her data again? (I once did a crappy robocopy /MIR that wiped out her audiobooks before I built the NAS, so I've not in very good standing atm.)

We had a power outage that lasted a long time when we were away from the house, and the batteries eventually failed.

Thanks for your help.

 

[Etorix]

Is this a lost cause?

Not necessarily… but with SMR drives and previous failures it doesn't smell good.
You will likely need a complete set of new drives (CMR!) to rebuild. What's the geometry of the pool? Raidz2, mirrors or… a 4-wide stripe?

I think I'd begin by taking out the data drives and check whether the NAS itself boots and has not been damaged by the power outage.

 

[ThaDoctor72]

With the drives out, NAS does boot. What's could potentially happen by booting the NAS with some (not all, or none) of the drives inserted?

 

[jgreco]

At one time I had a functional ZFS pool but one of the drives "disappeared" for whatever reason and shortly after that another drive started throwing codes and the whole SMB storage became almost unresponsive or massive delays so I rebooted it.

It sounds (and looks) like one of the disks is erroring out. Many consumer grade drives lack TLER/ERC which is a function that prevents the drive from taking excessively long to respond to I/O requests.

It also looks like your system is using generic AHCI SATA controllers, which is good. Your subject says "RAID5", which would be super-bad, as FreeNAS/TrueNAS does not support RAID5, which is typically implemented by a hardware controller of some sort. That is a way to a separate type of catastrophe.

The pool is broken up into a few iSCSI stores and a big fat SMB share.

iSCSI on RAIDZ (which is what I'm guessing you have) is bad, m'kay.

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files. ZFS will allocate long, contiguous stretches of disk...

www.truenas.com

I would expect that the safest path forward here would be to acquire four new (probably larger) non-SMR drives (make sure they are CMR). Then you could try to copy the "slow" drive to one of the new drives, and there's a good chance you might get your pool back. The bad news is that with SMR drives, sometimes the damage is so extensive that you exceed whatever redundacy level you had available (one disk for RAIDZ1, two disks for RAIDZ2, etc). I think that @Etorix was sort of implying that it's a good idea to figure this out now as it might not be worth the expense/effort if you have two disks toasted and only RAIDZ1, for example.

 

[ThaDoctor72]

The iSCSI blobs were 4GB in size, used for Commodore 64 file storage. They were just one of those "because I could" scenarios.
I must admit I bought the WD NAS drives because I assumed they were better because they cost more than the crappy WD Blues I used to buy, and thought in my primitive mind that it would perform better in a NAS environment. The RAID5 explanation is my interpretation of the setup I used, incorporating 4 drives with the available space of 3, assuming the spare would be a parity unit.

The bad drives are still under warranty and can be RMA'd but are you suggesting I can image the existing drives (onto other bare metal units, or save it as an img & recover later) and potentially restore that array once I get new drives installed?
I ask only because I'm apparently grossly unfamiliar with ZFS and True/FreeNAS ops in general.

 

[ThaDoctor72]

Do I wait for it to go through all these "scrubbed nnn blocks..." messages? I've left it alone for a few hours hoping it will self-recover.

 

[jgreco]

The iSCSI blobs were 4GB in size, used for Commodore 64 file storage. They were just one of those "because I could" scenarios.

Just be aware that there is the potential for iSCSI to be VERY space inefficient with RAIDZ, also not performant. Maybe neither of these things is an issue for Commodore 64 file storage, even a CBM 8250 is only about 2 megabytes and they were rare.

I must admit I bought the WD NAS drives because I assumed they were better because they cost more than the crappy WD Blues I used to buy, and thought in my primitive mind that it would perform better in a NAS environment. The RAID5 explanation is my interpretation of the setup I used, incorporating 4 drives with the available space of 3, assuming the spare would be a parity unit.

WD and Seagate screwed everyone with SMR drives (I think I am supposed to qualify this with "in my opinion"). As for RAID5, be aware that RAIDZ1 is NOT RAID5. It sort of slightly resembles RAID5 from the parity protection point of view, but the implementation and mechanism is completely different. Specifically, if you write a 4KB block to your pool, ZFS protects it with a 4KB parity block (100% overhead), which may not be what you expected. The "free space" reported by ZFS is therefore a loose estimate.

The bad drives are still under warranty and can be RMA'd

Good. Apparently WD has been willing to exchange SMR for CMR drives in at least some cases. There's talk about this on the forum somewhere, not with 100% success though.

are you suggesting I can image the existing drives (onto other bare metal units, or save it as an img & recover later) and potentially restore that array once I get new drives installed?

ABSOLUTELY. The question is whether or not there is enough left to rebuild the thing. SMR has a sucky tendency to do simultaneous damage on multiple platters, because SMR is not designed to cope with the crushing amount of writes ZFS will throw at drives during a rebuild.

Do I wait for it to go through all these "scrubbed nnn blocks..." messages? I've left it alone for a few hours hoping it will self-recover.

No advice here. I haven't had the misfortune of having to deal with SMR disks so I mostly have secondhand experience of working through the misfortune of forum users who show up with SMR disks.

 

[ThaDoctor72]

@jgreco thanks so much for your input and guidance. After not-so-patiently waiting all day for the NAS to resume service, it's back online (degraded of course) but at least to the point to where I can start scrambling to get the live data onto my 8TB USB standby.

When I hot-unplugged one of the (remaining 1 of 3) drives not realizing I had pulled the wrong drive I about soiled myself. The DATTO doesn't go left to right for drives 0-3, it's 2,0,3,1 for whatever reason. I was this many years old before I learned what SMR/CMR drives even were.
Knowing how dynamic their operation is, what's the best use case for SMR drives anyway? Single drive bulk storage?

 

[sretalla]

what's the best use case for SMR drives anyway? Single drive bulk storage?

Something that isn't ZFS... things that don't involve continuous writing (exceeding the size of the staging area on the drive).

 

[Etorix]

After not-so-patiently waiting all day for the NAS to resume service, it's back online (degraded of course) but at least to the point to where I can start scrambling to get the live data onto my 8TB USB standby.

Good news. While backing up the most critical data, you may have a look at
zpool status -v and
smartctl -a /dev/ada0 to …ada3
and share the result here (within CODE tags </>, not inline >_) so we can have a look at the damage and its possible cause. But plan to replace the SMR drives with CMR ones in the NAS, regardless of the outcome.
If some SMR drives are still healthy, bulk storage of essentially static data in a desktop (non-ZFS) environment is what would best suit them.

 

[ThaDoctor72]

Status:

Code:

% zpool status -v
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:04 with 0 errors on Wed Jun 21 03:45:04 2023
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada3p2    ONLINE       0     0     0

errors: No known data errors

  pool: heavy-sixer
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub in progress since Sun Jun 11 00:08:11 2023
        15.5T scanned at 11.0M/s, 15.4T issued at 7.49M/s, 20.1T total
        0 repaired, 76.57% done, no estimated completion time
config:

        NAME                                            STATE     READ WRITE CKSUM
        heavy-sixer                                     DEGRADED     0     0     0
          raidz1-0                                      DEGRADED     0     0     0
            gptid/2ca40026-f288-11ec-8e85-1c1b0d0c7f05  ONLINE       0     0     0
            gptid/2c913616-f288-11ec-8e85-1c1b0d0c7f05  ONLINE       0     0     0
            gptid/2caffc6a-f288-11ec-8e85-1c1b0d0c7f05  ONLINE       0     0     0
            13011148730424112466                        OFFLINE      0     0     0  was /dev/gptid/2cbedbd4-f288-11ec-8e85-1c1b0d0c7f05

errors: No known data errors

ada0

Code:

root@nas2[~]# smartctl -a /dev/ada0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD60EFAX-68JH4N1
Serial Number:    WD-WXB2D71HAXUR
LU WWN Device Id: 5 0014ee 21477e979
Firmware Version: 83.00A83
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 26 07:09:06 2023 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (10784) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 765) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3039) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   225   224   021    Pre-fail  Always       -       3733
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       15
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       8832
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       14
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       12
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       4
194 Temperature_Celsius     0x0022   117   112   000    Old_age   Always       -       33
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

ada1

Code:

root@nas2[~]# smartctl -a /dev/ada1
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD60EFAX-68JH4N1
Serial Number:    WD-WXB2D7132CFZ
LU WWN Device Id: 5 0014ee 2bf230341
Firmware Version: 83.00A83
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 26 07:10:27 2023 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (57780) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 775) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3039) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   222   221   021    Pre-fail  Always       -       3883
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       15
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       8832
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       14
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       12
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       4
194 Temperature_Celsius     0x0022   116   111   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

ada2

Code:

root@nas2[~]# smartctl -a /dev/ada2
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD60EFAX-68JH4N1
Serial Number:    WD-WXB2D714H7P5
LU WWN Device Id: 5 0014ee 2bf231122
Firmware Version: 83.00A83
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 26 07:11:08 2023 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (65160) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 783) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3039) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   221   221   021    Pre-fail  Always       -       3916
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       16
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       8826
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       15
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       13
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       4
194 Temperature_Celsius     0x0022   116   112   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

ada3

Code:

root@nas2[~]# smartctl -a /dev/ada3
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     KINGSTON SUV400S37120G
Serial Number:    50026B777100B9B2
LU WWN Device Id: 0 550380 440010000
Firmware Version: 0C3J96R9
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      M.2
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-4, ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 26 07:12:25 2023 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 247) Self-test routine in progress...
                                        70% of test remaining.
Total time to complete Offline
data collection:                (    5) seconds.
Offline data collection
capabilities:                    (0x71) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (   5) minutes.
Conveyance self-test routine
recommended polling time:        (   0) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 48
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       39770
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       47
100 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       12237616
101 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       3238224
170 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
174 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       44
175 Program_Fail_Count_Chip 0x0032   100   100   000    Old_age   Always       -       0
176 Erase_Fail_Count_Chip   0x0032   100   100   000    Old_age   Always       -       0
177 Wear_Leveling_Count     0x0032   078   078   000    Old_age   Always       -       7215
178 Used_Rsvd_Blk_Cnt_Chip  0x0002   100   100   000    Old_age   Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0002   100   100   000    Old_age   Always       -       543
183 Runtime_Bad_Block       0x0032   076   076   000    Old_age   Always       -       151
187 Reported_Uncorrect      0x0033   100   100   000    Pre-fail  Always       -       0
194 Temperature_Celsius     0x0022   032   100   000    Old_age   Always       -       32 (Min/Max 20/46)
195 Hardware_ECC_Recovered  0x0032   100   100   000    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x0012   100   100   000    Old_age   Always       -       0
201 Unknown_SSD_Attribute   0x0032   100   100   000    Old_age   Always       -       0
204 Soft_ECC_Correction     0x0032   100   100   000    Old_age   Always       -       0
231 Temperature_Celsius     0x0032   076   076   000    Old_age   Always       -       24
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       22951
234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       18851
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       20295
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       30419
250 Read_Error_Retry_Rate   0x0032   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 0
No Errors Logged

SMART Self-test log structure revision number 0
Warning: ATA Specification requires self-test log structure revision number = 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Self-test routine in progress 70%     39770         -
# 2  Short offline       Completed without error       00%     39769         -
# 3  Short offline       Completed without error       00%     39768         -
# 4  Short offline       Completed without error       00%     39767         -
# 5  Short offline       Completed without error       00%     39766         -
# 6  Short offline       Completed without error       00%     39765         -
# 7  Short offline       Completed without error       00%     39764         -
# 8  Short offline       Completed without error       00%     39763         -
# 9  Short offline       Completed without error       00%     39762         -
#10  Short offline       Completed without error       00%     39761         -
#11  Short offline       Completed without error       00%     39750         -
#12  Short offline       Completed without error       00%     39749         -
#13  Short offline       Completed without error       00%     39748         -
#14  Short offline       Completed without error       00%     39747         -
#15  Short offline       Completed without error       00%     39746         -
#16  Short offline       Completed without error       00%     39745         -
#17  Short offline       Completed without error       00%     39744         -
#18  Short offline       Completed without error       00%     39743         -
#19  Short offline       Completed without error       00%     39742         -
#20  Short offline       Completed without error       00%     39741         -
#21  Short offline       Completed without error       00%     39740         -

SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[jgreco]

@jgreco thanks so much for your input and guidance. After not-so-patiently waiting all day for the NAS to resume service, it's back online (degraded of course) but at least to the point to where I can start scrambling to get the live data onto my 8TB USB standby.

That's awesome. I'm sorry it went this way in the first place, but "live and learn".

When I hot-unplugged one of the (remaining 1 of 3) drives not realizing I had pulled the wrong drive I about soiled myself. The DATTO doesn't go left to right for drives 0-3, it's 2,0,3,1 for whatever reason. I was this many years old before I learned what SMR/CMR drives even were.

It took us all off guard when SMR suddenly became a thing some years back.

Knowing how dynamic their operation is, what's the best use case for SMR drives anyway? Single drive bulk storage?

They will work fine in low write/large sequential write environments. Most desktop filesystems (NTFS, EXT3, FAT32, APFS) are oriented towards writing small (let's say sub-1GByte) blocks of data and in most cases not all that often. A SMR drive works a little bit like an SSD in that the units of storage on the drive are actually larger than a sector; the drive has to pull an entire "shingle", update a block, and then rewrite the whole thing to the disk. So this gets very stressy on the drive if you have a filesystem like ZFS that expects to be able to have random access.

 

[Etorix]

Good. The remaining three drives look healthy (but should run regular SMART tests). I suppose the failed drive is not plugged and that's why we have no information.
Once the external backup is done, bring in a new CMR drive to replace the failed SMR drive.
Preferably, replace all SMR drives by (burnt-in) CMR drives.
Much preferably, move to a safer raidz2 layout… but this requires building a new pool and migrating data from the old pool.

 

[jgreco]

Excellent strategy by @Etorix ... if you really value your data, RAIDZ2 with some 10TB drives is a safer bet. Your 4x6TB nets you about 18TB usable in RAIDZ1, but 4x10TB in RAIDZ2 nets you about the same (20TB usable).