Pool degraded, faulted drive

[ossy]

Hi,
my pool has degraded and one of my drives has faulted. If I understand correctly this does not necessarily mean the drive is dead.

I have started a scrub and I wanted to ask if these outputs could get me some more info:

ossy@jes01:~$ sudo zpool status -v iroh
pool: iroh
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0B in 23:01:44 with 0 errors on Mon Nov 27 01:01:50 2023
config:

NAME STATE READ WRITE CKSUM
iroh DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
02dc06e8-b70a-4f60-a421-a58a4eec1eef ONLINE 2 0 0
601b245e-f067-4322-ba37-33ccc08e4761 ONLINE 0 0 0
23b6ef6c-5011-4047-99d0-2b9b7554adfb ONLINE 0 0 0
2f1eff46-5428-44bb-92a6-905de1324a6a FAULTED 13 43 0 too many errors

errors: No known data errors

Faulted disk sda:

ossy@jes01:~$ sudo smartctl -a /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: ST4000VN006-3CW104
Serial Number: ZW601853
LU WWN Device Id: 5 000c50 0e4ec51b6
Firmware Version: SC60
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Dec 20 11:24:21 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 457) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 082 064 006 Pre-fail Always - 145616562
3 Spin_Up_Time 0x0003 096 096 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 20
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 088 060 045 Pre-fail Always - 577065762
9 Power_On_Hours 0x0032 088 088 000 Old_age Always - 10663
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 20
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 4295032833
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 068 054 040 Old_age Always - 32 (Min/Max 23/44)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 14
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 20
194 Temperature_Celsius 0x0022 032 046 000 Old_age Always - 32 (0 21 0 0 0)
195 Hardware_ECC_Recovered 0x001a 082 064 000 Old_age Always - 145616562
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 10662 (192 47 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 32760759549
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1008091134678

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 10662 -
# 2 Short offline Completed without error 00% 10654 -
# 3 Short offline Completed without error 00% 10630 -
# 4 Short offline Completed without error 00% 10606 -
# 5 Short offline Completed without error 00% 10582 -
# 6 Short offline Completed without error 00% 10558 -
# 7 Extended offline Completed without error 00% 10542 -
# 8 Short offline Completed without error 00% 10510 -
# 9 Short offline Completed without error 00% 10486 -
#10 Short offline Completed without error 00% 10462 -
#11 Short offline Completed without error 00% 10438 -
#12 Short offline Completed without error 00% 10414 -
#13 Short offline Completed without error 00% 10390 -
#14 Short offline Completed without error 00% 10366 -
#15 Short offline Completed without error 00% 10342 -
#16 Short offline Completed without error 00% 10318 -
#17 Short offline Completed without error 00% 10294 -
#18 Short offline Completed without error 00% 10270 -
#19 Short offline Completed without error 00% 10246 -
#20 Short offline Completed without error 00% 10222 -
#21 Short offline Completed without error 00% 10198 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I've noticed sdc has 2 errors aswell so just adding this to be sure:

ossy@jes01:~$ sudo smartctl -a /dev/sdc
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: ST4000VN006-3CW104
Serial Number: ZW60188X
LU WWN Device Id: 5 000c50 0e4ec6c18
Firmware Version: SC60
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Dec 20 11:23:24 2023 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 448) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x70bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 074 064 006 Pre-fail Always - 26148608
3 Spin_Up_Time 0x0003 096 096 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 20
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 088 060 045 Pre-fail Always - 570921966
9 Power_On_Hours 0x0032 088 088 000 Old_age Always - 10663
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 20
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 066 054 040 Old_age Always - 34 (Min/Max 23/45)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 14
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 20
194 Temperature_Celsius 0x0022 034 046 000 Old_age Always - 34 (0 21 0 0 0)
195 Hardware_ECC_Recovered 0x001a 074 064 000 Old_age Always - 26148608
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 1
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 10662 (121 107 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 33915199765
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 574757039176

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 10654 -
# 2 Short offline Completed without error 00% 10630 -
# 3 Short offline Completed without error 00% 10606 -
# 4 Short offline Completed without error 00% 10582 -
# 5 Short offline Completed without error 00% 10558 -
# 6 Extended offline Completed without error 00% 10542 -
# 7 Short offline Completed without error 00% 10510 -
# 8 Short offline Completed without error 00% 10486 -
# 9 Short offline Completed without error 00% 10462 -
#10 Short offline Completed without error 00% 10438 -
#11 Short offline Completed without error 00% 10414 -
#12 Short offline Completed without error 00% 10390 -
#13 Short offline Completed without error 00% 10366 -
#14 Short offline Completed without error 00% 10342 -
#15 Short offline Completed without error 00% 10318 -
#16 Short offline Completed without error 00% 10294 -
#17 Short offline Completed without error 00% 10270 -
#18 Short offline Completed without error 00% 10246 -
#19 Short offline Completed without error 00% 10222 -
#20 Short offline Completed without error 00% 10198 -
#21 Short offline Completed without error 00% 10174 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Don't really know much so any help would be much appreciated

 

[ossy]

Adding dmesg -H -T output from the day of the fault, don't really know what any of it means but maybe it could be useful:

Code:

[Sun Dec 17 11:44:51 2023] Process 224037(.NET Finalizer) has RLIMIT_CORE set to 1
[Sun Dec 17 11:44:51 2023] Aborting core
[Sun Dec 17 18:43:05 2023] ata4.00: exception Emask 0x50 SAct 0x110000 SErr 0x280900 action 0x6 frozen
[Sun Dec 17 18:43:05 2023] ata4.00: irq_stat 0x08000000, interface fatal error
[Sun Dec 17 18:43:05 2023] ata4: SError: { UnrecovData HostInt 10B8B BadCRC }
[Sun Dec 17 18:43:05 2023] ata4.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:43:05 2023] ata4.00: cmd 60/40:80:d0:6e:46/00:00:3d:01:00/40 tag 16 ncq dma 32768 in
                                    res 40/00:a0:58:60:88/00:00:02:01:00/40 Emask 0x50 (ATA bus error)
[Sun Dec 17 18:43:05 2023] ata4.00: status: { DRDY }
[Sun Dec 17 18:43:05 2023] ata4.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:43:05 2023] ata4.00: cmd 60/58:a0:58:60:88/00:00:02:01:00/40 tag 20 ncq dma 45056 in
                                    res 40/00:a0:58:60:88/00:00:02:01:00/40 Emask 0x50 (ATA bus error)
[Sun Dec 17 18:43:05 2023] ata4.00: status: { DRDY }
[Sun Dec 17 18:43:05 2023] ata4: hard resetting link
[Sun Dec 17 18:43:05 2023] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Sun Dec 17 18:43:05 2023] ata4.00: configured for UDMA/133
[Sun Dec 17 18:43:05 2023] sd 3:0:0:0: [sdc] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[Sun Dec 17 18:43:05 2023] sd 3:0:0:0: [sdc] tag#16 Sense Key : Illegal Request [current]
[Sun Dec 17 18:43:05 2023] sd 3:0:0:0: [sdc] tag#16 Add. Sense: Unaligned write command
[Sun Dec 17 18:43:05 2023] sd 3:0:0:0: [sdc] tag#16 CDB: Read(16) 88 00 00 00 00 01 3d 46 6e d0 00 00 00 40 00 00
[Sun Dec 17 18:43:05 2023] blk_update_request: I/O error, dev sdc, sector 5322993360 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[Sun Dec 17 18:43:05 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/02dc06e8-b70a-4f60-a421-a58a4eec1eef error=5 type=1 offset=2723225051136 size=32768 flags=180880
[Sun Dec 17 18:43:05 2023] sd 3:0:0:0: [sdc] tag#20 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[Sun Dec 17 18:43:05 2023] sd 3:0:0:0: [sdc] tag#20 Sense Key : Illegal Request [current]
[Sun Dec 17 18:43:05 2023] sd 3:0:0:0: [sdc] tag#20 Add. Sense: Unaligned write command
[Sun Dec 17 18:43:05 2023] sd 3:0:0:0: [sdc] tag#20 CDB: Read(16) 88 00 00 00 00 01 02 88 60 58 00 00 00 58 00 00
[Sun Dec 17 18:43:05 2023] blk_update_request: I/O error, dev sdc, sector 4337459288 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[Sun Dec 17 18:43:05 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/02dc06e8-b70a-4f60-a421-a58a4eec1eef error=5 type=1 offset=2218631606272 size=45056 flags=180980
[Sun Dec 17 18:43:05 2023] ata4: EH complete
[Sun Dec 17 18:44:19 2023] ata2.00: exception Emask 0x40 SAct 0x80fff81f SErr 0xa80801 action 0x6 frozen
[Sun Dec 17 18:44:19 2023] ata2: SError: { RecovData HostInt 10B8B BadCRC LinkSeq }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/50:00:c0:d1:57/00:00:05:01:00/40 tag 0 ncq dma 40960 in
                                    res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/50:08:18:d2:57/00:00:05:01:00/40 tag 1 ncq dma 40960 in
                                    res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: WRITE FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 61/18:10:38:a5:1a/00:00:a2:00:00/40 tag 2 ncq dma 12288 out
                                    res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/48:18:78:27:10/00:00:37:01:00/40 tag 3 ncq dma 36864 in
                                    res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/10:20:10:36:b4/00:00:6e:00:00/40 tag 4 ncq dma 8192 in
                                    res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: WRITE FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 61/d0:58:e8:a9:53/04:00:3d:01:00/40 tag 11 ncq dma 630784 out
                                    res 40/00:ff:ff:00:00/00:00:00:00:00/40 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: WRITE FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 61/88:60:b8:ae:53/01:00:3d:01:00/40 tag 12 ncq dma 200704 out
                                    res 40/00:ff:ff:00:00/00:00:00:00:00/40 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/10:68:58:66:d7/00:00:36:01:00/40 tag 13 ncq dma 8192 in
                                    res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/50:70:60:99:be/00:00:d1:00:00/40 tag 14 ncq dma 40960 in
                                    res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/50:78:b8:99:be/00:00:d1:00:00/40 tag 15 ncq dma 40960 in
                                    res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/50:80:10:9a:be/00:00:d1:00:00/40 tag 16 ncq dma 40960 in
                                    res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/50:88:68:9a:be/00:00:d1:00:00/40 tag 17 ncq dma 40960 in
                                    res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/50:90:c0:9a:be/00:00:d1:00:00/40 tag 18 ncq dma 40960 in
                                    res 40/00:01:06:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/10:98:48:eb:17/00:00:37:01:00/40 tag 19 ncq dma 8192 in
                                    res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: WRITE FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 61/50:a0:20:b3:1a/00:00:a2:00:00/40 tag 20 ncq dma 40960 out
                                    res 40/00:ff:ff:00:00/00:00:00:00:00/40 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/08:a8:80:8f:df/00:00:36:01:00/40 tag 21 ncq dma 4096 in
                                    res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:19 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:19 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:19 2023] ata2.00: cmd 60/08:b0:50:80:0e/00:00:00:00:00/40 tag 22 ncq dma 4096 in
                                    res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:20 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:20 2023] ata2.00: failed command: WRITE FPDMA QUEUED
[Sun Dec 17 18:44:20 2023] ata2.00: cmd 61/18:b8:40:a7:1a/00:00:a2:00:00/40 tag 23 ncq dma 12288 out
                                    res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:20 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:20 2023] ata2.00: failed command: READ FPDMA QUEUED
[Sun Dec 17 18:44:20 2023] ata2.00: cmd 60/50:f8:68:d1:57/00:00:05:01:00/40 tag 31 ncq dma 40960 in
                                    res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x44 (timeout)
[Sun Dec 17 18:44:20 2023] ata2.00: status: { DRDY }
[Sun Dec 17 18:44:20 2023] ata2: hard resetting link
[Sun Dec 17 18:44:20 2023] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Sun Dec 17 18:44:20 2023] ata2.00: configured for UDMA/133
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=50s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#0 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#0 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#0 CDB: Read(16) 88 00 00 00 00 01 05 57 d1 c0 00 00 00 50 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 4384608704 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=2242772107264 size=40960 flags=180980
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=50s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#1 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#1 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#1 CDB: Read(16) 88 00 00 00 00 01 05 57 d2 18 00 00 00 50 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 4384608792 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=2242772152320 size=40960 flags=180980
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=49s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#2 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#2 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#2 CDB: Write(16) 8a 00 00 00 00 00 a2 1a a5 38 00 00 00 18 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 2719655224 op 0x1:(WRITE) flags 0x700 phys_seg 3 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=2 offset=1390315925504 size=12288 flags=180880
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=48s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#3 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#3 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#3 CDB: Read(16) 88 00 00 00 00 01 37 10 27 78 00 00 00 48 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 5218772856 op 0x0:(READ) flags 0x700 phys_seg 3 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=2669864153088 size=36864 flags=180880
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=47s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#4 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#4 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#4 CDB: Read(16) 88 00 00 00 00 00 6e b4 36 10 00 00 00 10 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 1857304080 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=948792139776 size=8192 flags=180880
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=50s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#11 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#11 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#11 CDB: Write(16) 8a 00 00 00 00 01 3d 53 a9 e8 00 00 04 d0 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 5323860456 op 0x1:(WRITE) flags 0x700 phys_seg 115 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=2 offset=2723669004288 size=630784 flags=40080c80
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=51s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#12 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#12 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#12 CDB: Write(16) 8a 00 00 00 00 01 3d 53 ae b8 00 00 01 88 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 5323861688 op 0x1:(WRITE) flags 0x700 phys_seg 15 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=2 offset=2723669635072 size=200704 flags=40080c80
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=50s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#13 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#13 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#13 CDB: Read(16) 88 00 00 00 00 01 36 d7 66 58 00 00 00 10 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 5215053400 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=2667959791616 size=8192 flags=180880
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=50s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#14 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#14 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#14 CDB: Read(16) 88 00 00 00 00 00 d1 be 99 60 00 00 00 50 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 3518929248 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=1799544225792 size=40960 flags=180880
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=50s
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#15 Sense Key : Illegal Request [current]
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#15 Add. Sense: Unaligned write command
[Sun Dec 17 18:44:20 2023] sd 1:0:0:0: [sda] tag#15 CDB: Read(16) 88 00 00 00 00 00 d1 be 99 b8 00 00 00 50 00 00
[Sun Dec 17 18:44:20 2023] blk_update_request: I/O error, dev sda, sector 3518929336 op 0x0:(READ) flags 0x700 phys_seg 2 prio class 0
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=1799544270848 size=40960 flags=180880
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=1799544315904 size=40960 flags=180880
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=1799544360960 size=40960 flags=180880
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=1799544406016 size=40960 flags=180880
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=2670124699648 size=8192 flags=180880
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=2 offset=1390317748224 size=40960 flags=180880
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=2668233621504 size=4096 flags=180880
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=2 offset=1390316191744 size=12288 flags=180880
[Sun Dec 17 18:44:20 2023] zio pool=iroh vdev=/dev/disk/by-partuuid/2f1eff46-5428-44bb-92a6-905de1324a6a error=5 type=1 offset=2242772062208 size=40960 flags=180980
[Sun Dec 17 18:44:20 2023] ata2: EH complete