One of our pools entered into a degraded state as shown below:
Output of "zpool status -v"
Output of /var/log/messages
I tried a reboot first, but that didn't seem to online the drive, so I did a physical re-seat of the drive. After doing that, the pool came back online, but the drive mounted as a different device (previously da1, and now da13) and now shows one checksum error after resilver.
Output of "zpool status -v" after pool came online
I ran an extended SMART test thinking the drive could be going bad indicated by the above log errors
Output of "smartctl -a /dev/da13"
However, I don't really see anything that indicates drive failure. I'm not really clear on what the checksum error truly means, and if I should worry about it. Over the past year or so, I have had 3 different drives exhibit the same behavior (one drive off-lined twice), albeit without the checksum error. When the drives get physically re-seated, the pool comes back online and everything returns to normal. My first thought would be a potential problem/bug on the SAS expander/controller or the cabling itself. However, I was wondering if there could be other issues that I'm not aware of here.
Hardware Specs:
Supermicro X10DRH-i
64GB DDR4 ECC RDIMM
Intel Xeon(R) CPU E5-2620 v3 @ 2.40GHz
LSI SAS3008 HBA
14 x HGST HUS728T8TALE6L4 8TB 7200rpm drives
1 x Samsung 850 pro boot drive
Output of "zpool status -v"
Code:
pool: User-Storage
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 0 days 14:08:04 with 0 errors on Sun Apr 12 14:08:05 2020
config:
NAME STATE READ WRITE CKSUM
User-Storage DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
gptid/1c3538f7-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
13219515766353509539 UNAVAIL 0 0 0 was /dev/gptid/21d3c2e9-7d60-11e9-b756-0cc47aa9f8f8
gptid/26ca3238-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/2bd1e89f-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/316952fc-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/3673f550-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/3b7bcbd0-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
Output of /var/log/messages
Code:
syslog-ng[1192]: Configuration reload finished;
(da1:mpr0:0:9:0): READ(16). CDB: 88 00 00 00 00 02 0a 4c 46 30 00 00 00 08 00 00 length 4096 SMID 573 terminated ioc 804b loginfo 31120101 scsi 0 state c xfer 4096
(da1:mpr0:0:9:0): READ(16). CDB: 88 00 00 00 00 02 0a 4c 46 30 00 00 00 08 00 00
(da1:mpr0:0:9:0): CAM status: CCB request completed with an error
(da1:mpr0:0:9:0): Retrying command
(da1:mpr0:0:9:0): READ(16). CDB: 88 00 00 00 00 02 0a 4c 46 30 00 00 00 08 00 00 length 4096 SMID 330 terminated ioc 804b loginfo 31170000 scsi 0 state c xfer 0
(da1:mpr0:0:9:0): READ(16). CDB: 88 00 00 00 00 02 0a 4c 46 30 00 00 00 08 00 00
(da1:mpr0:0:9:0): CAM status: CCB request completed with an error
(da1:mpr0:0:9:0): Retrying command
(da1:mpr0:0:9:0): READ(16). CDB: 88 00 00 00 00 02 0a 4c 46 30 00 00 00 08 00 00 length 4096 SMID 318 terminated ioc 804b loginfo 31170000 scsi 0 state 0 xfer 0
(da1:mpr0:0:9:0): READ(16). CDB: 88 00 00 00 00 02 0a 4c 46 30 00 00 00 08 00 00
(da1:mpr0:0:9:0): CAM status: CCB request completed with an error
(da1:mpr0:0:9:0): Retrying command
mpr0: mprsas_prepare_remove: Sending reset for target ID 9
da1 at mpr0 bus 0 scbus0 target 9 lun 0
da1: (da1:mpr0:0:9:0): READ(16). CDB: 88 00 00 00 00 02 0a 4c 46 30 00 00 00 08 00 00 length 4096 SMID 851 terminated ioc 804b log<ATA HGST HUS728T8TAL W410>info 31130000 scsi 0 state c xfer 0
mpr0: s/n VAJ26RRL detached
(da1:mpr0:0:9:0): READ(16). CDB: 88 00 00 00 00 02 0a 4c 46 30 00 00 00 08 00 00
clearing target 9 handle 0x000b
mpr0: At enclosure level 0, slot 1, connector name ( )
mpr0: Unfreezing devq for target ID 9
(da1:mpr0:0:9:0): CAM status: CCB request completed with an error
(da1:mpr0:0:9:0): Error 5, Periph was invalidated
ZFS: vdev state changed, pool_guid=3479512686869659690 vdev_guid=13219515766353509539
ZFS: vdev is removed, pool_guid=3479512686869659690 vdev_guid=13219515766353509539
(da1:mpr0:0:9:0): Periph destroyed
ZFS: vdev state changed, pool_guid=3479512686869659690 vdev_guid=13219515766353509539
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mpr0: SAS Address from SAS device page0 = 500304801eb75701
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mprsas_get_sata_identify: error reading SATA PASSTHRU; iocstatus = 0x804b
mpr0: Sleeping 3 seconds after SATA ID error to wait for spinup
mpr0: Found device <81<SataDev>,End Device> <12.0Gbps> handle<0x000b> enclosureHandle<0x0002> slot 1
mpr0: At enclosure level 0 and connector name ( )
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 762 terminated ioc 804b loginfo 31111000 scsi 0 state c xfer 0
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:mpr0:0:9:0): CAM status: CCB request completed with an error
(probe0:mpr0:0:9:0): Retrying command
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 1000 terminated ioc 804b loginfo 31111000 scsi 0 state c xfer 0
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:mpr0:0:9:0): CAM status: CCB request completed with an error
(probe0:mpr0:0:9:0): Retrying command
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 170 terminated ioc 804b loginfo 31111000 scsi 0 state c xfer 0
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:mpr0:0:9:0): CAM status: CCB request completed with an error
(probe0:mpr0:0:9:0): Retrying command
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 238 terminated ioc 804b loginfo 31111000 scsi 0 state c xfer 0
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:mpr0:0:9:0): CAM status: CCB request completed with an error
(probe0:mpr0:0:9:0): Retrying command
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 820 terminated ioc 804b loginfo 31111000 scsi 0 state c xfer 0
(probe0:mpr0:0:9:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:mpr0:0:9:0): CAM status: CCB request completed with an error
(probe0:mpr0:0:9:0): Error 5, Retries exhausted
I tried a reboot first, but that didn't seem to online the drive, so I did a physical re-seat of the drive. After doing that, the pool came back online, but the drive mounted as a different device (previously da1, and now da13) and now shows one checksum error after resilver.
Output of "zpool status -v" after pool came online
Code:
pool: User-Storage
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: resilvered 44.4G in 0 days 00:20:43 with 0 errors on Mon Apr 27 11:59:36 2020
config:
NAME STATE READ WRITE CKSUM
User-Storage ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/1c3538f7-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/21d3c2e9-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 1
gptid/26ca3238-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/2bd1e89f-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/316952fc-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/3673f550-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
gptid/3b7bcbd0-7d60-11e9-b756-0cc47aa9f8f8 ONLINE 0 0 0
errors: No known data errors
I ran an extended SMART test thinking the drive could be going bad indicated by the above log errors
Output of "smartctl -a /dev/da13"
Code:
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: HGST HUS728T8TALE6L4
Serial Number: VAJ26RRL
LU WWN Device Id: 5 000cca 099dd3106
Firmware Version: V8GNW410
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Apr 28 13:49:53 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 87) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 882) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 132 132 054 Pre-fail Offline - 96
3 Spin_Up_Time 0x0007 100 100 024 Pre-fail Always - 0
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 3
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 099 099 000 Old_age Always - 9362
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 3
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 388
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 388
194 Temperature_Celsius 0x0002 150 150 000 Old_age Always - 40 (Min/Max 24/47)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 9357 -
# 2 Short offline Completed without error 00% 9252 -
# 3 Short offline Completed without error 00% 9228 -
# 4 Short offline Completed without error 00% 9204 -
# 5 Short offline Completed without error 00% 9180 -
# 6 Short offline Completed without error 00% 9156 -
# 7 Short offline Completed without error 00% 9132 -
# 8 Short offline Completed without error 00% 9108 -
# 9 Short offline Completed without error 00% 9084 -
#10 Short offline Completed without error 00% 9060 -
#11 Short offline Completed without error 00% 9036 -
#12 Extended offline Completed without error 00% 9030 -
#13 Short offline Completed without error 00% 9012 -
#14 Short offline Completed without error 00% 8988 -
#15 Short offline Completed without error 00% 8964 -
#16 Short offline Completed without error 00% 8940 -
#17 Short offline Completed without error 00% 8916 -
#18 Short offline Completed without error 00% 8892 -
#19 Short offline Completed without error 00% 8868 -
#20 Short offline Completed without error 00% 8844 -
#21 Short offline Completed without error 00% 8820 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
However, I don't really see anything that indicates drive failure. I'm not really clear on what the checksum error truly means, and if I should worry about it. Over the past year or so, I have had 3 different drives exhibit the same behavior (one drive off-lined twice), albeit without the checksum error. When the drives get physically re-seated, the pool comes back online and everything returns to normal. My first thought would be a potential problem/bug on the SAS expander/controller or the cabling itself. However, I was wondering if there could be other issues that I'm not aware of here.
Hardware Specs:
Supermicro X10DRH-i
64GB DDR4 ECC RDIMM
Intel Xeon(R) CPU E5-2620 v3 @ 2.40GHz
LSI SAS3008 HBA
14 x HGST HUS728T8TALE6L4 8TB 7200rpm drives
1 x Samsung 850 pro boot drive
