Baenwort
Explorer
- Joined
- Feb 19, 2015
- Messages
- 93
So I got the following after a reboot:
I also got an alert stating:
However, when I look at the pool it still shows healthy in the "Storage" tab and when I check the Volume Status I find the attached. It shows no unrecoverable error.
I did check the SMART attributes and got the below:
Now, I think my next step should be replacing the disk that showed errors I think?
Code:
Dec 9 09:49:09 thing dhclient: New Routers (em0): 192.168.1.1 Dec 9 22:14:32 thing collectd[3046]: aggregation plugin: Unable to read the current rate of "thing.local/cpu-1/cpu-idle". Dec 9 22:14:32 thing collectd[3046]: utils_vl_lookup: The user object callback failed with status 2. Dec 9 22:32:52 thing collectd[3046]: aggregation plugin: Unable to read the current rate of "thing.local/cpu-2/cpu-interrupt". Dec 9 22:32:52 thing collectd[3046]: utils_vl_lookup: The user object callback failed with status 2. Dec 10 00:00:00 thing syslog-ng[1496]: Configuration reload request received, reloading configuration; Dec 10 00:15:18 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=7420478099537572347 Dec 10 00:15:18 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=9857921623318705200 Dec 10 00:15:18 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=18407950380888425658 Dec 10 00:15:18 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=8087918601935714222 Dec 10 00:41:47 thing (da0:mps0:0:0:0): READ(16). CDB: 88 00 00 00 00 01 38 28 4a 20 00 00 00 d0 00 00 length 106496 SMID 502 terminated ioc 804b scsi 0 state 0 xfer 0 Dec 10 00:41:47 thing (da0:mps0:0:0:0): READ(16). CDB: 88 00 00 00 00 01 38 28 4a 20 00 00 00 d0 00 00 Dec 10 00:41:47 thing (da0:mps0:0:0:0): CAM status: CCB request completed with an error Dec 10 00:41:47 thing (da0:mps0:0:0:0): Retrying command Dec 10 00:41:47 thing (da0:mps0:0:0:0): READ(16). CDB: 88 00 00 00 00 01 38 28 49 70 00 00 00 b0 00 00 Dec 10 00:41:47 thing (da0:mps0:0:0:0): CAM status: SCSI Status Error Dec 10 00:41:47 thing (da0:mps0:0:0:0): SCSI status: Check Condition Dec 10 00:41:47 thing (da0:mps0:0:0:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 10 00:41:47 thing (da0:mps0:0:0:0): Info: 0x138284970 Dec 10 00:41:47 thing (da0:mps0:0:0:0): Error 5, Unretryable error Dec 10 00:41:48 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=8087918601935714222 Dec 10 01:17:22 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 38 34 5b e0 00 00 a8 00 length 86016 SMID 346 terminated ioc 804b scsi 0 state 0 xfer 0 Dec 10 01:17:22 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 38 34 5b e0 00 00 a8 00 Dec 10 01:17:22 thing (da0:mps0:0:0:0): CAM status: CCB request completed with an error Dec 10 01:17:22 thing (da0:mps0:0:0:0): Retrying command Dec 10 01:17:22 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 38 34 5b 38 00 00 a8 00 Dec 10 01:17:22 thing (da0:mps0:0:0:0): CAM status: SCSI Status Error Dec 10 01:17:22 thing (da0:mps0:0:0:0): SCSI status: Check Condition Dec 10 01:17:22 thing (da0:mps0:0:0:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 10 01:17:22 thing (da0:mps0:0:0:0): Info: 0x38345b38 Dec 10 01:17:22 thing (da0:mps0:0:0:0): Error 5, Unretryable error Dec 10 01:17:22 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=8087918601935714222 Dec 10 02:50:08 thing (da1:mps0:0:2:0): READ(10). CDB: 28 00 92 99 dd c0 00 00 98 00 length 77824 SMID 362 terminated ioc 804b scsi 0 state 0 xfer 0 Dec 10 02:50:08 thing (da1:mps0:0:2:0): READ(10). CDB: 28 00 92 99 dd c0 00 00 98 00 Dec 10 02:50:08 thing (da1:mps0:0:2:0): CAM status: CCB request completed with an error Dec 10 02:50:08 thing (da1:mps0:0:2:0): Retrying command Dec 10 02:50:08 thing (da1:mps0:0:2:0): READ(10). CDB: 28 00 92 99 dd 28 00 00 98 00 Dec 10 02:50:08 thing (da1:mps0:0:2:0): CAM status: SCSI Status Error Dec 10 02:50:08 thing (da1:mps0:0:2:0): SCSI status: Check Condition Dec 10 02:50:08 thing (da1:mps0:0:2:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 10 02:50:08 thing (da1:mps0:0:2:0): Info: 0x9299dd28 Dec 10 02:50:08 thing (da1:mps0:0:2:0): Error 5, Unretryable error Dec 10 02:50:09 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=18407950380888425658 Dec 10 03:45:03 thing ZFS: vdev state changed, pool_guid=12363175241056571006 vdev_guid=9903920484091501851 Dec 10 03:45:03 thing ZFS: vdev state changed, pool_guid=12363175241056571006 vdev_guid=4621403864633465360 Dec 10 07:34:15 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 0f cb 71 60 00 00 c8 00 length 102400 SMID 511 terminated ioc 804b scsi 0 state 0 xfer 0 Dec 10 07:34:15 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 0f cb 71 60 00 00 c8 00 Dec 10 07:34:15 thing (da0:mps0:0:0:0): CAM status: CCB request completed with an error Dec 10 07:34:15 thing (da0:mps0:0:0:0): Retrying command Dec 10 07:34:15 thing (da0:mps0:0:0:0): READ(10). CDB: 28 00 0f cb 70 a8 00 00 b8 00 Dec 10 07:34:15 thing (da0:mps0:0:0:0): CAM status: SCSI Status Error Dec 10 07:34:15 thing (da0:mps0:0:0:0): SCSI status: Check Condition Dec 10 07:34:15 thing (da0:mps0:0:0:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Dec 10 07:34:15 thing (da0:mps0:0:0:0): Info: 0xfcb70a8 Dec 10 07:34:15 thing (da0:mps0:0:0:0): Error 5, Unretryable error Dec 10 07:34:15 thing ZFS: vdev state changed, pool_guid=15701039071623260518 vdev_guid=8087918601935714222
I also got an alert stating:
Code:
CRITICAL: Dec. 10, 2017, 12:42 a.m. - The volume cousin_itt state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
However, when I look at the pool it still shows healthy in the "Storage" tab and when I check the Volume Status I find the attached. It shows no unrecoverable error.
I did check the SMART attributes and got the below:
Code:
smartctl -a /dev/da0 smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD60EFRX-68L0BN1 Serial Number: WD-WX11D26H46XE LU WWN Device Id: 5 0014ee 20d7abe02 Firmware Version: 82.00A82 User Capacity: 6,001,175,126,016 bytes [6.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5700 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Dec 10 20:11:20 2017 CST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 4244) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 696) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 207 197 021 Pre-fail Always - 8641 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 16 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 086 086 000 Old_age Always - 10868 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 16 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 11 193 Load_Cycle_Count 0x0032 197 197 000 Old_age Always - 9658 194 Temperature_Celsius 0x0022 129 119 000 Old_age Always - 23 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 10865 - # 2 Short offline Completed without error 00% 10841 - # 3 Short offline Completed without error 00% 10817 - # 4 Extended offline Interrupted (host reset) 80% 10809 - # 5 Short offline Completed without error 00% 10793 - # 6 Short offline Completed without error 00% 10769 - # 7 Short offline Completed without error 00% 10745 - # 8 Short offline Completed without error 00% 10721 - # 9 Short offline Completed without error 00% 10697 - #10 Short offline Completed without error 00% 10673 - #11 Extended offline Completed without error 00% 10651 - #12 Short offline Completed without error 00% 10625 - #13 Extended offline Completed without error 00% 10603 - #14 Short offline Completed without error 00% 10577 - #15 Short offline Completed without error 00% 10553 - #16 Short offline Completed without error 00% 10529 - #17 Short offline Completed without error 00% 10505 - #18 Short offline Completed without error 00% 10481 - #19 Short offline Completed without error 00% 10457 - #20 Extended offline Completed without error 00% 10435 - #21 Short offline Completed without error 00% 10409 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Now, I think my next step should be replacing the disk that showed errors I think?
