Howard Swope
Dabbler
- Joined
- Nov 19, 2015
- Messages
- 26
I just upgraded from freenas from an earlier version 8 to 9.3.1. I am now seeing errors that I am having difficulty interpreting. I am seeing error reports from smartctrl and zpool. And it seems like my array switches from healthy to degraded every other day.
First the zpool status:
This looks to me like there is a problem with drive 0. And it looks like there is some type of corruption in two system files. But I am not sure what to do with this information.
smartctl output:
Just basic output shows passed across the board:
But looking at all information reveals errors on disk 0 and 3:
Disk 0:
Disk 3
Despite the fact that overall the drives show as PASSED, drive 0 and 3 are showing errors.
Any advice on how to proceed is appreciated.
First the zpool status:
Code:
[XXX] ~# zpool status -x -v pool: volume0 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 20.1M in 9h7m with 155 errors on Thu Nov 19 18:07:18 2015 config: NAME STATE READ WRITE CKSUM volume0 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/a653dc0f-8853-11e2-8909-002590a8e414 ONLINE 0 0 1 gptid/a6be6a5f-8853-11e2-8909-002590a8e414 ONLINE 0 0 0 gptid/a729e2c2-8853-11e2-8909-002590a8e414 ONLINE 0 0 0 gptid/a78f764d-8853-11e2-8909-002590a8e414 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /var/db/system/cores/syslog-ng.core /var/db/system/syslog-567b59bd58814d6ebad2ebdfdfba568a/log/telemetry.json.bz2
This looks to me like there is a problem with drive 0. And it looks like there is some type of corruption in two system files. But I am not sure what to do with this information.
smartctl output:
Just basic output shows passed across the board:
Code:
[XXX] ~# smartctl -H /dev/ada0 smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED [XXX] ~# smartctl -H /dev/ada1 smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED [XXX] ~# smartctl -H /dev/ada2 smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED [XXX] ~# smartctl -H /dev/ada3 smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED
But looking at all information reveals errors on disk 0 and 3:
Disk 0:
Code:
[XXX] ~# smartctl -a /dev/ada0 smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-9YN166 Serial Number: W1F18JE7 LU WWN Device Id: 5 000c50 053880e63 Firmware Version: CC4B User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Fri Nov 20 04:54:56 2015 EST ==> WARNING: A firmware update for this drive may be available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 241) Self-test routine in progress... 10% of test remaining. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 342) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 097 097 006 Pre-fail Always - 87436760 3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 45 5 Reallocated_Sector_Ct 0x0033 051 051 036 Pre-fail Always - 64736 7 Seek_Error_Rate 0x000f 064 060 030 Pre-fail Always - 2965858 9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 23562 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 45 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 094 083 000 Old_age Always - 188 188 188 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 067 058 045 Old_age Always - 33 (Min/Max 28/34) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 39 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 98 194 Temperature_Celsius 0x0022 033 042 000 Old_age Always - 33 (0 15 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 23562h+42m+12.795s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 1987424826513 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 14117367880170 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Self-test routine in progress 10% 23562 - # 2 Short offline Completed without error 00% 23558 - # 3 Short offline Interrupted (host reset) 10% 23557 - # 4 Short offline Completed: read failure 10% 23533 505182249 # 5 Short offline Completed without error 00% 23509 - # 6 Short offline Completed without error 00% 23485 - # 7 Short offline Interrupted (host reset) 10% 23462 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Disk 3
Code:
[XXX] ~# smartctl -a /dev/ada3 smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-9YN166 Serial Number: W1F1ADE8 LU WWN Device Id: 5 000c50 053872c9f Firmware Version: CC4B User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Fri Nov 20 05:01:54 2015 EST ==> WARNING: A firmware update for this drive may be available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: ( 575) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 341) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 087 077 006 Pre-fail Always - 235749051 3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 48 5 Reallocated_Sector_Ct 0x0033 053 052 036 Pre-fail Always - 62032 7 Seek_Error_Rate 0x000f 064 060 030 Pre-fail Always - 3000421 9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 23565 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 48 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 3054 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1 1 1 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 072 062 045 Old_age Always - 28 (Min/Max 24/30) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 43 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 99 194 Temperature_Celsius 0x0022 028 040 000 Old_age Always - 28 (0 14 0 0 0) 197 Current_Pending_Sector 0x0012 084 084 000 Old_age Always - 2704 198 Offline_Uncorrectable 0x0010 084 084 000 Old_age Offline - 2704 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 23565h+20m+39.111s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2071987855371 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 8952376693249 SMART Error Log Version: 1 ATA Error Count: 3040 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 3040 occurred at disk power-on lifetime: 23565 hours (981 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 50 ff ff ff 4f 00 39d+23:22:51.211 READ FPDMA QUEUED b0 d5 01 00 4f c2 40 00 39d+23:22:51.210 SMART READ LOG 2f 00 01 10 00 00 00 00 39d+23:22:51.095 READ LOG EXT 60 00 50 ff ff ff 4f 00 39d+23:22:47.959 READ FPDMA QUEUED b0 da 00 00 4f c2 40 00 39d+23:22:47.724 SMART RETURN STATUS Error 3039 occurred at disk power-on lifetime: 23565 hours (981 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 50 ff ff ff 4f 00 39d+23:22:47.959 READ FPDMA QUEUED b0 da 00 00 4f c2 40 00 39d+23:22:47.724 SMART RETURN STATUS 2f 00 01 10 00 00 00 00 39d+23:22:47.602 READ LOG EXT 60 00 50 ff ff ff 4f 00 39d+23:22:45.059 READ FPDMA QUEUED 35 00 08 ff ff ff 4f 00 39d+23:22:45.059 WRITE DMA EXT Error 3038 occurred at disk power-on lifetime: 23565 hours (981 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 50 ff ff ff 4f 00 39d+23:22:45.059 READ FPDMA QUEUED 35 00 08 ff ff ff 4f 00 39d+23:22:45.059 WRITE DMA EXT b0 d1 01 01 4f c2 40 00 39d+23:22:45.051 SMART READ ATTRIBUTE THRESHOLDS [OBS-4] 25 00 00 ff ff ff 4f 00 39d+23:22:40.668 READ DMA EXT b0 d0 01 00 4f c2 40 00 39d+23:22:40.393 SMART READ DATA Error 3037 occurred at disk power-on lifetime: 23565 hours (981 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 00 ff ff ff 4f 00 39d+23:22:40.668 READ DMA EXT b0 d0 01 00 4f c2 40 00 39d+23:22:40.393 SMART READ DATA 25 00 00 ff ff ff 4f 00 39d+23:22:37.786 READ DMA EXT ec 00 01 00 00 00 40 00 39d+23:22:37.778 IDENTIFY DEVICE 25 00 00 ff ff ff 4f 00 39d+23:22:33.977 READ DMA EXT Error 3036 occurred at disk power-on lifetime: 23565 hours (981 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 00 ff ff ff 4f 00 39d+23:22:37.786 READ DMA EXT ec 00 01 00 00 00 40 00 39d+23:22:37.778 IDENTIFY DEVICE 25 00 00 ff ff ff 4f 00 39d+23:22:33.977 READ DMA EXT 25 00 00 ff ff ff 4f 00 39d+23:22:27.818 READ DMA EXT 25 00 00 ff ff ff 4f 00 39d+23:22:24.033 READ DMA EXT SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed: read failure 90% 23561 340012704 # 2 Short offline Completed: read failure 90% 23559 340468640 # 3 Short offline Completed: read failure 90% 23536 339926120 # 4 Short offline Completed: read failure 10% 23512 506323470 # 5 Short offline Completed: read failure 90% 23488 340468576 # 6 Short offline Completed: read failure 10% 23465 340467664 SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Despite the fact that overall the drives show as PASSED, drive 0 and 3 are showing errors.
Any advice on how to proceed is appreciated.