atomicrabbit
Cadet
- Joined
- Nov 19, 2018
- Messages
- 6
Setup:
I ran a short test on both drives and they returned 0 errors (smartctl -t short /dev/da1). I ran a long test on da1 (smartctl -t long /dev/da1) yesterday and it just finished with 0 errors. I'm now running a long test on da3 and it will be done by tomorrow.
Here's the output of smartctl -a /dev/da1:
Here's the output of smartctl -a /dev/da3:
I see a lot of "Error ### occurred at disk power-on lifetime..." errors on the output, but I also see those on /dev/da2 and /dev/da4, but no critical errors for those devices show up in the FreeNAS alerts. Any troubleshooting suggestions you guys can offer?
- FreeNAS 11.1 VM running in ESXi 6.5 Update 1
- LSI 9211-8i HBA in IT mode, hardware pass-through to VM
- 4x 4TB WD Reds connected to the HBA via SFF-8087 to 4 X SATA cable
Code:
CRITICAL: Nov. 20, 2018, 10:08 p.m. - Device: /dev/da3 [SAT], ATA error count increased from 590 to 594 CRITICAL: Nov. 20, 2018, 10:08 p.m. - Device: /dev/da1 [SAT], ATA error count increased from 615 to 617
I ran a short test on both drives and they returned 0 errors (smartctl -t short /dev/da1). I ran a long test on da1 (smartctl -t long /dev/da1) yesterday and it just finished with 0 errors. I'm now running a long test on da3 and it will be done by tomorrow.
Here's the output of smartctl -a /dev/da1:
Code:
root@freenas:~ # smartctl -a /dev/da1 smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD80EFZX-68UW8N0 Serial Number: xxxxxxxx LU WWN Device Id: 5 000cca 261d2e08d Firmware Version: 83.H0A83 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Nov 21 18:52:39 2018 PST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 101) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: (1167) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0 2 Throughput_Performance 0x0005 131 131 054 Pre-fail Offline - 116 3 Spin_Up_Time 0x0007 146 146 024 Pre-fail Always - 448 (Average 450) 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 14 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 140 140 020 Pre-fail Offline - 15 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 2706 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 12 22 Helium_Level 0x0023 100 100 025 Pre-fail Always - 100 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 131 193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 131 194 Temperature_Celsius 0x0002 154 154 000 Old_age Always - 42 (Min/Max 23/49) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 619 SMART Error Log Version: 1 ATA Error Count: 619 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 619 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 28 81 40 00 1d+04:14:54.669 READ FPDMA QUEUED 2f 00 01 10 00 00 00 00 1d+04:14:54.630 READ LOG EXT 2f 00 01 10 00 00 00 00 1d+04:14:54.629 READ LOG EXT 60 00 00 00 28 81 40 00 1d+04:14:54.629 READ FPDMA QUEUED 60 00 00 00 27 81 40 00 1d+04:14:54.628 READ FPDMA QUEUED Error 618 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 28 81 40 00 1d+04:14:54.629 READ FPDMA QUEUED 60 00 00 00 27 81 40 00 1d+04:14:54.628 READ FPDMA QUEUED 60 00 00 00 26 81 40 00 1d+04:14:54.628 READ FPDMA QUEUED 60 00 00 00 03 00 40 00 1d+04:14:54.627 READ FPDMA QUEUED 60 00 00 00 02 00 40 00 1d+04:14:54.627 READ FPDMA QUEUED Error 617 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 01 00 40 00 1d+04:00:28.926 READ FPDMA QUEUED 60 00 00 00 00 00 40 00 1d+04:00:28.922 READ FPDMA QUEUED ec 00 00 00 00 00 00 00 1d+04:00:28.922 IDENTIFY DEVICE ec 00 00 00 00 00 00 00 1d+04:00:28.888 IDENTIFY DEVICE ec 00 00 00 00 00 00 00 1d+04:00:28.843 IDENTIFY DEVICE Error 616 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 01 00 40 00 1d+04:00:28.126 READ FPDMA QUEUED 60 00 00 00 00 00 40 00 1d+04:00:28.112 READ FPDMA QUEUED ec 00 00 00 00 00 00 00 1d+04:00:28.110 IDENTIFY DEVICE ec 00 00 00 00 00 00 00 1d+04:00:28.020 IDENTIFY DEVICE b0 d5 01 01 4f c2 00 00 1d+03:40:26.509 SMART READ LOG Error 615 occurred at disk power-on lifetime: 2684 hours (111 days + 20 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 e0 10 20 26 81 40 00 1d+02:39:48.395 READ FPDMA QUEUED 60 e0 18 20 28 81 40 00 1d+02:39:48.354 READ FPDMA QUEUED 60 e0 08 20 02 00 40 00 1d+02:39:48.354 READ FPDMA QUEUED 60 e0 00 20 00 00 40 00 1d+02:39:48.354 READ FPDMA QUEUED 2f 00 01 10 00 00 00 00 1d+02:39:48.337 READ LOG EXT SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 2704 - # 2 Short offline Completed without error 00% 2686 - # 3 Short offline Completed without error 00% 1688 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Here's the output of smartctl -a /dev/da3:
Code:
root@freenas:~ # smartctl -a /dev/da3 smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD80EFZX-68UW8N0 Serial Number: xxxxxxxx LU WWN Device Id: 5 000cca 263c72b30 Firmware Version: 83.H0A83 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Nov 21 18:59:49 2018 PST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 249) Self-test routine in progress... 90% of test remaining. Total time to complete Offline data collection: ( 101) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: (1242) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0 2 Throughput_Performance 0x0005 131 131 054 Pre-fail Offline - 116 3 Spin_Up_Time 0x0007 146 146 024 Pre-fail Always - 453 (Average 447) 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 15 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 140 140 020 Pre-fail Offline - 15 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 2706 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 13 22 Helium_Level 0x0023 100 100 025 Pre-fail Always - 100 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 133 193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 133 194 Temperature_Celsius 0x0002 144 144 000 Old_age Always - 45 (Min/Max 23/52) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 596 SMART Error Log Version: 1 ATA Error Count: 596 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 596 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 02 00 40 00 1d+04:23:03.114 READ FPDMA QUEUED 60 00 00 00 01 00 40 00 1d+04:23:03.113 READ FPDMA QUEUED 60 00 00 00 00 00 40 00 1d+04:23:03.113 READ FPDMA QUEUED ec 00 00 00 00 00 00 00 1d+04:23:03.113 IDENTIFY DEVICE ec 00 00 00 00 00 00 00 1d+04:23:03.083 IDENTIFY DEVICE Error 595 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 00 00 40 00 1d+04:14:54.571 READ FPDMA QUEUED ec 00 00 00 00 00 00 00 1d+04:14:54.568 IDENTIFY DEVICE ec 00 00 00 00 00 00 00 1d+04:14:54.539 IDENTIFY DEVICE ec 00 00 00 00 00 00 00 1d+04:14:54.474 IDENTIFY DEVICE 60 00 00 00 29 81 40 00 1d+04:14:54.433 READ FPDMA QUEUED Error 594 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 00 00 40 00 1d+04:00:28.868 READ FPDMA QUEUED ec 00 00 00 00 00 00 00 1d+04:00:28.867 IDENTIFY DEVICE ec 00 00 00 00 00 00 00 1d+04:00:28.843 IDENTIFY DEVICE ec 00 00 00 00 00 00 00 1d+04:00:28.798 IDENTIFY DEVICE 60 00 00 00 29 81 40 00 1d+04:00:28.763 READ FPDMA QUEUED Error 593 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 29 81 40 00 1d+04:00:28.586 READ FPDMA QUEUED 60 00 00 00 28 81 40 00 1d+04:00:28.585 READ FPDMA QUEUED 60 00 00 00 27 81 40 00 1d+04:00:28.584 READ FPDMA QUEUED 60 00 00 00 26 81 40 00 1d+04:00:28.584 READ FPDMA QUEUED 60 00 00 00 03 00 40 00 1d+04:00:28.584 READ FPDMA QUEUED Error 592 occurred at disk power-on lifetime: 2685 hours (111 days + 21 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 41 00 00 00 00 00 Error: ICRC, ABRT at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 00 28 81 40 00 1d+04:00:28.107 READ FPDMA QUEUED 60 00 00 00 27 81 40 00 1d+04:00:28.106 READ FPDMA QUEUED 60 00 00 00 26 81 40 00 1d+04:00:28.103 READ FPDMA QUEUED 60 00 00 00 03 00 40 00 1d+04:00:28.075 READ FPDMA QUEUED 2f 00 01 10 00 00 00 00 1d+04:00:28.068 READ LOG EXT SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 2685 - # 2 Short offline Completed without error 00% 1848 - # 3 Short offline Completed without error 00% 1824 - # 4 Short offline Completed without error 00% 1800 - # 5 Short offline Completed without error 00% 1788 - # 6 Short offline Completed without error 00% 1776 - # 7 Short offline Completed without error 00% 1752 - # 8 Short offline Completed without error 00% 1728 - # 9 Short offline Completed without error 00% 1704 - #10 Short offline Completed without error 00% 1688 - #11 Short offline Completed without error 00% 1681 - #12 Short offline Completed without error 00% 1672 - #13 Short offline Completed without error 00% 1513 - #14 Short offline Completed without error 00% 1489 - #15 Short offline Completed without error 00% 1484 - #16 Short offline Completed without error 00% 1420 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
I see a lot of "Error ### occurred at disk power-on lifetime..." errors on the output, but I also see those on /dev/da2 and /dev/da4, but no critical errors for those devices show up in the FreeNAS alerts. Any troubleshooting suggestions you guys can offer?
Last edited: