Hi,
I have recently repurposed an older PC to be a NAS however I have been getting a lot of errors.
I have no important data on this system. Still in the testing phase.
Hardware:
CPU: i7-3820 CPU
Memory: 64GB, DDR3-1600 CL10
MOBO: Asus Sabertooth X79
Controller: H310 RAID Controller Card
PSU: Cooler Master Silent Pro Hybrid 850 W 80+ Gold
Storage: 8x ST16000NM001G-2KK103
Boot Storage: ST320LT007-9ZV142
Everything has direct airflow and is running at a good temp. I initially installed core but migrated to scale.
This has been running for about a week with minimal load and so far three drives have started having issues.
HDD: ZL2AM7B3
HDD:ZL26AYKJ
HDD:ZL235TEN
The last one I have left in while I have a camera record to the pool. ZFS reporting errors.
Of note, the 2nd and 3rd drives passed extended tests before they failed.
This failure rate seems very high. Perhaps it is a bad batch of drives but perhaps I am also missing something.
Is there a recommended burn in test I can put the rest of these drives through? At this point my plan is to wipe the whole thing and start fresh with replacement drives, but I want to be sure its not some other issue and I would also like to increase my confidence that the remaining 5 drives are not going to fail in the next month.
Thanks for any help.
I have recently repurposed an older PC to be a NAS however I have been getting a lot of errors.
I have no important data on this system. Still in the testing phase.
Hardware:
CPU: i7-3820 CPU
Memory: 64GB, DDR3-1600 CL10
MOBO: Asus Sabertooth X79
Controller: H310 RAID Controller Card
PSU: Cooler Master Silent Pro Hybrid 850 W 80+ Gold
Storage: 8x ST16000NM001G-2KK103
Boot Storage: ST320LT007-9ZV142
Everything has direct airflow and is running at a good temp. I initially installed core but migrated to scale.
This has been running for about a week with minimal load and so far three drives have started having issues.
HDD: ZL2AM7B3
Code:
root@truenas[~]# smartctl -a /dev/da4 smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p2 amd64] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Exos X16 Device Model: ST16000NM001G-2KK103 Serial Number: ZL2AM7B3 LU WWN Device Id: 5 000c50 0c7bfca19 Firmware Version: SN04 User Capacity: 16,000,900,661,248 bytes [16.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Feb 7 00:33:08 2023 AEDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: ( 575) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1435) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 066 064 044 Pre-fail Always - 4297554 3 Spin_Up_Time 0x0003 098 098 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 3 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 253 045 Pre-fail Always - 437840 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 169 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 3 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 070 052 040 Old_age Always - 30 (Min/Max 28/48) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 1 193 Load_Cycle_Count 0x0032 099 099 000 Old_age Always - 2020 194 Temperature_Celsius 0x0022 030 048 000 Old_age Always - 30 (0 21 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 32 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 32 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 28h+44m+11.203s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 4292756 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 4798 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 134 4178936 # 2 Extended offline Completed: read failure 90% 134 4178936 # 3 Conveyance offline Completed without error 00% 134 - # 4 Extended offline Completed: read failure 90% 133 4178936 # 5 Short offline Completed without error 00% 133 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
HDD:ZL26AYKJ
Code:
root@truenas[/dev]# smartctl -a /dev/sda smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Exos X16 Device Model: ST16000NM001G-2KK103 Serial Number: ZL26AYKJ LU WWN Device Id: 5 000c50 0c63a8746 Firmware Version: SN04 User Capacity: 16,000,900,661,248 bytes [16.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Wed Feb 8 21:57:14 2023 AEDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: ( 567) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1441) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 070 064 044 Pre-fail Always - 9288806 3 Spin_Up_Time 0x0003 096 096 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 4 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 070 060 045 Pre-fail Always - 9299644 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 214 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 4 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 066 060 040 Old_age Always - 34 (Min/Max 29/35) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 2 193 Load_Cycle_Count 0x0032 099 099 000 Old_age Always - 2237 194 Temperature_Celsius 0x0022 034 040 000 Old_age Always - 34 (0 22 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 8 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 8 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 53h+04m+19.441s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 7992572 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1296234 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 214 138458528 # 2 Conveyance offline Completed without error 00% 214 - # 3 Short offline Completed without error 00% 214 - # 4 Extended offline Completed without error 00% 155 - # 5 Short offline Completed without error 00% 133 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
HDD:ZL235TEN
Code:
root@truenas[~]# smartctl -a /dev/sdf smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Exos X16 Device Model: ST16000NM001G-2KK103 Serial Number: ZL235TEN LU WWN Device Id: 5 000c50 0c47df593 Firmware Version: SN04 User Capacity: 16,000,900,661,248 bytes [16.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sat Feb 11 08:46:51 2023 AEDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: ( 567) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: (1459) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 057 057 044 Pre-fail Always - 57438431 3 Spin_Up_Time 0x0003 095 095 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 5 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 2368 7 Seek_Error_Rate 0x000f 073 060 045 Pre-fail Always - 19789590 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 273 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 5 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 016 016 000 Old_age Always - 84 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 65537 190 Airflow_Temperature_Cel 0x0022 064 051 040 Old_age Always - 36 (Min/Max 32/42) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 3 193 Load_Cycle_Count 0x0032 099 099 000 Old_age Always - 2268 194 Temperature_Celsius 0x0022 036 049 000 Old_age Always - 36 (0 21 0 0 0) 197 Current_Pending_Sector 0x0012 092 092 000 Old_age Always - 4632 198 Offline_Uncorrectable 0x0010 092 092 000 Old_age Offline - 4632 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 116h+14m+38.644s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 52778755 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 4659676 SMART Error Log Version: 1 ATA Error Count: 84 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 84 occurred at disk power-on lifetime: 273 hours (11 days + 9 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 a0 81 88 0d Error: WP at LBA = 0x0d8881a0 = 227049888 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 61 00 28 18 da 10 43 00 2d+09:46:47.693 WRITE FPDMA QUEUED 61 00 10 ff ff ff 4f 00 2d+09:46:47.311 WRITE FPDMA QUEUED 61 00 10 ff ff ff 4f 00 2d+09:46:47.310 WRITE FPDMA QUEUED 61 00 10 90 02 40 40 00 2d+09:46:47.309 WRITE FPDMA QUEUED 60 00 10 ff ff ff 4f 00 2d+09:46:47.309 READ FPDMA QUEUED Error 83 occurred at disk power-on lifetime: 273 hours (11 days + 9 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 78 81 88 0d Error: UNC at LBA = 0x0d888178 = 227049848 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 28 c8 81 88 4d 00 2d+09:46:44.582 READ FPDMA QUEUED 60 00 28 98 81 88 4d 00 2d+09:46:44.489 READ FPDMA QUEUED 60 00 30 68 81 88 4d 00 2d+09:46:44.489 READ FPDMA QUEUED 60 00 20 90 7c 88 4d 00 2d+09:46:44.487 READ FPDMA QUEUED 60 00 30 60 7c 88 4d 00 2d+09:46:44.487 READ FPDMA QUEUED Error 82 occurred at disk power-on lifetime: 273 hours (11 days + 9 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 e8 74 88 0d Error: WP at LBA = 0x0d8874e8 = 227046632 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 61 00 10 50 da 10 43 00 2d+09:46:41.992 WRITE FPDMA QUEUED 61 00 10 ff ff ff 4f 00 2d+09:46:39.671 WRITE FPDMA QUEUED 61 00 10 ff ff ff 4f 00 2d+09:46:39.671 WRITE FPDMA QUEUED 61 00 10 90 02 40 40 00 2d+09:46:39.670 WRITE FPDMA QUEUED 60 00 10 ff ff ff 4f 00 2d+09:46:39.670 READ FPDMA QUEUED Error 81 occurred at disk power-on lifetime: 273 hours (11 days + 9 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 d0 74 88 0d Error: WP at LBA = 0x0d8874d0 = 227046608 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 61 00 10 ff ff ff 4f 00 2d+09:46:36.830 WRITE FPDMA QUEUED 61 00 10 ff ff ff 4f 00 2d+09:46:36.829 WRITE FPDMA QUEUED 61 00 10 90 02 40 40 00 2d+09:46:36.828 WRITE FPDMA QUEUED 60 00 10 ff ff ff 4f 00 2d+09:46:36.827 READ FPDMA QUEUED 60 00 10 ff ff ff 4f 00 2d+09:46:36.827 READ FPDMA QUEUED Error 80 occurred at disk power-on lifetime: 273 hours (11 days + 9 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 a8 74 88 0d Error: UNC at LBA = 0x0d8874a8 = 227046568 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 30 88 74 88 4d 00 2d+09:46:34.075 READ FPDMA QUEUED 60 00 28 60 74 88 4d 00 2d+09:46:34.062 READ FPDMA QUEUED ea 00 00 00 00 00 00 00 2d+09:46:34.024 FLUSH CACHE EXT 61 00 10 40 da 10 43 00 2d+09:46:34.006 WRITE FPDMA QUEUED 61 00 10 ff ff ff 4f 00 2d+09:46:34.006 WRITE FPDMA QUEUED SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 273 218987208 # 2 Short offline Completed without error 00% 273 - # 3 Extended offline Completed without error 00% 238 - # 4 Extended offline Interrupted (host reset) 00% 215 - # 5 Extended offline Completed without error 00% 156 - # 6 Short offline Completed without error 00% 133 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
The last one I have left in while I have a camera record to the pool. ZFS reporting errors.
Of note, the 2nd and 3rd drives passed extended tests before they failed.
This failure rate seems very high. Perhaps it is a bad batch of drives but perhaps I am also missing something.
Is there a recommended burn in test I can put the rest of these drives through? At this point my plan is to wipe the whole thing and start fresh with replacement drives, but I want to be sure its not some other issue and I would also like to increase my confidence that the remaining 5 drives are not going to fail in the next month.
Thanks for any help.