Hey guys,
I'm in a tough spot here. I have a 10 disk Raid-Z2 config that has checksum errors that a scrub won't correct. It's on a TS140 server with a Perc H310 in IT mode. It has 32GB of ECC RAM that passes memtest86. Several of the drives are on the intel SATA controller, including /dev/ada2.
System log shows this:
zpool status:
Smart shows the following:
If I try to copy the file through samba, I get a file IO error and it refuses to continue. I thought Raid-Z2 should correct this type of issue on the fly, and if not, a scrub should do it, right? What could possibly be the issue here?
Thanks in advance.
I'm in a tough spot here. I have a 10 disk Raid-Z2 config that has checksum errors that a scrub won't correct. It's on a TS140 server with a Perc H310 in IT mode. It has 32GB of ECC RAM that passes memtest86. Several of the drives are on the intel SATA controller, including /dev/ada2.
System log shows this:
Code:
Apr 14 12:23:14 meowmeows smartd[2651]: Device: /dev/ada2, 1 Currently unreadable (pending) sectors Apr 14 12:23:14 meowmeows smartd[2651]: Device: /dev/ada2, 1 Offline uncorrectable sectors
zpool status:
Code:
zpool status pool: meowtank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 2h33m with 2 errors on Thu Apr 13 22:53:59 2017 config: NAME STATE READ WRITE CKSUM meowtank ONLINE 0 0 2 raidz2-0 ONLINE 0 0 4 gptid/bdbcd68a-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/be32356b-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/be633c02-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/be87fb6f-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/bfc1d69d-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/c00a52e2-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/c09b05e3-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/c0b60379-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/c0fe6af3-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 gptid/c1ff04b8-1ce8-11e7-980b-fc4dd4d2bec9 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files:
Smart shows the following:
Code:
smartctl -ax /dev/ada2 smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WL3000GSA6472E Serial Number: WOL240275551 LU WWN Device Id: 5 0014ee 25df939e5 Firmware Version: 01.01A01 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Apr 14 13:05:36 2017 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Disabled Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (31680) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 344) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 1 3 Spin_Up_Time POS--K 151 141 021 - 11441 4 Start_Stop_Count -O--CK 100 100 000 - 61 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 7 Seek_Error_Rate -OSR-K 200 200 000 - 0 9 Power_On_Hours -O--CK 064 064 000 - 26785 10 Spin_Retry_Count -O--CK 100 253 000 - 0 11 Calibration_Retry_Count -O--CK 100 253 000 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 61 183 Runtime_Bad_Block -O--CK 100 100 000 - 0 192 Power-Off_Retract_Count -O--CK 200 200 000 - 26 193 Load_Cycle_Count -O--CK 200 200 000 - 34 194 Temperature_Celsius -O---K 115 103 000 - 37 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 197 Current_Pending_Sector -O--CK 200 200 000 - 1 198 Offline_Uncorrectable ----CK 200 200 000 - 1 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 7 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 5 Comprehensive SMART error log 0x03 GPL R/O 6 Ext. Comprehensive SMART error log 0x04 GPL,SL R/O 8 Device Statistics log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 SATA NCQ Queued Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x24 GPL R/O 1 Current Device Internal Status Data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xa0-0xa7 GPL,SL VS 16 Device vendor specific log 0xa8-0xb7 GPL,SL VS 1 Device vendor specific log 0xbd GPL,SL VS 1 Device vendor specific log 0xc0 GPL,SL VS 1 Device vendor specific log 0xc1 GPL VS 93 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (6 sectors) No Errors Logged SMART Error Log Version: 1 No Errors Logged SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 26781 - # 2 Short offline Completed without error 00% 26050 - # 3 Short offline Completed without error 00% 25955 - # 4 Extended offline Completed without error 00% 25795 - # 5 Short offline Completed without error 00% 25646 - # 6 Short offline Completed without error 00% 25551 - # 7 Extended offline Completed without error 00% 25461 - # 8 Short offline Completed without error 00% 25407 - # 9 Short offline Completed without error 00% 25311 - #10 Short offline Completed without error 00% 25167 - #11 Extended offline Completed without error 00% 25125 - #12 Short offline Completed without error 00% 25047 - #13 Short offline Completed without error 00% 24903 - #14 Short offline Completed without error 00% 24807 - #15 Extended offline Completed without error 00% 24718 - #16 Short offline Completed without error 00% 24663 - #17 Short offline Completed without error 00% 24567 - #18 Short offline Completed without error 00% 24424 - SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 26781 - # 2 Short offline Completed without error 00% 26050 - # 3 Short offline Completed without error 00% 25955 - # 4 Extended offline Completed without error 00% 25795 - # 5 Short offline Completed without error 00% 25646 - # 6 Short offline Completed without error 00% 25551 - # 7 Extended offline Completed without error 00% 25461 - # 8 Short offline Completed without error 00% 25407 - # 9 Short offline Completed without error 00% 25311 - #10 Short offline Completed without error 00% 25167 - #11 Extended offline Completed without error 00% 25125 - #12 Short offline Completed without error 00% 25047 - #13 Short offline Completed without error 00% 24903 - #14 Short offline Completed without error 00% 24807 - #15 Extended offline Completed without error 00% 24718 - #16 Short offline Completed without error 00% 24663 - #17 Short offline Completed without error 00% 24567 - #18 Short offline Completed without error 00% 24424 - #19 Extended offline Completed without error 00% 24382 - #20 Short offline Completed without error 00% 24304 - #21 Short offline Completed without error 00% 24160 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 3 SCT Version (vendor specific): 258 (0x0102) SCT Support Level: 1 Device State: Active (0) Current Temperature: 37 Celsius Power Cycle Min/Max Temperature: 25/37 Celsius Lifetime Min/Max Temperature: 0/49 Celsius Under/Over Temperature Limit Count: 0/0 Vendor specific: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -41/85 Celsius Temperature History Size (Index): 478 (263) Index Estimated Time Temperature Celsius 264 2017-04-14 05:08 34 *************** 265 2017-04-14 05:09 34 *************** 266 2017-04-14 05:10 33 ************** ... ..( 91 skipped). .. ************** 358 2017-04-14 06:42 33 ************** 359 2017-04-14 06:43 34 *************** ... ..( 52 skipped). .. *************** 412 2017-04-14 07:36 34 *************** 413 2017-04-14 07:37 35 **************** ... ..( 18 skipped). .. **************** 432 2017-04-14 07:56 35 **************** 433 2017-04-14 07:57 36 ***************** ... ..( 24 skipped). .. ***************** 458 2017-04-14 08:22 36 ***************** 459 2017-04-14 08:23 37 ****************** ... ..( 47 skipped). .. ****************** 29 2017-04-14 09:11 37 ****************** 30 2017-04-14 09:12 36 ***************** ... ..( 69 skipped). .. ***************** 100 2017-04-14 10:22 36 ***************** 101 2017-04-14 10:23 35 **************** ... ..(133 skipped). .. **************** 235 2017-04-14 12:37 35 **************** 236 2017-04-14 12:38 34 *************** ... ..( 26 skipped). .. *************** 263 2017-04-14 13:05 34 *************** SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) Device Statistics (GP Log 0x04) Page Offset Size Value Flags Description 0x01 ===== = = === == General Statistics (rev 2) == 0x01 0x008 4 61 --- Lifetime Power-On Resets 0x01 0x010 4 26785 --- Power-on Hours 0x01 0x018 6 5634759502216 --- Logical Sectors Written 0x01 0x020 6 704910901 --- Number of Write Commands 0x01 0x028 6 140317546912 --- Logical Sectors Read 0x01 0x030 6 1061479940 --- Number of Read Commands 0x03 ===== = = === == Rotating Media Statistics (rev 1) == 0x03 0x008 4 25305 --- Spindle Motor Power-on Hours 0x03 0x010 4 25305 --- Head Flying Hours 0x03 0x018 4 61 --- Head Load Events 0x03 0x020 4 200 N-- Number of Reallocated Logical Sectors 0x03 0x028 4 427 --- Read Recovery Attempts 0x03 0x030 4 0 --- Number of Mechanical Start Failures 0x04 ===== = = === == General Errors Statistics (rev 1) == 0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors 0x04 0x010 4 0 --- Resets Between Cmd Acceptance and Completion 0x05 ===== = = === == Temperature Statistics (rev 1) == 0x05 0x008 1 37 --- Current Temperature 0x05 0x010 1 34 --- Average Short Term Temperature 0x05 0x018 1 34 --- Average Long Term Temperature 0x05 0x020 1 49 --- Highest Temperature 0x05 0x028 1 23 --- Lowest Temperature 0x05 0x030 1 46 --- Highest Average Short Term Temperature 0x05 0x038 1 30 --- Lowest Average Short Term Temperature 0x05 0x040 1 45 --- Highest Average Long Term Temperature 0x05 0x048 1 34 --- Lowest Average Long Term Temperature 0x05 0x050 4 0 --- Time in Over-Temperature 0x05 0x058 1 60 --- Specified Maximum Operating Temperature 0x05 0x060 4 0 --- Time in Under-Temperature 0x05 0x068 1 0 --- Specified Minimum Operating Temperature 0x06 ===== = = === == Transport Statistics (rev 1) == 0x06 0x008 4 636 --- Number of Hardware Resets 0x06 0x010 4 1749 --- Number of ASR Events 0x06 0x018 4 0 --- Number of Interface CRC Errors |||_ C monitored condition met ||__ D supports DSN |___ N normalized value SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 2 0 Command failed due to ICRC error 0x0002 2 0 R_ERR response for data FIS 0x0003 2 0 R_ERR response for device-to-host data FIS 0x0004 2 0 R_ERR response for host-to-device data FIS 0x0005 2 0 R_ERR response for non-data FIS 0x0006 2 0 R_ERR response for device-to-host non-data FIS 0x0007 2 0 R_ERR response for host-to-device non-data FIS 0x0008 2 0 Device-to-host non-data FIS retries 0x0009 2 12 Transition from drive PhyRdy to drive PhyNRdy 0x000a 2 12 Device-to-host register FISes sent due to a COMRESET 0x000b 2 0 CRC errors within host-to-device FIS 0x000f 2 0 R_ERR response for host-to-device data FIS, CRC 0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC 0x8000 4 60450 Vendor specific
If I try to copy the file through samba, I get a file IO error and it refuses to continue. I thought Raid-Z2 should correct this type of issue on the fly, and if not, a scrub should do it, right? What could possibly be the issue here?
Thanks in advance.
Last edited by a moderator: