Pliqui
Dabbler
- Joined
- Apr 24, 2018
- Messages
- 25
This morning I received alert from cron stating that my zpool is in a degrade state.
The only recent activity was to upgrade to 11.1U6
System:
CPU: Intel Xeon E3-1230 V6 Kaby Lake 3.5 GHz - Changed from E3-1225 V6
Mobo: SUPERMICRO MBD-X11SSL-CF Micro ATX*- Changed from SUPERMICRO MBD-X11SSH-F-O Micro ATX
RAM: 4 x Crucial CT16G4WFD824A 16Gb Ddr4 Ecc Unbuff Cl17 (64Gb maxed out)
Case: Fractal Design Node 804
PSU: Seasonic FOCUS Plus Series SSR-650FX 650W
Boot: 1 x SSD 128GB
Pool: stripped-mirror 6 Disks (4 x 8TB and 2 x 10TB)
SAS Cables: 2 x Supermicro MiniSAS HD to 4x SATA
This one of the emails i got
ZPOOL email at 3:00 am
and this is the status now
But I checked the SMART and did not find anything out of the ordinary for the disk with the problem DA0 (the usual culprits IDs 5, 197, and 198 )
The only values that are different from other disks are:
0x0009 2 188 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 189 Device-to-host register FISes sent due to a COMRESET
I'm running a long test on the drive.
I also running schedule tests
SMART:
Long: at 2:00 AM on 08 and 22 of every month all days of the week
Short: at 1:00 AM every 2 days of every month all days of the week
SCRUB:
HOME pool: Threshold days 10, at 2:00 am, on day 1 and 15 of every month all days of the week
Any input will be appreciated.
Thanks
The only recent activity was to upgrade to 11.1U6
System:
CPU: Intel Xeon E3-1230 V6 Kaby Lake 3.5 GHz - Changed from E3-1225 V6
Mobo: SUPERMICRO MBD-X11SSL-CF Micro ATX*- Changed from SUPERMICRO MBD-X11SSH-F-O Micro ATX
RAM: 4 x Crucial CT16G4WFD824A 16Gb Ddr4 Ecc Unbuff Cl17 (64Gb maxed out)
Case: Fractal Design Node 804
PSU: Seasonic FOCUS Plus Series SSR-650FX 650W
Boot: 1 x SSD 128GB
Pool: stripped-mirror 6 Disks (4 x 8TB and 2 x 10TB)
SAS Cables: 2 x Supermicro MiniSAS HD to 4x SATA
Code:
[root@freenas ~]# lspci 00:00.0 Host bridge: Intel Corporation Device 5918 (rev 05) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 05) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 05) 00:13.0 Non-VGA unclassified device: Intel Corporation Sunrise Point-H Integrated Sensor Hub (rev 31) 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31) 00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31) 00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31) 00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31) 00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1) 00:1c.5 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #6 (rev f1) 00:1c.6 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #7 (rev f1) 00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1) 00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31) 00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31) 00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31) 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) 03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) 06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
This one of the emails i got
Code:
Device: /dev/da0 [SAT], failed to read SMART Attribute Data Device: /dev/da0 [SAT], not capable of SMART self-check Device: /dev/da0 [SAT], Read SMART Self-Test Log Failed The volume HOME state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. Device: /dev/da0 [SAT], Read SMART Error Log Failed
ZPOOL email at 3:00 am
Code:
Checking status of zfs pools: NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT HOME 23.6T 12.1T 11.5T - 8% 51% 1.00x DEGRADED /mnt freenas-boot 118G 3.30G 115G - - 2% 1.00x ONLINE - pool: HOME state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0 in 0 days 11:06:22 with 0 errors on Tue Jan 1 13:06:22 2019 config: NAME STATE READ WRITE CKSUM HOME DEGRADED 0 0 0 mirror-0 ONLINE 0 0 0 gptid/df322087-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0 gptid/dfb5acdc-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gptid/e03c826e-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0 gptid/e0c5c181-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0 mirror-2 DEGRADED 0 0 0 gptid/d80c60dc-d264-11e8-afaf-ac1f6b83f450 FAULTED 6 4 0 too many errors gptid/d88326db-d264-11e8-afaf-ac1f6b83f450 ONLINE 0 0 0 errors: No known data errors
and this is the status now
Code:
[root@freenas ~]# zpool status -v pool: HOME state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://illumos.org/msg/ZFS-8000-9P scan: scrub repaired 0 in 0 days 11:06:22 with 0 errors on Tue Jan 1 13:06:22 2019 config: NAME STATE READ WRITE CKSUM HOME DEGRADED 0 0 0 mirror-0 ONLINE 0 0 0 gptid/df322087-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0 gptid/dfb5acdc-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 gptid/e03c826e-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0 gptid/e0c5c181-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0 mirror-2 DEGRADED 0 0 0 gptid/d80c60dc-d264-11e8-afaf-ac1f6b83f450 DEGRADED 0 0 413 too many errors gptid/d88326db-d264-11e8-afaf-ac1f6b83f450 ONLINE 0 0 0 errors: No known data errors
But I checked the SMART and did not find anything out of the ordinary for the disk with the problem DA0 (the usual culprits IDs 5, 197, and 198 )
Code:
[root@freenas ~]# smartctl -x /dev/da0 smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: HGST HDN721010ALE604 Serial Number: 1DGS64JZ LU WWN Device Id: 5 000cca 26cca8b9b Firmware Version: 83XN User Capacity: 10,000,831,348,736 bytes [10.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sat Jan 5 13:48:56 2019 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Disabled Rd look-ahead is: Enabled Write cache is: Enabled DSN feature is: Unavailable ATA Security is: Disabled, NOT FROZEN [SEC1] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 249) Self-test routine in progress... 90% of test remaining. Total time to complete Offline data collection: ( 93) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: (1304) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0 2 Throughput_Performance --S--- 134 134 054 - 96 3 Spin_Up_Time POS--- 100 100 024 - 0 4 Start_Stop_Count -O--C- 100 100 000 - 2 5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0 7 Seek_Error_Rate -O-R-- 100 100 067 - 0 8 Seek_Time_Performance --S--- 128 128 020 - 18 9 Power_On_Hours -O--C- 100 100 000 - 1915 10 Spin_Retry_Count -O--C- 100 100 060 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 2 22 Unknown_Attribute PO---K 100 100 025 - 100 192 Power-Off_Retract_Count -O--CK 100 100 000 - 79 193 Load_Cycle_Count -O--C- 100 100 000 - 79 194 Temperature_Celsius -O---- 162 162 000 - 37 (Min/Max 25/40) 196 Reallocated_Event_Count -O--CK 100 100 000 - 0 197 Current_Pending_Sector -O---K 100 100 000 - 0 198 Offline_Uncorrectable ---R-- 100 100 000 - 0 199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 0 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 1 Comprehensive SMART error log 0x03 GPL R/O 1 Ext. Comprehensive SMART error log 0x04 GPL R/O 256 Device Statistics log 0x04 SL R/O 255 Device Statistics log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x08 GPL R/O 2 Power Conditions log 0x09 SL R/W 1 Selective self-test log 0x0c GPL R/O 5501 Pending Defects log 0x10 GPL R/O 1 NCQ Command Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x12 GPL R/O 1 SATA NCQ Non-Data log 0x13 GPL R/O 1 SATA NCQ Send and Receive log 0x15 GPL R/W 1 Rebuild Assist log 0x21 GPL R/O 1 Write stream error log 0x22 GPL R/O 1 Read stream error log 0x24 GPL R/O 256 Current Device Internal Status Data log 0x25 GPL R/O 256 Saved Device Internal Status Data log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (1 sectors) No Errors Logged SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Aborted by host 90% 1914 - # 2 Short offline Completed without error 00% 1855 - # 3 Short offline Completed without error 00% 1807 - # 4 Short offline Completed without error 00% 1783 - # 5 Short offline Completed without error 00% 1735 - # 6 Short offline Completed without error 00% 1687 - # 7 Short offline Completed without error 00% 1638 - # 8 Short offline Completed without error 00% 1590 - # 9 Extended offline Completed without error 00% 1588 - #10 Short offline Completed without error 00% 1542 - #11 Short offline Completed without error 00% 1494 - #12 Short offline Completed without error 00% 1446 - #13 Short offline Completed without error 00% 1398 - #14 Short offline Completed without error 00% 1350 - #15 Short offline Completed without error 00% 1302 - #16 Short offline Completed without error 00% 1254 - #17 Extended offline Completed without error 00% 1252 - #18 Short offline Completed without error 00% 1206 - #19 Short offline Completed without error 00% 1158 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 3 SCT Version (vendor specific): 256 (0x0100) SCT Support Level: 0 Device State: DST executing in background (3) Current Temperature: 37 Celsius Power Cycle Min/Max Temperature: 27/38 Celsius Lifetime Min/Max Temperature: 25/40 Celsius Under/Over Temperature Limit Count: 0/0 SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -40/70 Celsius Temperature History Size (Index): 128 (37) Index Estimated Time Temperature Celsius 38 2019-01-05 11:41 35 **************** ... ..( 29 skipped). .. **************** 68 2019-01-05 12:11 35 **************** 69 2019-01-05 12:12 36 ***************** 70 2019-01-05 12:13 35 **************** ... ..( 27 skipped). .. **************** 98 2019-01-05 12:41 35 **************** 99 2019-01-05 12:42 36 ***************** ... ..( 14 skipped). .. ***************** 114 2019-01-05 12:57 36 ***************** 115 2019-01-05 12:58 35 **************** 116 2019-01-05 12:59 36 ***************** 117 2019-01-05 13:00 35 **************** 118 2019-01-05 13:01 35 **************** 119 2019-01-05 13:02 36 ***************** ... ..( 40 skipped). .. ***************** 32 2019-01-05 13:43 36 ***************** 33 2019-01-05 13:44 37 ****************** ... ..( 2 skipped). .. ****************** 36 2019-01-05 13:47 37 ****************** 37 2019-01-05 13:48 35 **************** SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) Device Statistics (GP/SMART Log 0x04) not supported Pending Defects log (GP Log 0x0c) supported [please try: '-l defects'] SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 2 0 Command failed due to ICRC error 0x0002 2 0 R_ERR response for data FIS 0x0003 2 0 R_ERR response for device-to-host data FIS 0x0004 2 0 R_ERR response for host-to-device data FIS 0x0005 2 0 R_ERR response for non-data FIS 0x0006 2 0 R_ERR response for device-to-host non-data FIS 0x0007 2 0 R_ERR response for host-to-device non-data FIS 0x0008 2 0 Device-to-host non-data FIS retries 0x0009 2 188 Transition from drive PhyRdy to drive PhyNRdy 0x000a 2 189 Device-to-host register FISes sent due to a COMRESET 0x000b 2 0 CRC errors within host-to-device FIS 0x000d 2 0 Non-CRC errors within host-to-device FIS
The only values that are different from other disks are:
0x0009 2 188 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 189 Device-to-host register FISes sent due to a COMRESET
I'm running a long test on the drive.
I also running schedule tests
SMART:
Long: at 2:00 AM on 08 and 22 of every month all days of the week
Short: at 1:00 AM every 2 days of every month all days of the week
SCRUB:
HOME pool: Threshold days 10, at 2:00 am, on day 1 and 15 of every month all days of the week
Any input will be appreciated.
Thanks