Pliqui
Dabbler
- Joined
- Apr 24, 2018
- Messages
- 25
This morning I received alert from cron stating that my zpool is in a degrade state.
The only recent activity was to upgrade to 11.1U6
System:
CPU: Intel Xeon E3-1230 V6 Kaby Lake 3.5 GHz - Changed from E3-1225 V6
Mobo: SUPERMICRO MBD-X11SSL-CF Micro ATX*- Changed from SUPERMICRO MBD-X11SSH-F-O Micro ATX
RAM: 4 x Crucial CT16G4WFD824A 16Gb Ddr4 Ecc Unbuff Cl17 (64Gb maxed out)
Case: Fractal Design Node 804
PSU: Seasonic FOCUS Plus Series SSR-650FX 650W
Boot: 1 x SSD 128GB
Pool: stripped-mirror 6 Disks (4 x 8TB and 2 x 10TB)
SAS Cables: 2 x Supermicro MiniSAS HD to 4x SATA
This one of the emails i got
ZPOOL email at 3:00 am
and this is the status now
But I checked the SMART and did not find anything out of the ordinary for the disk with the problem DA0 (the usual culprits IDs 5, 197, and 198 )
The only values that are different from other disks are:
0x0009 2 188 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 189 Device-to-host register FISes sent due to a COMRESET
I'm running a long test on the drive.
I also running schedule tests
SMART:
Long: at 2:00 AM on 08 and 22 of every month all days of the week
Short: at 1:00 AM every 2 days of every month all days of the week
SCRUB:
HOME pool: Threshold days 10, at 2:00 am, on day 1 and 15 of every month all days of the week
Any input will be appreciated.
Thanks
The only recent activity was to upgrade to 11.1U6
System:
CPU: Intel Xeon E3-1230 V6 Kaby Lake 3.5 GHz - Changed from E3-1225 V6
Mobo: SUPERMICRO MBD-X11SSL-CF Micro ATX*- Changed from SUPERMICRO MBD-X11SSH-F-O Micro ATX
RAM: 4 x Crucial CT16G4WFD824A 16Gb Ddr4 Ecc Unbuff Cl17 (64Gb maxed out)
Case: Fractal Design Node 804
PSU: Seasonic FOCUS Plus Series SSR-650FX 650W
Boot: 1 x SSD 128GB
Pool: stripped-mirror 6 Disks (4 x 8TB and 2 x 10TB)
SAS Cables: 2 x Supermicro MiniSAS HD to 4x SATA
Code:
[root@freenas ~]# lspci 00:00.0 Host bridge: Intel Corporation Device 5918 (rev 05) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 05) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 05) 00:13.0 Non-VGA unclassified device: Intel Corporation Sunrise Point-H Integrated Sensor Hub (rev 31) 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31) 00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31) 00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31) 00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31) 00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1) 00:1c.5 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #6 (rev f1) 00:1c.6 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #7 (rev f1) 00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1) 00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31) 00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31) 00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31) 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) 03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) 06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
This one of the emails i got
Code:
Device: /dev/da0 [SAT], failed to read SMART Attribute Data Device: /dev/da0 [SAT], not capable of SMART self-check Device: /dev/da0 [SAT], Read SMART Self-Test Log Failed The volume HOME state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. Device: /dev/da0 [SAT], Read SMART Error Log Failed
ZPOOL email at 3:00 am
Code:
Checking status of zfs pools:
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
HOME 23.6T 12.1T 11.5T - 8% 51% 1.00x DEGRADED /mnt
freenas-boot 118G 3.30G 115G - - 2% 1.00x ONLINE -
pool: HOME
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0 in 0 days 11:06:22 with 0 errors on Tue Jan 1 13:06:22 2019
config:
NAME STATE READ WRITE CKSUM
HOME DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
gptid/df322087-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0
gptid/dfb5acdc-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/e03c826e-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0
gptid/e0c5c181-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0
mirror-2 DEGRADED 0 0 0
gptid/d80c60dc-d264-11e8-afaf-ac1f6b83f450 FAULTED 6 4 0 too many errors
gptid/d88326db-d264-11e8-afaf-ac1f6b83f450 ONLINE 0 0 0
errors: No known data errorsand this is the status now
Code:
[root@freenas ~]# zpool status -v
pool: HOME
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: scrub repaired 0 in 0 days 11:06:22 with 0 errors on Tue Jan 1 13:06:22 2019
config:
NAME STATE READ WRITE CKSUM
HOME DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
gptid/df322087-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0
gptid/dfb5acdc-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/e03c826e-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0
gptid/e0c5c181-7949-11e8-8cf1-ac1f6b83f450 ONLINE 0 0 0
mirror-2 DEGRADED 0 0 0
gptid/d80c60dc-d264-11e8-afaf-ac1f6b83f450 DEGRADED 0 0 413 too many errors
gptid/d88326db-d264-11e8-afaf-ac1f6b83f450 ONLINE 0 0 0
errors: No known data errors
But I checked the SMART and did not find anything out of the ordinary for the disk with the problem DA0 (the usual culprits IDs 5, 197, and 198 )
Code:
[root@freenas ~]# smartctl -x /dev/da0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: HGST HDN721010ALE604
Serial Number: 1DGS64JZ
LU WWN Device Id: 5 000cca 26cca8b9b
Firmware Version: 83XN
User Capacity: 10,000,831,348,736 bytes [10.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jan 5 13:48:56 2019 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: ( 93) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (1304) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate PO-R-- 100 100 016 - 0
2 Throughput_Performance --S--- 134 134 054 - 96
3 Spin_Up_Time POS--- 100 100 024 - 0
4 Start_Stop_Count -O--C- 100 100 000 - 2
5 Reallocated_Sector_Ct PO--CK 100 100 005 - 0
7 Seek_Error_Rate -O-R-- 100 100 067 - 0
8 Seek_Time_Performance --S--- 128 128 020 - 18
9 Power_On_Hours -O--C- 100 100 000 - 1915
10 Spin_Retry_Count -O--C- 100 100 060 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 2
22 Unknown_Attribute PO---K 100 100 025 - 100
192 Power-Off_Retract_Count -O--CK 100 100 000 - 79
193 Load_Cycle_Count -O--C- 100 100 000 - 79
194 Temperature_Celsius -O---- 162 162 000 - 37 (Min/Max 25/40)
196 Reallocated_Event_Count -O--CK 100 100 000 - 0
197 Current_Pending_Sector -O---K 100 100 000 - 0
198 Offline_Uncorrectable ---R-- 100 100 000 - 0
199 UDMA_CRC_Error_Count -O-R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 1 Comprehensive SMART error log
0x03 GPL R/O 1 Ext. Comprehensive SMART error log
0x04 GPL R/O 256 Device Statistics log
0x04 SL R/O 255 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x0c GPL R/O 5501 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x12 GPL R/O 1 SATA NCQ Non-Data log
0x13 GPL R/O 1 SATA NCQ Send and Receive log
0x15 GPL R/W 1 Rebuild Assist log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x24 GPL R/O 256 Current Device Internal Status Data log
0x25 GPL R/O 256 Saved Device Internal Status Data log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Aborted by host 90% 1914 -
# 2 Short offline Completed without error 00% 1855 -
# 3 Short offline Completed without error 00% 1807 -
# 4 Short offline Completed without error 00% 1783 -
# 5 Short offline Completed without error 00% 1735 -
# 6 Short offline Completed without error 00% 1687 -
# 7 Short offline Completed without error 00% 1638 -
# 8 Short offline Completed without error 00% 1590 -
# 9 Extended offline Completed without error 00% 1588 -
#10 Short offline Completed without error 00% 1542 -
#11 Short offline Completed without error 00% 1494 -
#12 Short offline Completed without error 00% 1446 -
#13 Short offline Completed without error 00% 1398 -
#14 Short offline Completed without error 00% 1350 -
#15 Short offline Completed without error 00% 1302 -
#16 Short offline Completed without error 00% 1254 -
#17 Extended offline Completed without error 00% 1252 -
#18 Short offline Completed without error 00% 1206 -
#19 Short offline Completed without error 00% 1158 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 0
Device State: DST executing in background (3)
Current Temperature: 37 Celsius
Power Cycle Min/Max Temperature: 27/38 Celsius
Lifetime Min/Max Temperature: 25/40 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -40/70 Celsius
Temperature History Size (Index): 128 (37)
Index Estimated Time Temperature Celsius
38 2019-01-05 11:41 35 ****************
... ..( 29 skipped). .. ****************
68 2019-01-05 12:11 35 ****************
69 2019-01-05 12:12 36 *****************
70 2019-01-05 12:13 35 ****************
... ..( 27 skipped). .. ****************
98 2019-01-05 12:41 35 ****************
99 2019-01-05 12:42 36 *****************
... ..( 14 skipped). .. *****************
114 2019-01-05 12:57 36 *****************
115 2019-01-05 12:58 35 ****************
116 2019-01-05 12:59 36 *****************
117 2019-01-05 13:00 35 ****************
118 2019-01-05 13:01 35 ****************
119 2019-01-05 13:02 36 *****************
... ..( 40 skipped). .. *****************
32 2019-01-05 13:43 36 *****************
33 2019-01-05 13:44 37 ******************
... ..( 2 skipped). .. ******************
36 2019-01-05 13:47 37 ******************
37 2019-01-05 13:48 35 ****************
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) supported [please try: '-l defects']
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 188 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 189 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FISThe only values that are different from other disks are:
0x0009 2 188 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 189 Device-to-host register FISes sent due to a COMRESET
I'm running a long test on the drive.
I also running schedule tests
SMART:
Long: at 2:00 AM on 08 and 22 of every month all days of the week
Short: at 1:00 AM every 2 days of every month all days of the week
SCRUB:
HOME pool: Threshold days 10, at 2:00 am, on day 1 and 15 of every month all days of the week
Any input will be appreciated.
Thanks
