Hello!
I just received an mail from my freenas-fileserver with the following message:
The pool is an z2 pool with 4 drives (Seagate Archive 8TB) attached to an IBM1015 controller. I just ran an lengthy write operation to the volume (40GByter) and I fear the problem might have occured due to the hard disks overheating, as the server ist standing in a badly ventilated position at the moment. Or maybe it is caused by these SMR hard drives to become slower and slower with persistent writes. I expect the hard disks to probably run fine again after cooling down. Also I want to check the SATA connectors. But at the moment I have the system just running, because I don't want to change anything or even shutdown before consulting you.
Volume status:
Smart Data for da2:
Log for da2:
Smart Data for da3:
Log for da3:
I hope you can help me get this running again.
Thank you very much!
PS:
I have shut the server down now, because I think keeping it running in this state until there's a solution is too critical.
I just received an mail from my freenas-fileserver with the following message:
The volume z2pool state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state.
The pool is an z2 pool with 4 drives (Seagate Archive 8TB) attached to an IBM1015 controller. I just ran an lengthy write operation to the volume (40GByter) and I fear the problem might have occured due to the hard disks overheating, as the server ist standing in a badly ventilated position at the moment. Or maybe it is caused by these SMR hard drives to become slower and slower with persistent writes. I expect the hard disks to probably run fine again after cooling down. Also I want to check the SATA connectors. But at the moment I have the system just running, because I don't want to change anything or even shutdown before consulting you.
Volume status:
root@freenas:~ # zpool status -v
pool: dotsystem
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:03 with 0 errors on Sat Jun 1 02:00:05 2019
config:
NAME STATE READ WRITE CKSUM
dotsystem ONLINE 0 0 0
gptid/96e8ade3-ce37-11e4-98ab-000c29055118 ONLINE 0 0 0
errors: No known data errors
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:31 with 0 errors on Mon Nov 11 03:45:31 2019
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0
errors: No known data errors
pool: z2pool
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: resilvered 372K in 0 days 00:00:01 with 0 errors on Tue Nov 12 20:14:13 2019
config:
NAME STATE READ WRITE CKSUM
z2pool DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/2c5d710c-b611-814e-a225-1aa51e68a575 FAULTED 0 355 0 too many errors
gptid/cc2da5ee-733a-c148-bf15-d20be715c5f2 ONLINE 0 0 0
gptid/45adb761-b939-284f-9019-e0ccf35e9c73 FAULTED 0 249 0 too many errors
gptid/9ce5247f-331b-3148-9b38-c958d2bd057a ONLINE 0 0 0
errors: No known data errors
Smart Data for da2:
root@freenas:~ # smartctl -a -q noserial /dev/da2
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Archive HDD
Device Model: ST8000AS0002-1NA17Z
Firmware Version: AR13
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5980 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Nov 16 21:34:19 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 953) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x30a5) SCT Status supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 114 099 006 Pre-fail Always - 62055736
3 Spin_Up_Time 0x0003 092 088 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1679
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 080 060 030 Pre-fail Always - 106824203
9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 8180
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 741
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 061 033 045 Old_age Always In_the_past 39 (1 32 39 37 0)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 671
193 Load_Cycle_Count 0x0032 098 098 000 Old_age Always - 5698
194 Temperature_Celsius 0x0022 039 067 000 Old_age Always - 39 (0 17 0 0 0)
195 Hardware_ECC_Recovered 0x001a 114 099 000 Old_age Always - 62055736
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 2585 (24 225 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 14481535731
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 103506685664
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 6990 -
# 2 Extended offline Interrupted (host reset) 40% 6634 -
# 3 Short offline Completed without error 00% 6393 -
# 4 Short offline Completed without error 00% 6049 -
# 5 Short offline Completed without error 00% 5983 -
# 6 Short offline Completed without error 00% 5856 -
# 7 Short offline Completed without error 00% 5824 -
# 8 Short offline Completed without error 00% 5779 -
# 9 Short offline Completed without error 00% 5707 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Log for da2:
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): CAM status: SCSI Status Error
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI status: Check Condition
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): Retrying command (per sense data)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): CAM status: SCSI Status Error
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI status: Check Condition
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): Retrying command (per sense data)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): CAM status: SCSI Status Error
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI status: Check Condition
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): Retrying command (per sense data)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): CAM status: SCSI Status Error
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI status: Check Condition
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): Retrying command (per sense data)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): CAM status: SCSI Status Error
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI status: Check Condition
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 18:48:58 freenas (da2:mps0:0:0:0): Error 5, Retries exhausted
Nov 16 18:48:58 freenas GEOM_PART: da2 was automatically resized.
Nov 16 18:48:58 freenas Use `gpart commit da2` to save changes or `gpart undo da2` to revert them.
Nov 16 18:48:58 freenas GEOM_PART: integrity check failed (da2, GPT)
Smart Data for da3:
root@freenas:~ # smartctl -a -q noserial /dev/da3
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Archive HDD
Device Model: ST8000AS0002-1NA17Z
Firmware Version: AR13
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5980 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Nov 16 21:39:47 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 957) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x30a5) SCT Status supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 114 099 006 Pre-fail Always - 64060608
3 Spin_Up_Time 0x0003 092 089 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 099 099 020 Old_age Always - 1729
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 073 060 030 Pre-fail Always - 21582718075
9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 8187
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 740
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 061 035 045 Old_age Always In_the_past 39 (Min/Max 39/42 #417)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 668
193 Load_Cycle_Count 0x0032 098 098 000 Old_age Always - 5797
194 Temperature_Celsius 0x0022 039 065 000 Old_age Always - 39 (0 17 0 0 0)
195 Hardware_ECC_Recovered 0x001a 114 099 000 Old_age Always - 64060608
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 2709 (36 106 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 14507335995
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 107851787370
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 6990 -
# 2 Extended offline Interrupted (host reset) 40% 6634 -
# 3 Short offline Completed without error 00% 6393 -
# 4 Short offline Completed without error 00% 6049 -
# 5 Short offline Completed without error 00% 5983 -
# 6 Short offline Completed without error 00% 5856 -
# 7 Short offline Completed without error 00% 5803 -
# 8 Short offline Completed without error 00% 5779 -
# 9 Short offline Completed without error 00% 5707 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Log for da3:
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): Error 5, Retries exhausted
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): CAM status: SCSI Status Error
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI status: Check Condition
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): Retrying command (per sense data)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): CAM status: SCSI Status Error
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI status: Check Condition
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): Retrying command (per sense data)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): CAM status: SCSI Status Error
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI status: Check Condition
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): Retrying command (per sense data)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): CAM status: SCSI Status Error
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI status: Check Condition
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): Retrying command (per sense data)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): READ(6). CDB: 08 00 08 00 01 00
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): CAM status: SCSI Status Error
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI status: Check Condition
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Nov 16 20:14:17 freenas (da3:mps0:0:1:0): Error 5, Retries exhausted
Nov 16 20:14:17 freenas GEOM_PART: da3 was automatically resized.
Nov 16 20:14:17 freenas Use `gpart commit da3` to save changes or `gpart undo da3` to revert them.
Nov 16 20:14:17 freenas GEOM_PART: integrity check failed (da3, GPT)
I hope you can help me get this running again.
Thank you very much!
PS:
I have shut the server down now, because I think keeping it running in this state until there's a solution is too critical.
Last edited:
