Noob needs help with Drive Degradation.

Xa3phod

Dabbler
Joined
Mar 8, 2023
Messages
10
Hi, I am completely new to TrueNas and just build my first.
I installed 2 new WD red (4TB) drives to mirror. This is for home use. I am getting an error message about drive degradation.

"

Pool BigServer state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected.​

The following devices are not healthy:​

  • Disk WDC WD40EFZX-68AWUN0 WD-WXV2AB21MK7A is FAULTED

2023-03-29 09:57:59 (America/New_York)"


I am not sure what to do. Do I just swap that drive with another one and it will rebuild the mirror automatically? Can I continue to use my nas for now? Can I remove the bad drive and run it with 1 drive or do I have to wait for the new drive to arrive and then install it? Is there a procedure for rebuilding the mirror? should I backup my mirror to another drive elsewhere just in case?

Thanks


Here is the result of
zpool status -v
zpool list -v
find /dev/ada* -exec smartctl -a {} \;

ast login: Fri Mar 10 11:17:55 on pts/1
FreeBSD 13.1-RELEASE-p7 n245418-79e75956dbb TRUENAS

TrueNAS (c) 2009-2023, iXsystems, Inc.
All rights reserved.
TrueNAS code is released under the modified BSD license with some
files copyrighted by (c) iXsystems, Inc.

For more information, documentation, help or support, go here:
http://truenas.com
Welcome to TrueNAS

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]#
root@truenas[~]# zpool status -v
pool: BigServer
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: resilvered 49.5M in 00:00:04 with 0 errors on Fri Mar 24 15:58:20 2023
config:

NAME STATE READ WRITE CKSUM
BigServer DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ada0p2 FAULTED 121 0 0 too many errors
gptid/57cccaf4-bf5c-11ed-a856-e0d55ea1ef75 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

BigServer/iocage/jails/Nextcloud/root:<0x0>

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:04 with 0 errors on Thu Mar 30 03:45:04 2023
config:
boot-pool
state: ONLINE

NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
BigServer 3.62T 368G 3.27T - - 0% 9% 1.00x DEGRADED /mnt
mirror-0 3.62T 368G 3.27T - - 0% 9.90% - DEGRADED
ada0p2 3.64T - - - - - - - FAULTED


Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# find /dev/ada* -exec smartctl -a {} \;
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFZX-68AWUN0
Serial Number: WD-WXV2AB21MK7A
LU WWN Device Id: 5 0014ee 2c03a580a
Firmware Version: 81.00B81
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Mar 30 13:21:02 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 39) The self-test routine was interrupted
by the host with a hard or soft reset.
Total time to complete Offline
data collection: (41640) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 443) minutes.
SCT capabilities: (0x303d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 226 226 021 Pre-fail Always - 3666
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 19
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 185
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 18
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 1
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 29
194 Temperature_Celsius 0x0022 120 114 000 Old_age Always - 30
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 188 145 000 Old_age Always - 3834
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 70% 173 -
# 2 Extended offline Completed without error 00% 101 -
# 3 Extended offline Interrupted (host reset) 90% 64 -

Selective Self-tests/Logging not supported

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/ada0p1: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/ada0p2: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFZX-68AWUN0
Serial Number: WD-WXV2AB2AY86K
LU WWN Device Id: 5 0014ee 2c03a5630
Firmware Version: 81.00B81
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Mar 30 13:21:02 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (44520) seconds.
Offline data collection
capabilities: (0x11) SMART execute Offline immediate.
No Auto Offline data collection support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 472) minutes.
SCT capabilities: (0x303d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 228 227 021 Pre-fail Always - 3600
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 19
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 185
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 18
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 1
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 17
194 Temperature_Celsius 0x0022 118 115 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 72 -

Selective Self-tests/Logging not supported

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/ada1p1: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/ada1p2: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: SPCC Solid State Disk
Serial Number: GMAJ220434332
LU WWN Device Id: 5 000000 000005be7
Firmware Version: H220115
User Capacity: 256,060,514,304 bytes [256 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Mar 30 13:21:03 2023 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 33) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 2) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x0031) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 20
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0013 100 100 050 Pre-fail Always - 0
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0012 100 100 000 Old_age Always - 60
167 Unknown_Attribute 0x0022 100 100 000 Old_age Always - 0
168 Unknown_Attribute 0x0012 100 100 000 Old_age Always - 0
169 Unknown_Attribute 0x0013 100 100 010 Pre-fail Always - 0
171 Unknown_Attribute 0x0032 000 000 000 Old_age Always - 0
172 Unknown_Attribute 0x0032 000 000 000 Old_age Always - 0
173 Unknown_Attribute 0x0012 200 200 000 Old_age Always - 4295032833
175 Program_Fail_Count_Chip 0x0022 100 100 010 Old_age Always - 0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033 100 100 000 Pre-fail Always - 25
187 Reported_Uncorrect 0x0032 100 000 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0012 100 100 000 Old_age Always - 32
194 Temperature_Celsius 0x0022 040 040 000 Old_age Always - 40 (Min/Max 40/40)
206 Unknown_SSD_Attribute 0x0032 200 200 000 Old_age Always - 1
207 Unknown_SSD_Attribute 0x0032 200 200 000 Old_age Always - 1
208 Unknown_SSD_Attribute 0x0032 200 200 000 Old_age Always - 1
209 Unknown_SSD_Attribute 0x0032 200 200 000 Old_age Always - 1
210 Unknown_Attribute 0x0032 200 200 000 Old_age Always - 32
211 Unknown_Attribute 0x0032 200 200 000 Old_age Always - 3
231 Unknown_SSD_Attribute 0x0023 100 100 005 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0
234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 6
241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 1
242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 4
245 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 0 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/ada2p1: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/ada2p2: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/ada2p3: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.
 
Last edited:
Joined
Jan 7, 2015
Messages
1,155
This disk is not very old correct? 181 hours, that's barely a week. Also an extended SMART test has not finished on the degraded (WD-WXV2AB21MK7A) disk. I see lots of CRC errors which could point to wonky cabling in most cases. Id like to see your dmesg logs what was happening before this disk was booted from the mirror? Are these disks connected to the MB? Did you use new cables? Do you have any power splitters or the like?
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
I installed 2 new WD red (4TB) drives to mirror.
[..]

  • Disk WDC WD40EFZX-68AWUN0 WD-WXV2AB21MK7A is FAULTED

Are you sure this is a WD Red? The model WD40EFZX seems to be a WD Red Plus, i.e. a proper CMR drive.
 

Glorious1

Guru
Joined
Nov 23, 2014
Messages
1,211
My first response would be shut down, reseat the offending drive and both ends of its cable (or replace them if possible), then reboot.
 

Xa3phod

Dabbler
Joined
Mar 8, 2023
Messages
10
I want to thank everyone for the help.
First: It is a Red Plus drive.
Second: The cables are older than dirt I had sitting around. I didn't realize that cables went bad. I will reseat them and if that doesn't work I will order new ones.
Third: The drives are plugged directly into the Motherboard.
I am using a B350 Mobo with a 2200G CPU, because I thought not needing a GPU would be beneficial and it has a low wattage.

I ordered another drive thinking it was actually bad. If it's just the cables, I will return the drive I ordered. There isn't any benefit to add a 3rd drive? Can you if you are mirroring?


Thanks again
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
You can add a 3rd drive to a mirror - you don't get any more space, but you do get extra resiliance
 

Xa3phod

Dabbler
Joined
Mar 8, 2023
Messages
10
It was the SATA cord. I replaced it with a new one and all the errors went away. Thanks a ton guys! Not sure if I should incorporate the 3rd drive or return it. I can't imagine I need more than one extra drive for drive failure. Should I just return it or switch to another zraid?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Burn the extra drive in; run badblocks on it; and keep it as a spare on a shelf if it passes.
 

Glorious1

Guru
Joined
Nov 23, 2014
Messages
1,211
Yeah, gold stars (likes) to the people who advised you to check/replace cables.

You're actively using two drives. The smart move would be to keep the extra one as a spare.
 
Top