Error In Security Run Output - Help with interpretation

[Chris Hoeschen]

Today I got an e-mail from my FreeNAS box with a subject line of {Hostname} Security Run Report. The body of the e-mail has the following several times:

> (ada1:ahcich1:0:0:0): CAM status: ATA Status Error
> (ada1:ahcich1:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
> (ada1:ahcich1:0:0:0): RES: 41 40 20 5a c2 40 0f 00 00 00 00
> (ada1:ahcich1:0:0:0): Retrying command
> (ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 c8 98 59 c2 40 0f 00 00 00 00 00​

I have a cronjob setup to e-mail me a status report (SMART status and zpool status) and here is the contents of today's status report:

Drive Reports for 6 disks:

Disk 1
===================================
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Seagate Desktop HDD.15
Device Model: ST4000DM000-1F2168
Serial Number: S300E1X3
LU WWN Device Id: 5 000c50 06da247aa
Firmware Version: CC54
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Sep 29 08:00:00 2014 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Power mode is: ACTIVE or IDLE

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 127796952
3 Spin_Up_Time 0x0003 091 091 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 461
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 070 060 030 Pre-fail Always - 12204480
9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 2595
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 22
183 Runtime_Bad_Block 0x0032 099 099 000 Old_age Always - 1
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 069 051 045 Old_age Always - 31 (Min/Max 29/35)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1586
194 Temperature_Celsius 0x0022 031 049 000 Old_age Always - 31 (0 22 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1059h+52m+19.470s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 11802093261
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 74580590782

SMART Error Log Version: 1
No Errors Logged

===================================

Disk 2
===================================
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4ER84E7YN
LU WWN Device Id: 5 0014ee 2b5472906
Firmware Version: 80.00A80
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Sep 29 08:00:00 2014 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Power mode is: ACTIVE or IDLE

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 422
3 Spin_Up_Time 0x0027 180 179 021 Pre-fail Always - 8000
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 16
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 926
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 16
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 24
194 Temperature_Celsius 0x0022 121 113 000 Old_age Always - 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 16
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 29

SMART Error Log Version: 1
No Errors Logged

===================================

Disk 3
===================================
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4EF9FA9HV
LU WWN Device Id: 5 0014ee 20a980e64
Firmware Version: 80.00A80
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Sep 29 08:00:00 2014 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Power mode is: ACTIVE or IDLE

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 181 181 021 Pre-fail Always - 7908
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 16
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 926
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 16
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 24
194 Temperature_Celsius 0x0022 122 114 000 Old_age Always - 30
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

===================================

Disk 4
===================================
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4ER84E8N1
LU WWN Device Id: 5 0014ee 2b5475b0b
Firmware Version: 80.00A80
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Sep 29 08:00:00 2014 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Power mode is: ACTIVE or IDLE

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 176 176 021 Pre-fail Always - 8183
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 16
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 926
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 16
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 22
194 Temperature_Celsius 0x0022 120 114 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

===================================

Disk 5
===================================
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E9M06VU1
LU WWN Device Id: 5 0014ee 2b5291fcc
Firmware Version: 80.00A80
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Sep 29 08:00:01 2014 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Power mode is: ACTIVE or IDLE

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 174 174 021 Pre-fail Always - 8266
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 19
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 926
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 19
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 6
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 22
194 Temperature_Celsius 0x0022 123 114 000 Old_age Always - 29
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

===================================

Disk 6
===================================
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p10 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4EP5524JL
LU WWN Device Id: 5 0014ee 20a9c4679
Firmware Version: 80.00A80
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Sep 29 08:00:01 2014 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Power mode is: ACTIVE or IDLE

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 176 176 021 Pre-fail Always - 8200
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 16
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 926
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 16
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 24
194 Temperature_Celsius 0x0022 122 114 000 Old_age Always - 30
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

===================================


ZPool Status:
pool: DataVolume
state: ONLINE
scan: scrub repaired 3.16M in 8h48m with 0 errors on Sun Sep 28 08:48:46 2014
config:

NAME STATE READ WRITE CKSUM
DataVolume ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/22255324-296a-11e4-8cce-0cc47a0b78de ONLINE 0 0 0
gptid/23116ca1-296a-11e4-8cce-0cc47a0b78de ONLINE 0 0 0
gptid/23e1e86b-296a-11e4-8cce-0cc47a0b78de ONLINE 0 0 0
gptid/24ab8ec2-296a-11e4-8cce-0cc47a0b78de ONLINE 0 0 0
gptid/257c9dfa-296a-11e4-8cce-0cc47a0b78de ONLINE 0 0 0
gptid/2651c201-296a-11e4-8cce-0cc47a0b78de ONLINE 0 0 0

errors: No known data errors

NAME PROPERTY VALUE SOURCE
DataVolume size 21.8T -
DataVolume capacity 46% -
DataVolume altroot /mnt local
DataVolume health ONLINE -
DataVolume guid 3955683371965583158 default
DataVolume version - default
DataVolume bootfs - default
DataVolume delegation on default
DataVolume autoreplace off default
DataVolume cachefile /data/zfs/zpool.cache local
DataVolume failmode continue local
DataVolume listsnapshots off default
DataVolume autoexpand on local
DataVolume dedupditto 0 default
DataVolume dedupratio 1.00x -
DataVolume free 11.6T -
DataVolume allocated 10.2T -
DataVolume readonly off -
DataVolume comment - default
DataVolume expandsize 0 -
DataVolume freeing 0 default
DataVolume feature@async_destroy enabled local
DataVolume feature@empty_bpobj active local
DataVolume feature@lz4_compress active local
DataVolume feature@multi_vdev_crash_dump enabled local
DataVolume feature@spacemap_histogram active local
DataVolume feature@enabled_txg active local
DataVolume feature@hole_birth active local
DataVolume feature@extensible_dataset enabled local
DataVolume feature@bookmarks enabled local
​

My first drive I have a large number for RAW READ ERROR RATE and SEEK ERROR RATE but as I read this is common for Seagate drives and I should instead look at REALLOCATED SECTOR COUNT which is 0 for this drive. I got this tip from this post from another site:

The "seek error rate" and "raw read error rate" RAW_VALUES are virtually meaningless for anyone but Seagate's support. As others pointed out, raw values of parameters like "reallocated sector count" or entries in the drive's error log are more likely to indicate a higher probability of failure.

But you can take a look at the interpreted data in the VALUE, WORST and THRESH columns which are meant to be read as gauges:​

On my second drive I have 422 under RAW READ ERROR RATE but I also see I have 16 under CURRENT PENDING SECTOR where all of my other WD drives are 0 for both of these. Does this indicate I have a bad drive? All of my WD drives are new (less than 2 months old) so if it is faulty or potentially faulty I can try and RMA it. However, as it shows under my zpool status a scrub just got done yesterday and repaired some data. Could the errors in the security report be due to the scrub and data correction and not due to a drive issue?

 

[Ericloewe]

Yeah, Disk 2 needs replacement. The data repaired by the scrub is almost certainly due to the drive.

The drive itself detected that it failed several times. ZFS then compensated for the failures by calculating the appropriate values and writing them to disk again.
Run a Long Test (which will likely fail, which is good for the RMA process to go more smoothly).

 

[Chris Hoeschen]

Update: I sent in the logs and SMART results I received from my FreeNAS box to WD support and they RMAed the drive immediately. I replaced the drive and resilvered. So far I have 0 for both RAW READ ERROR RATE and CURRENT PENDING SECTOR on this drive as well as all of my other WD drives.