CAM status: Uncorrectable parity/CRC error

[Jost Wittmann]

Hello there!
I am encountering the constantly repeating error on a brand new hard drive:

Code:

Apr  8 15:54:28 Jonas (ada4:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 10 74 58 40 30 00 00 01 00 00
Apr  8 15:54:28 Jonas (ada4:ahcich0:0:0:0): CAM status: Uncorrectable parity/CRC error
Apr  8 15:54:28 Jonas (ada4:ahcich0:0:0:0): Retrying command

ada4 is a brandnew drive.

The system also sends me several emails:

Code:

The volume ext10tb state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

and

Code:

The boot volume state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
The volume ext10tb state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

My System Specs:
Freenas 11.1-U4
HP N54L Microserver ProLiant Home Server 8GB RAM
AMD Turion(tm) II Neo N54L Dual-Core Processor
Hard drives, in a mirror RAID configuration:
1) 2 Western Digital 4TB (for NAS)
2) 2 Western Digital 6TB (for NAS)

Since the data on this NAS got a bit messy during the past years (lot's of double data that shall get deleted), I purchased a 10TB hard drive (Ironwolf, 10TB) and backup all files prior sorting out duplicates. Better safe than sorry.
The Microserver has a fifth bay for 3,5" drives. Since I wanted to have the 10tb drive readable in windows through a usb-connector, I formatted it NTFS prior plugging it into the NAS. Lot's of complications later, I decided that readability in windows isn't so important after all, and I created a new datashare based on ZVOL. However, copying data doesn't seem to work properly because of the error above.

Code:

Jonas# Jonas#  smartctl -a /dev/ada4
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Seagate IronWolf
Device Model:	 ST10000VN0004-1ZD101
Serial Number:	ZA24X1HP
LU WWN Device Id: 5 000c50 0afc2c1c5
Firmware Version: SC60
User Capacity:	10,000,831,348,736 bytes [10.0 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	7200 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sun Apr  8 15:56:12 2018 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
										was completed without error.
										Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0) The previous self-test routine completed
										without error or no self-test has ever
										been run.
Total time to complete Offline
data collection:				(  575) seconds.
Offline data collection
capabilities:					(0x7b) SMART execute Offline immediate.
										Auto Offline data collection on/off support.
										Suspend Offline collection upon new
										command.
										Offline surface scan supported.
										Self-test supported.
										Conveyance Self-test supported.
										Selective Self-test supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x01) Error logging supported.
										General Purpose Logging supported.
Short self-test routine
recommended polling time:		(   1) minutes.
Extended self-test routine
recommended polling time:		( 866) minutes.
Conveyance self-test routine
recommended polling time:		(   2) minutes.
SCT capabilities:			  (0x50bd) SCT Status supported.
										SCT Error Recovery Control supported.
										SCT Feature Control supported.
										SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000f   084   065   044	Pre-fail  Always	   -	   240417432
  3 Spin_Up_Time			0x0003   088   088   000	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   11
  5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000f   061   060   045	Pre-fail  Always	   -	   1125775
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   21 (75 242 0)
 10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   9
184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
188 Command_Timeout		 0x0032   097   019   000	Old_age   Always	   -	   365077991266
189 High_Fly_Writes		 0x003a   100   100   000	Old_age   Always	   -	   0
190 Airflow_Temperature_Cel 0x0022   075   072   040	Old_age   Always	   -	   25 (Min/Max 21/25)
191 G-Sense_Error_Rate	  0x0032   100   100   000	Old_age   Always	   -	   32
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   1
193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   20
194 Temperature_Celsius	 0x0022   025   040   000	Old_age   Always	   -	   25 (0 19 0 0 0)
195 Hardware_ECC_Recovered  0x001a   023   003   000	Old_age   Always	   -	   240417432
197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x003e   200   199   000	Old_age   Always	   -	   4236
200 Multi_Zone_Error_Rate   0x0023   100   100   001	Pre-fail  Always	   -	   0
240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   6 (60 167 0)
241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   1242940452
242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   13306017

SMART Error Log Version: 1
No Errors Logged
-bash: Jonas#: command not found
Jonas# smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
-bash: syntax error near unexpected token `('
Jonas# Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
-bash: syntax error near unexpected token `C'
Jonas#
Jonas# === START OF INFORMATION SECTION ===
-bash: ===: command not found
Jonas# Model Family:	 Seagate IronWolf
-bash: Model: command not found
Jonas# Device Model:	 ST10000VN0004-1ZD101
-bash: Device: command not found
Jonas# Serial Number:	ZA24X1HP
-bash: Serial: command not found
Jonas# LU WWN Device Id: 5 000c50 0afc2c1c5
-bash: LU: command not found
Jonas# Firmware Version: SC60
-bash: Firmware: command not found
Jonas# User Capacity:	10,000,831,348,736 bytes [10.0 TB]
-bash: User: command not found
Jonas# Sector Sizes:	 512 bytes logical, 4096 bytes physical
-bash: Sector: command not found
Jonas# Rotation Rate:	7200 rpm
-bash: Rotation: command not found
Jonas# Form Factor:	  3.5 inches
-bash: Form: command not found
Jonas# Device is:		In smartctl database [for details use: -P show]
-bash: Device: command not found
Jonas# ATA Version is:   ACS-3 T13/2161-D revision 5
-bash: ATA: command not found
Jonas# SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
-bash: syntax error near unexpected token `('
Jonas# Local Time is:	Sun Apr  8 15:56:12 2018 CEST
-bash: Local: command not found
Jonas# SMART support is: Available - device has SMART capability.
-bash: SMART: command not found
Jonas# SMART support is: Enabled
-bash: SMART: command not found
Jonas#
Jonas# === START OF READ SMART DATA SECTION ===
-bash: ===: command not found
Jonas# SMART overall-health self-assessment test result: PASSED
-bash: SMART: command not found
Jonas#
Jonas# General SMART Values:
-bash: General: command not found
Jonas# Offline data collection status:  (0x82) Offline data collection activity
-bash: syntax error near unexpected token `('
Jonas#										 was completed without error.
-bash: was: command not found
Jonas#										 Auto Offline Data Collection: Enabled.
-bash: Auto: command not found
Jonas# Self-test execution status:	  (   0) The previous self-test routine completed
-bash: syntax error near unexpected token `('
Jonas#										 without error or no self-test has ever
-bash: without: command not found
Jonas#										 been run.
-bash: been: command not found
Jonas# Total time to complete Offline
-bash: Total: command not found
Jonas# data collection:				(  575) seconds.
-bash: syntax error near unexpected token `('
Jonas# Offline data collection
-bash: Offline: command not found
Jonas# capabilities:					(0x7b) SMART execute Offline immediate.
-bash: syntax error near unexpected token `0x7b'
Jonas#										 Auto Offline data collection on/off support.
-bash: Auto: command not found
Jonas#										 Suspend Offline collection upon new
-bash: Suspend: command not found
Jonas#										 command.
-bash: command.: command not found
Jonas#										 Offline surface scan supported.
-bash: Offline: command not found
Jonas#										 Self-test supported.
-bash: Self-test: command not found
Jonas#										 Conveyance Self-test supported.
-bash: Conveyance: command not found
Jonas#										 Selective Self-test supported.
-bash: Selective: command not found
Jonas# SMART capabilities:			(0x0003) Saves SMART data before entering
-bash: syntax error near unexpected token `('
Jonas#										 power-saving mode.
-bash: power-saving: command not found
Jonas#										 Supports SMART auto save timer.
-bash: Supports: command not found
Jonas# Error logging capability:		(0x01) Error logging supported.
-bash: syntax error near unexpected token `('
Jonas#										 General Purpose Logging supported.
-bash: General: command not found
Jonas# Short self-test routine
-bash: Short: command not found
Jonas# recommended polling time:		(   1) minutes.
-bash: syntax error near unexpected token `('
Jonas# Extended self-test routine
-bash: Extended: command not found
Jonas# recommended polling time:		( 866) minutes.
-bash: syntax error near unexpected token `('
Jonas# Conveyance self-test routine
-bash: Conveyance: command not found
Jonas# recommended polling time:		(   2) minutes.
-bash: syntax error near unexpected token `('
Jonas# SCT capabilities:			  (0x50bd) SCT Status supported.
-bash: syntax error near unexpected token `('
Jonas#										 SCT Error Recovery Control supported.
-bash: SCT: command not found
Jonas#										 SCT Feature Control supported.
-bash: SCT: command not found
Jonas#										 SCT Data Table supported.
-bash: SCT: command not found
Jonas#
Jonas# SMART Attributes Data Structure revision number: 10
-bash: SMART: command not found
Jonas# Vendor Specific SMART Attributes with Thresholds:
-bash: Vendor: command not found
Jonas# ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
-bash: ID#: command not found
Jonas#   1 Raw_Read_Error_Rate	 0x000f   084   065   044	Pre-fail  Always	   -	   240417432
-bash: 1: command not found
Jonas#   3 Spin_Up_Time			0x0003   088   088   000	Pre-fail  Always	   -	   0
-bash: 3: command not found
Jonas#   4 Start_Stop_Count		0x0032   100   100   020	Old_age   Always	   -	   11
-bash: 4: command not found
Jonas#   5 Reallocated_Sector_Ct   0x0033   100   100   010	Pre-fail  Always	   -	   0
-bash: 5: command not found
Jonas#   7 Seek_Error_Rate		 0x000f   061   060   045	Pre-fail  Always	   -	   1125775
-bash: 7: command not found
Jonas#   9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   21 (75 242 0)
-bash: syntax error near unexpected token `('
Jonas#  10 Spin_Retry_Count		0x0013   100   100   097	Pre-fail  Always	   -	   0
-bash: 10: command not found
Jonas#  12 Power_Cycle_Count	   0x0032   100   100   020	Old_age   Always	   -	   9
-bash: 12: command not found
Jonas# 184 End-to-End_Error		0x0032   100   100   099	Old_age   Always	   -	   0
-bash: 184: command not found
Jonas# 187 Reported_Uncorrect	  0x0032   100   100   000	Old_age   Always	   -	   0
-bash: 187: command not found
Jonas# 188 Command_Timeout		 0x0032   097   019   000	Old_age   Always	   -	   365077991266
-bash: 188: command not found
Jonas# 189 High_Fly_Writes		 0x003a   100   100   000	Old_age   Always	   -	   0
-bash: 189: command not found
Jonas# 190 Airflow_Temperature_Cel 0x0022   075   072   040	Old_age   Always	   -	   25 (Min/Max 21/25)
-bash: syntax error near unexpected token `('
Jonas# 191 G-Sense_Error_Rate	  0x0032   100   100   000	Old_age   Always	   -	   32
-bash: 191: command not found
Jonas# 192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   1
-bash: 192: command not found
Jonas# 193 Load_Cycle_Count		0x0032   100   100   000	Old_age   Always	   -	   20
-bash: 193: command not found
Jonas# 194 Temperature_Celsius	 0x0022   025   040   000	Old_age   Always	   -	   25 (0 19 0 0 0)
-bash: syntax error near unexpected token `('
Jonas# 195 Hardware_ECC_Recovered  0x001a   023   003   000	Old_age   Always	   -	   240417432
-bash: 195: command not found
Jonas# 197 Current_Pending_Sector  0x0012   100   100   000	Old_age   Always	   -	   0
-bash: 197: command not found
Jonas# 198 Offline_Uncorrectable   0x0010   100   100   000	Old_age   Offline	  -	   0
-bash: 198: command not found
Jonas# 199 UDMA_CRC_Error_Count	0x003e   200   199   000	Old_age   Always	   -	   4236
-bash: 199: command not found
Jonas# 200 Multi_Zone_Error_Rate   0x0023   100   100   001	Pre-fail  Always	   -	   0
-bash: 200: command not found
Jonas# 240 Head_Flying_Hours	   0x0000   100   253   000	Old_age   Offline	  -	   6 (60 167 0)
-bash: syntax error near unexpected token `('
Jonas# 241 Total_LBAs_Written	  0x0000   100   253   000	Old_age   Offline	  -	   1242940452
-bash: 241: command not found
Jonas# 242 Total_LBAs_Read		 0x0000   100   253   000	Old_age   Offline	  -	   13306017
-bash: 242: command not found
Jonas#
Jonas# SMART Error Log Version: 1
-bash: SMART: command not found
Jonas# No Errors Logged
-bash: No: command not found

Code:

Jonas# zpool status -v
  pool: FreeNAS
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
		still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
		the pool may no longer be accessible by software that does not support
		the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 07:59:05 with 0 errors on Sun Mar 25 08:59:50 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		FreeNAS										 ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/4f03c77f-ee7d-11e4-b017-001cc43234e7  ONLINE	   0	 0	 0
			gptid/4f75b35e-ee7d-11e4-b017-001cc43234e7  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/b4e29021-0ae9-11e6-b744-001cc43234e7  ONLINE	   0	 0	 0
			gptid/b5b4654e-0ae9-11e6-b744-001cc43234e7  ONLINE	   0	 0	 0

errors: No known data errors

  pool: ext10TB
 state: ONLINE
  scan: none requested
config:

		NAME										  STATE	 READ WRITE CKSUM
		ext10TB									   ONLINE	   0	 0	 0
		  gptid/22581c64-3b25-11e8-a138-28924a390995  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: resilvered 792K in 0 days 00:00:08 with 0 errors on Sun Apr  8 13:35:08 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  mirror-0  ONLINE	   0	 0	 0
			da0p2   ONLINE	   0	 0	 0
			da1p2   ONLINE	   0	 0	 0

errors: No known data errors
Jonas# Jonas# zpool status -v
  pool: ext10TB
 state: ONLINE
  scan: none requested
config:

		NAME										  STATE	 READ WRITE CKSUM
		ext10TB									   ONLINE	   0	 0	 0
		  gptid/22581c64-3b25-11e8-a138-28924a390995  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: resilvered 792K in 0 days 00:00:08 with 0 errors on Sun Apr  8 13:35:08 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  mirror-0  ONLINE	   0	 0	 0
			da0p2   ONLINE	   0	 0	 0
			da1p2   ONLINE	   0	 0	 0

errors: No known data errors
Jonas#

Setting up s SMART test, I get

Code:

Apr  8 16:11:48 Jonas smartd[46141]: Unable to register device /dev/da0 (no Directive -d removable). Exiting.

Chances me doing something wrong are much higher than the new drive been damaged, I am seeking your advice on how to find out what is wrong - and how to proceed. Your help is very much appreciated!

 

[Johnnie Black]

199 UDMA_CRC_Error_Count 0x003e 200 199 000 Old_age Always - 4236

This points to a connection problem, 9 times out of 10 a bad SATA cable.

 

[JohnDigital]

Its not clear to me if this 10tb drive is connected via USB or to a SATA port. There is alot to process here. In the past it has been suggested that lots of CAM errors are cable related, especially . That isnt necessarily my experience. In each case of CAM error, replacing the drive itself has "fixed" these errors. But to be noted, that in each case I replaced the cable too. But I have tried just replacing the cable before to no avail, so you be the judge.

Its my thought that FreeNAS reports these CAM errors where other OS hide them behind pretty colorful screens. You might temporarily switch the drive to a different port/cable on the HBA/motherboard and see if the CAM errors on da4 switches to da2 or whatever or does it still give error on da4? If the error switches, its the drive, if it doesnt then it would most likely be cable. My guess is the error will follow the hard drive. At which time it should be returned.

Apr 8 16:11:48 Jonas smartd[46141]: Unable to register device /dev/da0 (no Directive -d removable). Exiting.

This tells you that da0 is a USB external drive and SMART cant be run without a -d flag. Its most likely one of your boot drives and it may be failing. I see your boot pool was resilvered recently. Keep an eye on it.

Let me know a little more info, and I can help you further. Start by just switching the drive to a different port on the MB and see what happens. Does the CAM error switch to da2 or the like? Or does it stay as da4? Former: drive is bad, latter: cable is bad.

 

[Stux]

Does the CAM error switch to da2 or the like? Or does it stay as da4? Former: drive is bad, latter: cable is bad.

Identify the drives by serial though.

 

[Jost Wittmann]

The drive in question (10TB, da4) is connected by SATA, not USB.
I connected the drive to an idle SATA cable (so a different connector on the motherboard) and started up FreeNAS. I destroyed and recreated Volume and Dataset and started again to copy files about an hour ago. So far I did not receive any error messages.

The boot-drives are 2 8Gbyte USB sticks, in a mirror configuration.

Next steps:
1) Run SMART tests on all drives.
2) I should probably check whether another SATA cable connected to the original interface on the motherboard results in any problems.
Do you agree? Any other recommendations?

I'd love to provide you with more information - please let me know what commands to run. Though I've been using FreeNAS for 3-4 years now, I still feel that I have little clue of what I'm doing....

Thank you so much for your input!

 

[JohnDigital]

I think you are doing enough for now. Keep an eye on it.

 

[gimmefewil]

This points to a connection problem, 9 times out of 10 a bad SATA cable.

Yup, this saved me.

 

[James Doyle]

This points to a connection problem, 9 times out of 10 a bad SATA cable.

Three Years Later this advice is SPOT ON! I was about to buy a replacement SSD for my mirrored boot drive. After really hitting it with dd, and smartctl tests I was convinced it was bad blocks. No. Reseating the SATA cable fixed it. Vibrations and Thermal Cycling will eventually cause connections to loosen even in a stable system like mine.