cam status ATA status error

Aschtra · May 13, 2018

Hello,

I've been experiencing some problems with my Freenas server. When I run dmesg from the shell then I see alot of cam status ATA status errors which I don't know what they mean. Here is a screenshot of it when it occured again. It does not always occur:

I have the following hardware:

HP Microserver Gen8
16GB ECC RAM
4x4TB WD RED NAS drives (which I bought new, 2,5 weeks ago and replaces all my previous drives which were 4x2TB drives)
Freenas11.1-U4

The error message says something about ADA1, but all hard drives are new so I don't expect something to be wrong with it.
I also got this error sometimes before I replaced my 4 drives. I searched this error on google but everyones situation is different so that's why I made a new topic for this.
I read about replacing sata cables, but the Gen8 server does not use normal sata cables. But this one:

Zpool status -v output:

Anyone has a idea what I can do to fix this?

Thank you!

kdragon75 · May 13, 2018

I would start with the cable.Try reseating all ends. If that doesn't do it or it better for a while/gets worse, get a quality cable and not the cheapest on you can find. Also did you test your new drives or do any kind of burn in?

Aschtra · May 13, 2018

kdragon75 said:
I would start with the cable.Try reseating all ends. If that doesn't do it or it better for a while/gets worse, get a quality cable and not the cheapest on you can find. Also did you test your new drives or do any kind of burn in?

I will reseat them first. I did not burn my drives in. Don't know how.
I just did a short smartctl test and I think these are the results?:

Code:

root@freenas:~ # smartctl -a /dev/ada1
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD40EFRX-68N32N0
Serial Number:	WD-WCC7K6CHHZK3
LU WWN Device Id: 5 0014ee 2651b8011
Firmware Version: 82.00A82
User Capacity:	4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5400 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sun May 13 20:18:17 2018 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
										was never started.
										Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0) The previous self-test routine completed
										without error or no self-test has ever
										been run.
Total time to complete Offline
data collection:				(44700) seconds.
Offline data collection
capabilities:					(0x7b) SMART execute Offline immediate.
										Auto Offline data collection on/off support.
										Suspend Offline collection upon new
										command.
										Offline surface scan supported.
										Self-test supported.
										Conveyance Self-test supported.
										Selective Self-test supported.
SMART capabilities:			(0x0003) Saves SMART data before entering
										power-saving mode.
										Supports SMART auto save timer.
Error logging capability:		(0x01) Error logging supported.
										General Purpose Logging supported.
Short self-test routine
recommended polling time:		(   2) minutes.
Extended self-test routine
recommended polling time:		( 474) minutes.
Conveyance self-test routine
recommended polling time:		(   5) minutes.
SCT capabilities:			  (0x303d) SCT Status supported.
										SCT Error Recovery Control supported.
										SCT Feature Control supported.
										SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   113   113   051	Pre-fail  Always	   -	   3418
  3 Spin_Up_Time			0x0027   100   253   021	Pre-fail  Always	   -	   0
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   1
  5 Reallocated_Sector_Ct   0x0033   199   199   140	Pre-fail  Always	   -	   30
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   412
 10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
 11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   1
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   0
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   204
194 Temperature_Celsius	 0x0022   121   116   000	Old_age   Always	   -	   29
196 Reallocated_Event_Count 0x0032   192   192   000	Old_age   Always	   -	   8
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   100   253   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
ATA Error Count: 5
		CR = Command Register [HEX]
		FR = Features Register [HEX]
		SC = Sector Count Register [HEX]
		SN = Sector Number Register [HEX]
		CL = Cylinder Low Register [HEX]
		CH = Cylinder High Register [HEX]
		DH = Device/Head Register [HEX]
		DC = Device Command Register [HEX]
		ER = Error register [HEX]
		ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 5 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  10 51 00 00 1b 00 40  Error: IDNF at LBA = 0x00001b00 = 6912

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ca 00 00 00 1b 00 40 08	  00:06:29.023  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:21.976  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:14.929  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:07.882  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:00.833  WRITE DMA

Error 4 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  10 51 00 00 1b 00 40  Error: IDNF at LBA = 0x00001b00 = 6912

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ca 00 00 00 1b 00 40 08	  00:06:21.976  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:14.929  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:07.882  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:00.833  WRITE DMA

Error 3 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  10 51 00 00 1b 00 40  Error: IDNF at LBA = 0x00001b00 = 6912

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ca 00 00 00 1b 00 40 08	  00:06:14.929  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:07.882  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:00.833  WRITE DMA

Error 2 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  10 51 00 00 1b 00 40  Error: IDNF at LBA = 0x00001b00 = 6912

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ca 00 00 00 1b 00 40 08	  00:06:07.882  WRITE DMA
  ca 00 00 00 1b 00 40 08	  00:06:00.833  WRITE DMA

Error 1 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  10 51 00 00 1b 00 40  Error: IDNF at LBA = 0x00001b00 = 6912

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ca 00 00 00 1b 00 40 08	  00:06:00.833  WRITE DMA

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	   411		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@freenas:~ #

I am afraid I can't really reseat the sata cables. They look like this (yes I put the screws back in again)

kdragon75 · May 13, 2018

Well that's one way to build a backplane... I guess just reseating the drives in the slots and reseating the motherboard end.

Aschtra · May 13, 2018

Done that. I'll wait.
My server is running the following:

- Plex
- Radarr
- Sonarr
- Sabnzbd

4x4TB in a raidz1.

Anything else I can do for now to provide more information maybe?

edit:

It happend again, right after my girlfriend tried to watch a show on Plex. Will try something with that file, delete it or something. Play another show, see if that one works

kdragon75 · May 13, 2018

Do the errors always report ada1? If so try swapping the drives around. FreeNAS won't care and that can help narrow it down to a port or drive.

Aschtra · May 13, 2018

Yes always on ada1. I've replaced the episode she was watching with a different one and now it doesn't occur anymore when she watches it. Weird issue, it crashed my whole freenas.. :p

Will keep an eye on it though.

kdragon75 · May 13, 2018

Yeah a file won't cause that kind of an error but a bad disk will. Keep a close eye on it.

Aschtra · May 13, 2018

The drives are brand new so I don’t suspect the HDD right now. Will switch them up though just to see what happens.

Also will do more scrubs. Had them on each 35 days but will do 7 days now

Jailer · May 13, 2018

Increasing the scrub frequency won't help with your problem. You need to also schedule regular long and short smart tests for your drives.

kdragon75 · May 13, 2018

Aschtra said:
The drives are brand new

Exactly. A brand spanking new drive is more likely to fail than a drive that's been running for 6 months.

As @Jailer mentioned, smart tests long and short need to be scheduled to help monitor and test your drives.

Johnnie Black · May 14, 2018

Aschtra said:
The drives are brand new so I don’t suspect the HDD right now.

It may be new but this is not a good sign:

Aschtra said:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 113 113 051 Pre-fail Always - 3418

You want this to be zero, or single digits at most, anymore it's usually a sign of trouble, and most likely the disk will fail soon.

kdragon75 · May 14, 2018

Yeah its normal to have some raw read errors and have it count up over the life of the drive but if its shooting up... Better get a new drive.

Do some searching on the forum about drive testing and burn-in. It's important for all drives but especially new drives.

DrKK · May 14, 2018

My eye went immediately to attribute #196 not being zero on a brand-new drive. In fact, there have been 8 reallocations. That's "super bad" for a new drive.

Chris Moore · May 14, 2018

I have a server at work that has 60 hard drives in it and in the first 6 months four of them failed. One of them catastrophically.
Just because a drive is new, that doesn't mean it can't fail.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

Aschtra · May 16, 2018

Hello,
I've come to apologize. It is indeed the disk that appears to be failing. I will contact my store to replace it!

Thank you for your time!

Important Announcement for the TrueNAS Community.

cam status ATA status error

Aschtra

Dabbler

kdragon75

Wizard

Aschtra

Dabbler

kdragon75

Wizard

Aschtra

Dabbler

kdragon75

Wizard

Aschtra

Dabbler

kdragon75

Wizard

Aschtra

Dabbler

Jailer

Not strong, but bad

kdragon75

Wizard

Johnnie Black

Guru

kdragon75

Wizard

DrKK

FreeNAS Generalissimo

Chris Moore

Hall of Famer

Aschtra

Dabbler

Similar threads

Important Announcement for the TrueNAS Community.

cam status ATA status error

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Not strong, but bad

Wizard

Guru

Wizard

FreeNAS Generalissimo

Hall of Famer

Dabbler

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "cam status ATA status error"

Similar threads