FAULTED drive and failed to read SMART

[Visseroth]

So I've been having this problem. I thought it was related to the SMART issues that have been posted about but I just applied U6, turned smart on and received a email stating...

Code:

The volume Data state is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state.
Device: /dev/da8 [SAT], failed to read SMART Attribute Data
Device: /dev/da7 [SAT], failed to read SMART Attribute Data

I've also been having this issue with da7 (seemingly though possibly not related to SMART) da7 gets read and write errors as seen in the attached screen shot.
I pulled that drive and dd /dev/zero of=/dev/(mnt-point) the drive, ran a extended SMART test and even fdisk -lu /dev/(mnt-point) the disk which supposed forces the drive to mark any bad sectors but according to SMART the drive is just fine. Maybe a bad socket on the backplane or something? I'm not sure.

Here's the SMART from da7

Code:

smartctl -a /dev/da7
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 HGST Deskstar NAS
Device Model:	 HGST HDN724040ALE640
Serial Number:	---------------LD2S
LU WWN Device Id: 5 000cca 24ce845b0
Firmware Version: MJAOA5E0
User Capacity:	4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	7200 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Wed Aug 22 19:26:33 2018 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)   Offline data collection activity
				   was completed without error.
				   Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0)   The previous self-test routine completed
				   without error or no self-test has ever
				   been run.
Total time to complete Offline
data collection:		(   24) seconds.
Offline data collection
capabilities:			 (0x5b) SMART execute Offline immediate.
				   Auto Offline data collection on/off support.
				   Suspend Offline collection upon new
				   command.
				   Offline surface scan supported.
				   Self-test supported.
				   No Conveyance Self-test supported.
				   Selective Self-test supported.
SMART capabilities:			(0x0003)   Saves SMART data before entering
				   power-saving mode.
				   Supports SMART auto save timer.
Error logging capability:		(0x01)   Error logging supported.
				   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 ( 594) minutes.
SCT capabilities:		   (0x003d)   SCT Status supported.
				   SCT Error Recovery Control supported.
				   SCT Feature Control supported.
				   SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000b   100   100   016	Pre-fail  Always	   -	   0
  2 Throughput_Performance  0x0005   136   136   054	Pre-fail  Offline	  -	   82
  3 Spin_Up_Time			0x0007   180   180   024	Pre-fail  Always	   -	   414 (Average 447)
  4 Start_Stop_Count		0x0012   100   100   000	Old_age   Always	   -	   239
  5 Reallocated_Sector_Ct   0x0033   100   100   005	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000b   100   100   067	Pre-fail  Always	   -	   0
  8 Seek_Time_Performance   0x0005   124   124   020	Pre-fail  Offline	  -	   33
  9 Power_On_Hours		  0x0012   097   097   000	Old_age   Always	   -	   23282
 10 Spin_Retry_Count		0x0013   100   100   060	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   227
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   1107
193 Load_Cycle_Count		0x0012   100   100   000	Old_age   Always	   -	   1107
194 Temperature_Celsius	 0x0002   166   166   000	Old_age   Always	   -	   36 (Min/Max 18/49)
196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0022   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0008   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x000a   200   200   000	Old_age   Always	   -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	 23270		 -
# 2  Extended offline	Completed without error	   00%	 23266		 -
# 3  Short offline	   Completed without error	   00%	 20368		 -
# 4  Extended offline	Completed without error	   00%	 20227		 -
# 5  Short offline	   Completed without error	   00%	 20193		 -
# 6  Short offline	   Completed without error	   00%	 20041		 -
# 7  Short offline	   Completed without error	   00%	 19887		 -
# 8  Extended offline	Completed without error	   00%	 19874		 -
# 9  Short offline	   Completed without error	   00%	 19815		 -
#10  Extended offline	Completed without error	   00%	 19521		 -
#11  Short offline	   Completed without error	   00%	 19486		 -
#12  Short offline	   Completed without error	   00%	 19323		 -
#13  Short offline	   Completed without error	   00%	 19151		 -
#14  Short offline	   Completed without error	   00%	 18971		 -
#15  Short offline	   Completed without error	   00%	 18684		 -
#16  Short offline	   Completed without error	   00%	 18516		 -
#17  Extended offline	Completed without error	   00%	 18503		 -
#18  Short offline	   Completed without error	   00%	 18444		 -
#19  Short offline	   Completed without error	   00%	 18279		 -
#20  Extended offline	Completed without error	   00%	 18147		 -
#21  Short offline	   Completed without error	   00%	 18111		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

What do you guys think?

 

[joeschmuck]

I don't see anything wrong with the hard drive data you provided. In an effort to troubleshoot this failure I would recommend the following step. Please read these over and understand that there is minimal risk but there is always some risk when you go moving parts around.

1) Identify the da7 drive by serial number (PK1381PCJVLD2S) and then shutdown & power off the system.
2) Swap the hard drive (PK1381PCJVLD2S) electrical connections (or move within the enclosure) with another drive other than da8.
3) Power on and bootstrap the system.
4) Note which drive identifier serial number PK1381PCJVLD2S is assigned to (da2 ?) and monitor your system. You are looking to see if the problem follows the hard drive or the electrical connection.
5) If the problem follows the hard drive then you need to see if that is acceptable to you because the SMART tests are passing and there are no other alarm indications. If the problem follows the electrical connection then you may have a hardware failure and will need to troubleshoot it further.

 

[Visseroth]

You make a valid point.
As soon as I have a chance I'll do just that and report back my results in case others find themselves in the same situation.
I find it odd that I can run smartctl -a against da7 and da8 and see no errors.
Fortunately if I find the backplane faulty I can move the drives to the back, pull the backplane and send it to SuperMicro for repairs and return the drives to the front when the backplane is returned.

 

[joeschmuck]

I find it odd that I can run smartctl -a against da7 and da8 and see no errors.

The error message doesn't say that you have SMART errors, it says that there was a problem reading a SMART attribute. So was this a smartd issue? maybe.

 

[Jailer]

What are you using for an HBA? Also is it flashed to IT mode?

 

[Visseroth]

I'm not sure what the issue is with SMART or why it's having trouble reading the SMART attributes.
The irony is that I get errors about reading SMART attributes on da7 and 8. da7 has given me read/write errors as attached in my first post but upon pulling the drive and testing it, it seemed fine.

Jailer, you'll have to forgive me, I forgot what HBA means and yes, my controller is in IT mode. I do know better than to run a raid in a raid :P

 

[joeschmuck]

The problem is likely not your hard drives. Do the testing I recommended to try to isolate your problem more.

 

[Visseroth]

Agreed, but testing will be necessary.
I'll report back as I know more.

 

[joeschmuck]

Agreed, but testing will be necessary.
I'll report back as I know more.

Good luck and I hope the problem turns out to be something cheap to correct.

 

[Visseroth]

Haven't had a chance to do any testing as of yet but SMART has been kicking out random drives so I've had to turn SMART off because of it....

Code:

CRITICAL: 		 	 	 	Aug. 22, 2018, 6:04 p.m. - Device: /dev/da8 [SAT], failed to read SMART Attribute Data
CRITICAL: 		 	 	 	Aug. 22, 2018, 6:35 p.m. - Device: /dev/da7 [SAT], failed to read SMART Attribute Data
CRITICAL: 		 	 	 	Aug. 26, 2018, 8:04 p.m. - Device: /dev/da3 [SAT], failed to read SMART Attribute Data
CRITICAL: 		 	 	 	Aug. 30, 2018, 12:15 a.m. - Device: /dev/da2 [SAT], failed to read SMART Attribute Data
CRITICAL: 		 	 	 	Aug. 30, 2018, 12:21 a.m. - Device: /dev/da10 [SAT], failed to read SMART Attribute Data

 

[joeschmuck]

but SMART has been kicking out random drives

What do you mean by this? Are you just indicating that you are recieving a lot of reports and it's anoying? Or are you saying that you are having drives kicked offline? Your words make a difference.

You can run smartctl tests via script and report those results but if you disable SMART, you may miss an important error message. Try to troubleshoot your problem, don't just mask it or you will have a problem that causes you a lot of problems such as data loss.

 

[Visseroth]

every now and then, say every 2 to 5 days a drive will get kicked out because SMART failed to read the attribute data. One of these drives (da7) it happened a couple times, so as listed above I pulled the drive, I "dd of=/dev/zero if=/dev/da7" the drive (zero'd it out), during that time ran a extended SMART test. Everything came back good, no errors, nothing bad reported, ect.
I put the drive back in, rebuilt the array, then da8 dropped for the same reason.
Then I thought I'd wait tell I had a chance to shut the server down (which I haven't had yet) and move the drives to my other backplane to see if the problem persisted. Since then, as listed above, da2, da3, and da10 have been kicked out. Each time I mark the drive offline and re-add (replace) it to the pool.
Now it's still possible that it's a backplane issue but I no longer think it's port specific.
Ironically with SMART disabled I don't have problems with drives getting kicked out.

And I agree, I could miss a important error, but if a drive starts failing I'll also receive CAM errors in my periodic emails. So I'm not worried about a drive suddenly going belly up, there will be warning signs though I do still prefer to have SMART on but not if it's going to kick drives out when there is nothing wrong with them there by risking destroying my pool due to a issue with SMART, because it COULD decide one day that 3, 4 or more of my drives are faulty because it couldn't pull the attribute data and kick more out than I have available in redundancy.

In short I'm saying I am having drives kicked offline (faulted) so I've disabled SMART.

 

[Jailer]

I forgot what HBA means and yes, my controller is in IT mode.

Host Bus Adapter. If you don't know what it is how do you know it's in IT mode?

And you still haven't answered the question. How EXACTLY are you connecting the drives to the motherboard? HBA? On board SATA ports?There's an obvious communication error here so more information is needed.

 

[Visseroth]

Because it was flashed for me before I received it, knowing I was using it for FreeNAS and because I double checked it and it's firmware version while I had it on the bench and I forgot what HBA stood for.

If you looked at my signature you'd see I'm using the BPN-SAS2-846EL1 and BPN-SAS2-826EL1 Backplanes. They are both connected directly to the motherboard via a SFF-8087 to SFF-8087 cable. They are not daisy chained.
And yes I know that the x10SRH-CF utilizing the Broadcom 3008 controller is a SAS3 board while the backplanes are SAS2. SAS2 backplanes were what was in my budget range. I couldn't justify spending another $500 to $800 for SAS3 backplanes when I likely wouldn't even use the available speed.

Is there any other information needed?

 

[Ericloewe]

Signatures do not show up for mobile users, so please always explicitly include any relevant hardware.

 

[Visseroth]

Ahh, did not know that. Thanks for that, I'll be sure to include that in the future.

 

[Darklion]

I am having the same issue with drives and I have being buying more, I change the drives and then I got the same error on another one and change it again. So now I am starting to wonder that the drive are fine and is something else. I haven't done an smartctl on the drives to check if they where fine because swaping them will be easier but the problem persist.

this trent started on august 22 and the last message was on august 31 of 2018. There is no update if you where able to fix the issue Visseroth or not.
where you able to fix it or where the drives bad?

 

[joeschmuck]

I am having the same issue with drives and I have being buying more, I change the drives and then I got the same error on another one and change it again. So now I am starting to wonder that the drive are fine and is something else. I haven't done an smartctl on the drives to check if they where fine because swaping them will be easier but the problem persist.

this trent started on august 22 and the last message was on august 31 of 2018. There is no update if you where able to fix the issue Visseroth or not.
where you able to fix it or where the drives bad?

If you are just asking @Visseroth a question, this thread is fine. If you would like to troubleshoot your problem with help, I'd recommend a new thread and you should post your exact error messages and any troubleshooting you have done, also reference this thread as a similar problem. I'd recommend that you also run the testing outlined earlier and post those results as well. Also post your system hardware configuration so it will help us help you out.

Good Luck!

 

[Darklion]

thank you @joeschmuck and yes I was asking him the question, I wasn't trying to get help yet as I read everything and I am about to start doing the smartctl test and the hardware test also before I bother you guys.

I was just curious to see what was the result @Visseroth had and if he where able to fix the issue or end up swaping the disks.

thank you for keeping an eye on things, I appressiate it.

And taking advatage that you are here and answering, I want to ask a question. I am playing around just for the heck of learning this so I bought an old rackable system enclouser whit 24 bays, I only put 3 disks on raidz3 to test but I see the disks are being kept between 45 and 47 deegrees but my dell server keeps them at 34 -36. what is the maximum tempeture for the disks? will that tempeture damage them?

and just to bluf a little (lol) I olso have an ECM enclouser with 15 bays that came with 520 bit sectors drives that I was able to reformat to 512 with the camcontrol command. this forum has give me a lot of fun. I just still can't figure out how to do properly the firmware updates on the HBA Cards so I can only do it with the bootable CD that broadcome provides.

sorry for all the mininless information lol.

 

[joeschmuck]

what is the maximum tempeture for the disks? will that tempeture damage them?

Well that depends on the hard drive model to be honest but most would say anything above 50C is too hot. Ideal in my opinion is below 40C.

I am playing around just for the heck of learning this so I bought an old rackable system enclouser whit 24 bays, I only put 3 disks on raidz3 to test but I see the disks are being kept between 45 and 47 deegrees but my dell server keeps them at 34 -36.

I would request that you examine the new case specifically for operating fans and to figure out your airflow, I suspect you have improper airflow. For example, lets look at the 24 bay enclosure (I'm thinking of a generic case that pulls air from the front of the case to the rear) where you load it with up to 24 drives from the front, directly behind the hard drives is a series of fans hopefully, those fans will pull air across all those drives and even the non-populated drive slots. If you have a cheaper case you could be looking at only a few fans pulling air from the entire box or not well designed to pull air across all drive bays. I suspect that in your system you are pulling most of your air throught the empty drive bays and little air is being pulled across the actual hard drives. My advice, apply some tape to block air from coming in from most of the empty drive bays, do not block all of them, that is not good either. Take your time and use your brain to figure it out. If you think you have a fan issue, you should be able to address that too. If you need someones advice, a few good photos of the front, rear, and inside of the case would make it easier to offer assistance. I'm pretty big on air flow design, I like silent systems, I can't even hear my server it's so quiet under normal load and that wouldn't be possible without a good airflow strategy.