SMART Error this morning!

Kamal Soor · Oct 12, 2017

Help!

I just added a 2nd IBM 1015 (IT mode) to my system. I got errors when I created a New Volume, I thought I gotten a bad drive, OR I'd had overloaded my 500 watt power supply (13 drives and I was adding 3 new) so I deleted the volume and tried to figure out what was going on by adding the drives one at a time as single drive Volumes.

Last night I unplugged the New drives and found that I was getting a SMART Error this morning. I can't see any errors, can someone help? Is my drive at da5 failing?

Thanks,
Kamal

----------------------------------------------------------------
SMART Error!:

CRITICAL: Oct. 12, 2017, 7:29 a.m. - Device: /dev/da5 [SAT], FAILED SMART self-check. BACK UP DATA NOW!
CRITICAL: Oct. 12, 2017, 7:29 a.m. - Device: /dev/da5 [SAT], Failed SMART usage Attribute: 7 Seek_Error_Rate.

----------------------------------------------------------------
my system.

FreeNAS-9.10.2-U4 (27ae72978)
motherboard: ASUS 787-A
cpu: Core(TM) i5-4670K CPU @ 3.40GHz
Memory: 16226MBS
Load Average: 0.22, 0.20, 0.17
2 IBM 1015 (IT Mode)
Intel ethernet card.
500 watt power supply
13 drives

my zpools look ok
----------------------------------------------------------------

Code:

[root@freenas] ~# zpool status

  pool: freenas-boot

state: ONLINE

  scan: scrub repaired 0 in 0h8m with 0 errors on Tue Oct  3 03:53:25 2017

config:


	NAME										  STATE	 READ WRITE CKSUM

	freenas-boot								  ONLINE	   0	 0	 0

	  gptid/ed1791d2-c2d7-11e4-87ea-e03f49ea0e7e  ONLINE	   0	 0	 0


errors: No known data errors


  pool: vault1

state: ONLINE

  scan: scrub repaired 0 in 8h42m with 0 errors on Mon Sep  4 14:43:02 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	vault1										  ONLINE	   0	 0	 0

	  raidz1-0									  ONLINE	   0	 0	 0

		gptid/1e116bda-e94b-11e3-b7da-e03f49ea0e7e  ONLINE	   0	 0	 0

		gptid/1e6aac9e-e94b-11e3-b7da-e03f49ea0e7e  ONLINE	   0	 0	 0

		gptid/1ec504be-e94b-11e3-b7da-e03f49ea0e7e  ONLINE	   0	 0	 0


errors: No known data errors


  pool: vault2

state: ONLINE

  scan: scrub repaired 0 in 9h55m with 0 errors on Tue Sep  5 15:55:59 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	vault2										  ONLINE	   0	 0	 0

	  raidz1-0									  ONLINE	   0	 0	 0

		gptid/863543f1-eeb0-11e3-86d5-e03f49ea0e7e  ONLINE	   0	 0	 0

		gptid/868a7e4d-eeb0-11e3-86d5-e03f49ea0e7e  ONLINE	   0	 0	 0

		gptid/7cf2e64a-fdb2-11e4-aa9c-e03f49ea0e7e  ONLINE	   0	 0	 0


errors: No known data errors


  pool: vault3

state: ONLINE

  scan: scrub repaired 0 in 11h18m with 0 errors on Wed Sep 13 17:18:52 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	vault3										  ONLINE	   0	 0	 0

	  raidz1-0									  ONLINE	   0	 0	 0

		gptid/9ff29509-6473-11e4-bc08-e03f49ea0e7e  ONLINE	   0	 0	 0

		gptid/a1f384be-6473-11e4-bc08-e03f49ea0e7e  ONLINE	   0	 0	 0

		gptid/a3d72edb-6473-11e4-bc08-e03f49ea0e7e  ONLINE	   0	 0	 0


errors: No known data errors


  pool: vault4

state: ONLINE

  scan: scrub in progress since Thu Oct 12 06:00:00 2017

		4.35T scanned out of 10.2T at 259M/s, 6h37m to go

		0 repaired, 42.44% done

config:


	NAME											STATE	 READ WRITE CKSUM

	vault4										  ONLINE	   0	 0	 0

	  raidz1-0									  ONLINE	   0	 0	 0

		gptid/e01879f6-71de-11e5-90fe-e03f49ea0e7e  ONLINE	   0	 0	 0

		gptid/e29ba397-71de-11e5-90fe-e03f49ea0e7e  ONLINE	   0	 0	 0

		gptid/e540944c-71de-11e5-90fe-e03f49ea0e7e  ONLINE	   0	 0	 0


errors: No known data errors


  pool: vault5

state: ONLINE

  scan: scrub repaired 0 in 2h40m with 0 errors on Sun Sep 17 07:40:01 2017

config:


	NAME										  STATE	 READ WRITE CKSUM

	vault5										ONLINE	   0	 0	 0

	  gptid/e5225018-6332-11e7-bdb8-e03f49ea0e7e  ONLINE	   0	 0	 0


errors: No known data errors

[root@freenas] ~#

here is the out put from the SMARTCTL and I don't see any errors here either.

Code:

[root@freenas] ~# smartctl -a /dev/da5

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)

Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===

Model Family:	 HGST Deskstar NAS

Device Model:	 HGST HDN724040ALE640

Serial Number:	PK2334PCJSDPKB

LU WWN Device Id: 5 000cca 24ce6d333

Firmware Version: MJAOA5E0

User Capacity:	4,000,787,030,016 bytes [4.00 TB]

Sector Sizes:	 512 bytes logical, 4096 bytes physical

Rotation Rate:	7200 rpm

Form Factor:	  3.5 inches

Device is:		In smartctl database [for details use: -P show]

ATA Version is:   ATA8-ACS T13/1699-D revision 4

SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)

Local Time is:	Thu Oct 12 10:54:44 2017 EDT

SMART support is: Available - device has SMART capability.

SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED


General SMART Values:

Offline data collection status:  (0x82)	Offline data collection activity

					was completed without error.

					Auto Offline Data Collection: Enabled.

Self-test execution status:	  (   0)	The previous self-test routine completed

					without error or no self-test has ever

					been run.

Total time to complete Offline

data collection:		 (   24) seconds.

Offline data collection

capabilities:			 (0x5b) SMART execute Offline immediate.

					Auto Offline data collection on/off support.

					Suspend Offline collection upon new

					command.

					Offline surface scan supported.

					Self-test supported.

					No Conveyance Self-test supported.

					Selective Self-test supported.

SMART capabilities:			(0x0003)	Saves SMART data before entering

					power-saving mode.

					Supports SMART auto save timer.

Error logging capability:		(0x01)	Error logging supported.

					General Purpose Logging supported.

Short self-test routine

recommended polling time:	 (   1) minutes.

Extended self-test routine

recommended polling time:	 ( 569) minutes.

SCT capabilities:		   (0x003d)	SCT Status supported.

					SCT Error Recovery Control supported.

					SCT Feature Control supported.

					SCT Data Table supported.


SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate	 0x000b   100   100   016	Pre-fail  Always	   -	   0

  2 Throughput_Performance  0x0005   123   123   054	Pre-fail  Offline	  -	   125

  3 Spin_Up_Time			0x0007   253   253   024	Pre-fail  Always	   -	   112 (Average 121)

  4 Start_Stop_Count		0x0012   100   100   000	Old_age   Always	   -	   1062

  5 Reallocated_Sector_Ct   0x0033   100   100   005	Pre-fail  Always	   -	   0

  7 Seek_Error_Rate		 0x000b   100   100   067	Pre-fail  Always	   -	   0

  8 Seek_Time_Performance   0x0005   119   119   020	Pre-fail  Offline	  -	   35

  9 Power_On_Hours		  0x0012   098   098   000	Old_age   Always	   -	   15695

10 Spin_Retry_Count		0x0013   100   100   060	Pre-fail  Always	   -	   0

12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   143

192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   1071

193 Load_Cycle_Count		0x0012   100   100   000	Old_age   Always	   -	   1071

194 Temperature_Celsius	 0x0002   193   193   000	Old_age   Always	   -	   31 (Min/Max 19/38)

196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0

197 Current_Pending_Sector  0x0022   100   100   000	Old_age   Always	   -	   0

198 Offline_Uncorrectable   0x0008   100   100   000	Old_age   Offline	  -	   0

199 UDMA_CRC_Error_Count	0x000a   200   200   000	Old_age   Always	   -	   0


SMART Error Log Version: 1

No Errors Logged


SMART Self-test log structure revision number 1

Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error

# 1  Extended offline	Completed without error	   00%	 15561		 -

# 2  Extended offline	Completed without error	   00%	 15441		 -

# 3  Extended offline	Completed without error	   00%	 15258		 -

# 4  Extended offline	Completed without error	   00%	 15138		 -

# 5  Short offline	   Completed without error	   00%	 15101		 -

# 6  Short offline	   Completed without error	   00%	 15100		 -

# 7  Short offline	   Completed without error	   00%	 15077		 -

# 8  Short offline	   Completed without error	   00%	 15076		 -

# 9  Extended offline	Completed without error	   00%	 15018		 -

#10  Extended offline	Completed without error	   00%	 14767		 -

#11  Extended offline	Completed without error	   00%	 14637		 -

#12  Extended offline	Completed without error	   00%	 14517		 -

#13  Extended offline	Completed without error	   00%	 14398		 -

#14  Short offline	   Completed without error	   00%	 14361		 -

#15  Short offline	   Completed without error	   00%	 14360		 -

#16  Short offline	   Completed without error	   00%	 14337		 -

#17  Short offline	   Completed without error	   00%	 14336		 -

#18  Extended offline	Completed without error	   00%	 14278		 -

#19  Extended offline	Completed without error	   00%	 14159		 -

#20  Extended offline	Completed without error	   00%	 14039		 -

#21  Extended offline	Completed without error	   00%	 14015		 -


SMART Selective self-test log data structure revision number 1

SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS

	1		0		0  Not_testing

	2		0		0  Not_testing

	3		0		0  Not_testing

	4		0		0  Not_testing

	5		0		0  Not_testing

Selective self-test flags (0x0):

  After scanning selected spans, do NOT read-scan remainder of disk.

If Selective self-test is pending on power-up, resume after 0 minute delay.


[root@freenas] ~#

Inxsible · Oct 12, 2017

Your drive da5 as per SMART does look ok. You might want to check the logs to see what else was going on at 7:29 AM. There is a scrub that is currently going on in vault4. Is da5 in vault4?

Other than that, you have a few issues with your configuration:

You are not using ECC RAM
The amount of RAM you have is not enough for the amount of storage you have (1GB/TB) **
Your CPU doesn't support ECC RAM
Your motherboard doesn't support ECC RAM
You are using a consumer grade board with audio and display ports which are useless for a FreeNAS application
You have a high number of drives for a 500W PSU ( I have 6 drives running with 450W GOLD psu)
You have 5 pools instead of having 1 pool with 5 vdevs. Any reason for doing that?
You are using RAIDZ1, which offers only 1 drive redundancy. Maybe you should look into setting this whole thing up as RAIDZ2 especially now that you have 16 drives. You can set up 1 pool with 2 vdevs of 8 drives each in RAIDZ2

** You haven't specified all your drive sizes except da5 which is 4TB. So I am assuming they are all 4TB. 13x4TB = 52TB currently Adding 3x4TB drives would take that to 64TB. So you'd need at least 64GB of RAM.

m0nkey_ · Oct 12, 2017

OK, so the output from smartctl looks normal. Can you kick off a long SMART test on the drive and post the results when it's complete? It could be a false positive, or this really is an early warning about a bad drive.

Chris Moore · Oct 12, 2017

I am more inclined to think that the da number changed because of all the dinking around with drives and controllers. You should look at the SMART status of all the drives in your system and see if one of them lists a failed SMART test. You very likely do have a failing drive, you might just be looking at the wrong drive. Those da numbers do not always stay the same, especially when you have multiple controllers in a system and you are adding and removing drives.

Inxsible · Oct 12, 2017

Just don't run SMART on vault4 until the scrub completes !!!

Chris Moore · Oct 12, 2017

Kamal Soor said:
OR I'd had overloaded my 500 watt power supply

The power supply is absolutely a problem. What is the total number of drives you intend to have, we can suggest a power supply for you.
Also, please list your country as that helps us to know what resources might be available to you for purchasing hardware.

m0nkey_ · Oct 12, 2017

Inxsible said:
Just don't run SMART on vault4 until the scrub completes !!!

Oh boy. This.

I had a long SMART test on all drives collide with a scrub. I thought my drives or pool were failing. Make sure only one or the other are running at the time.

Chris Moore · Oct 12, 2017

Inxsible said:
** You haven't specified all your drive sizes except da5 which is 4TB. So I am assuming they are all 4TB. 13x4TB = 52TB currently Adding 3x4TB drives would take that to 64TB. So you'd need at least 64GB of RAM.

Agreed, many hardware issues here. One is that the CPU only supports up to 32GB of RAM and has an integrated GPU that takes part of RAM for video render.

Kamal Soor said:
my system.

FreeNAS-9.10.2-U4 (27ae72978)
motherboard: ASUS 787-A
cpu: Core(TM) i5-4670K CPU @ 3.40GHz
Memory: 16226MBS
Load Average: 0.22, 0.20, 0.17
2 IBM 1015 (IT Mode)
Intel ethernet card.
500 watt power supply
13 drives

To do this right, you should replace some of these things with newer, better hardware. It would also be good if you could flesh out the details of the hardware a little. Fore example, the drive sizes, are they all the same? What are they?

SweetAndLow · Oct 12, 2017

A little off-topic but does that motherboard even work? Most of the time everyone would say no but it seems like you have it working. What is your network performance? That board has a realtek nic so i'm curious.

EDIT: I see you have an extra intel nic so you can probably ignore my comment.

SweetAndLow · Oct 12, 2017

Inxsible said:
Your drive da5 as per SMART does look ok. You might want to check the logs to see what else was going on at 7:29 AM. There is a scrub that is currently going on in vault4. Is da5 in vault4?

Other than that, you have a few issues with your configuration:

You are not using ECC RAM

The amount of RAM you have is not enough for the amount of storage you have (1GB/TB) **

Your CPU doesn't support ECC RAM

Your motherboard doesn't support ECC RAM

You are using a consumer grade board with audio and display ports which are useless for a FreeNAS application

You have a high number of drives for a 500W PSU ( I have 6 drives running with 450W GOLD psu)

You have 5 pools instead of having 1 pool with 5 vdevs. Any reason for doing that?

You are using RAIDZ1, which offers only 1 drive redundancy. Maybe you should look into setting this whole thing up as RAIDZ2 especially now that you have 16 drives. You can set up 1 pool with 2 vdevs of 8 drives each in RAIDZ2

** You haven't specified all your drive sizes except da5 which is 4TB. So I am assuming they are all 4TB. 13x4TB = 52TB currently Adding 3x4TB drives would take that to 64TB. So you'd need at least 64GB of RAM.

Wow you unloaded on this person about their hardware but the things you pointed out don't really matter that much for the problem they are having.

1. ECC = who cares it doesn't affect how things work
2. As long as you have ~16GB you are fine. That silly 1GB to 1TB ratio is not used anymore, anything over 8GB is just fine usually.
3. ECC again, who cares
4. ECC again, who cares
5. You are close here but it seems to boot and they actually have an intel nic to fix the realteck nic problem.
6. Yeah the drive count is getting up there for the power levels. But I think it's still fine.
7 and 8 are ok comments.

Inxsible · Oct 12, 2017

SweetAndLow said:
Wow you unloaded on this person about their hardware but the things you pointed out don't really matter that much for the problem they are having.

Au contraire !!
The very first line of my comment pointed out that a scrub was ongoing in pool vault4 which could be problematic when running a simultaneous SMART test on the drives in the same pool.

Then, I clearly said "Other than that..." which would indicate that these were things that the OP should look into and compare them to the best practices for using FreeNAS.

SweetAndLow said:
1. ECC = who cares it doesn't affect how things work
2. As long as you have ~16GB you are fine. That silly 1GB to 1TB ratio is not used anymore, anything over 8GB is just fine usually.
3. ECC again, who cares
4. ECC again, who cares

The age old saying on this forum is : ECC doesn't matter until it does !! Take it the way you want it, I guess.

SweetAndLow said:
5. You are close here but it seems to boot and they actually have an intel nic to fix the realteck nic problem.

Don't know what you are on about here. I was talking about the audio and display ports which are unnecessary for FreeNAS

SweetAndLow said:
6. Yeah the drive count is getting up there for the power levels. But I think it's still fine.
7 and 8 are ok comments.

7 & 8 are ok comments, why? because the forum discourages RAIDZ1 -- nothing technically wrong with using RAIDZ1. So if you want to take "don't use RAIDZ1" as an OK comment, then using ECC should also have the same cachet since both statements are equally encouraged on the forums.

In the end, those comments were just to inform the OP that he/she is using non-recommended configuration -- hardware wise and software wise. Nothing about unloading on that person. ;)

Kamal Soor · Oct 12, 2017

Thanks to everyone for your help and feedback.

Inxsible:
When I built this system, I didn't realize the recommended hardware for a system build had EEC memory.
One of my reasons for using RaidZ1 is most (but not all ) files on the NAS are backups and part of Plex media server, and if one of the drives starts to fail the system can be taken offline until I get a replacement. I realize that if I loose two I'm toast.
After I get the system back up and running, I'd like to find out how to migrate to RaidZ2, but that will have to wait.

Here is some more information:

- da5 is part of Valut 4
- Scrub on vault 4 is complete without any issues.

How do I check my logs to see what else was going on at 7:29 AM? Where are the logs and what are the commands?

I just started the SMART Long test using the following is this right? This is going to take 9+ hours, wow !
smartctl -t long /dev/da5

Also I ran the SMART check on all my drives, it seems there are some errors, but I think these are old. . Am I right?
(fiy ada0, ada1, ada2 are part of Vault 1 and Vault 2)

Code:


@@@@@@@@@@@@
[root@freenas] ~# smartctl -a /dev/da0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 HGST Deskstar NAS
Device Model:	 HGST HDN724040ALE640
Serial Number:	PK2334PCH6D39B
LU WWN Device Id: 5 000cca 24cd0ff2d
Firmware Version: MJAOA5E0
User Capacity:	4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	7200 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Thu Oct 12 15:29:32 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection:		 (   24) seconds.
Offline data collection
capabilities:			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 ( 563) minutes.
SCT capabilities:			(0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000b   100   100   016	Pre-fail  Always	   -	   0
  2 Throughput_Performance  0x0005   136   136   054	Pre-fail  Offline	  -	   81
  3 Spin_Up_Time			0x0007   142   142   024	Pre-fail  Always	   -	   564 (Average 526)
  4 Start_Stop_Count		0x0012   100   100   000	Old_age   Always	   -	   179
  5 Reallocated_Sector_Ct   0x0033   100   100   005	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000b   100   100   067	Pre-fail  Always	   -	   0
  8 Seek_Time_Performance   0x0005   121   121   020	Pre-fail  Offline	  -	   34
  9 Power_On_Hours		  0x0012   097   097   000	Old_age   Always	   -	   23911
 10 Spin_Retry_Count		0x0013   100   100   060	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   173
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   698
193 Load_Cycle_Count		0x0012   100   100   000	Old_age   Always	   -	   698
194 Temperature_Celsius	 0x0002   166   166   000	Old_age   Always	   -	   36 (Min/Max 19/43)
196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0022   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0008   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x000a   200   200   000	Old_age   Always	   -	   148

SMART Error Log Version: 1
ATA Error Count: 148 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 148 occurred at disk power-on lifetime: 6987 hours (291 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 b1 2f 98 37 0e  Error: ICRC, ABRT at LBA = 0x0e37982f = 238524463

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 e0 98 37 40 00	  02:56:54.538  READ FPDMA QUEUED
  60 00 00 e0 97 37 40 00	  02:56:54.537  READ FPDMA QUEUED
  60 00 08 e0 96 37 40 00	  02:56:54.537  READ FPDMA QUEUED
  60 00 00 e0 95 37 40 00	  02:56:54.536  READ FPDMA QUEUED
  60 00 08 e0 94 37 40 00	  02:56:54.535  READ FPDMA QUEUED

Error 147 occurred at disk power-on lifetime: 6987 hours (291 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 01 2f b5 40 08  Error: ICRC, ABRT at LBA = 0x0840b52f = 138458415

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 88 00 b0 b5 40 40 00	  02:47:15.870  READ FPDMA QUEUED
  60 f0 08 40 b4 40 40 00	  02:47:15.869  READ FPDMA QUEUED
  60 d0 00 68 b3 40 40 00	  02:47:15.869  READ FPDMA QUEUED
  60 00 08 68 b2 40 40 00	  02:47:15.868  READ FPDMA QUEUED
  60 00 00 68 b1 40 40 00	  02:47:15.867  READ FPDMA QUEUED

Error 146 occurred at disk power-on lifetime: 6986 hours (291 days + 2 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 09 a7 31 d4 0b  Error: ICRC, ABRT at LBA = 0x0bd431a7 = 198455719

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 48 00 68 31 d4 40 00	  02:40:20.637  READ FPDMA QUEUED
  60 40 08 38 da f3 40 00	  02:40:20.637  READ FPDMA QUEUED
  60 38 00 20 e2 d3 40 00	  02:40:20.637  READ FPDMA QUEUED
  60 48 08 28 99 6a 40 00	  02:40:20.636  READ FPDMA QUEUED
  60 58 00 d0 98 6a 40 00	  02:40:20.636  READ FPDMA QUEUED

Error 145 occurred at disk power-on lifetime: 6986 hours (291 days + 2 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 41 0f 93 3a 0a  Error: ICRC, ABRT at LBA = 0x0a3a930f = 171610895

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 50 93 3a 40 00	  02:12:54.993  READ FPDMA QUEUED
  60 00 00 50 92 3a 40 00	  02:12:54.992  READ FPDMA QUEUED
  60 00 08 50 91 3a 40 00	  02:12:54.991  READ FPDMA QUEUED
  60 00 00 50 90 3a 40 00	  02:12:54.991  READ FPDMA QUEUED
  60 88 08 c8 8f 3a 40 00	  02:12:54.990  READ FPDMA QUEUED

Error 144 occurred at disk power-on lifetime: 6986 hours (291 days + 2 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 d1 7f b0 2b 0b  Error: ICRC, ABRT at LBA = 0x0b2bb07f = 187412607

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 50 b1 2b 40 00	  02:01:13.259  READ FPDMA QUEUED
  60 00 00 50 b0 2b 40 00	  02:01:13.259  READ FPDMA QUEUED
  60 00 08 50 af 2b 40 00	  02:01:13.258  READ FPDMA QUEUED
  60 00 00 50 ae 2b 40 00	  02:01:13.257  READ FPDMA QUEUED
  60 00 08 50 ad 2b 40 00	  02:01:13.257  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	 23879		 -
# 2  Extended offline	Completed without error	   00%	 23773		 -
# 3  Short offline	   Completed without error	   00%	 23763		 -
# 4  Extended offline	Completed without error	   00%	 23653		 -
# 5  Short offline	   Completed without error	   00%	 23643		 -
# 6  Short offline	   Completed without error	   00%	 23580		 -
# 7  Extended offline	Completed without error	   00%	 23470		 -
# 8  Short offline	   Completed without error	   00%	 23460		 -
# 9  Extended offline	Completed without error	   00%	 23350		 -
#10  Short offline	   Completed without error	   00%	 23340		 -
#11  Extended offline	Completed without error	   00%	 23231		 -
#12  Short offline	   Completed without error	   00%	 23220		 -
#13  Short offline	   Completed without error	   00%	 23101		 -
#14  Short offline	   Completed without error	   00%	 22983		 -
#15  Extended offline	Completed without error	   00%	 22969		 -
#16  Short offline	   Completed without error	   00%	 22959		 -
#17  Extended offline	Completed without error	   00%	 22849		 -
#18  Short offline	   Completed without error	   00%	 22839		 -
#19  Extended offline	Completed without error	   00%	 22731		 -
#20  Short offline	   Completed without error	   00%	 22719		 -
#21  Extended offline	Completed without error	   00%	 22610		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas] ~# 




=========================================================================================================================

@@@@@@@@@@@@



[root@freenas] ~# 
[root@freenas] ~# smartctl -a /dev/da1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 HGST Deskstar NAS
Device Model:	 HGST HDN724040ALE640
Serial Number:	PK1334PCHR6A4S
LU WWN Device Id: 5 000cca 24cd82f6b
Firmware Version: MJAOA5E0
User Capacity:	4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	7200 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Thu Oct 12 15:40:10 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection:		 (   24) seconds.
Offline data collection
capabilities:			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 ( 566) minutes.
SCT capabilities:			(0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000b   100   100   016	Pre-fail  Always	   -	   0
  2 Throughput_Performance  0x0005   136   136   054	Pre-fail  Offline	  -	   80
  3 Spin_Up_Time			0x0007   140   140   024	Pre-fail  Always	   -	   570 (Average 531)
  4 Start_Stop_Count		0x0012   100   100   000	Old_age   Always	   -	   224
  5 Reallocated_Sector_Ct   0x0033   100   100   005	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000b   100   100   067	Pre-fail  Always	   -	   0
  8 Seek_Time_Performance   0x0005   121   121   020	Pre-fail  Offline	  -	   34
  9 Power_On_Hours		  0x0012   097   097   000	Old_age   Always	   -	   23910
 10 Spin_Retry_Count		0x0013   100   100   060	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   173
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   337
193 Load_Cycle_Count		0x0012   100   100   000	Old_age   Always	   -	   337
194 Temperature_Celsius	 0x0002   176   176   000	Old_age   Always	   -	   34 (Min/Max 19/41)
196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0022   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0008   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x000a   200   200   000	Old_age   Always	   -	   130

SMART Error Log Version: 1
ATA Error Count: 130 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 130 occurred at disk power-on lifetime: 6987 hours (291 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 f1 d7 97 a7 0f  Error: ICRC, ABRT at LBA = 0x0fa797d7 = 262641623

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 c8 97 a7 40 00	  02:56:40.594  READ FPDMA QUEUED
  60 00 00 c8 96 a7 40 00	  02:56:40.594  READ FPDMA QUEUED
  60 00 08 c8 95 a7 40 00	  02:56:40.594  READ FPDMA QUEUED
  60 00 00 c8 94 a7 40 00	  02:56:40.594  READ FPDMA QUEUED
  60 00 08 c8 93 a7 40 00	  02:56:40.593  READ FPDMA QUEUED

Error 129 occurred at disk power-on lifetime: 6986 hours (291 days + 2 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 11 87 93 2e 0e  Error: ICRC, ABRT at LBA = 0x0e2e9387 = 237933447

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 08 08 d8 45 1c 40 00	  02:18:58.828  READ FPDMA QUEUED
  60 80 00 18 93 2e 40 00	  02:18:58.828  READ FPDMA QUEUED
  60 80 08 98 92 2e 40 00	  02:18:58.828  READ FPDMA QUEUED
  60 80 00 98 91 2e 40 00	  02:18:58.828  READ FPDMA QUEUED
  60 80 08 18 92 2e 40 00	  02:18:58.827  READ FPDMA QUEUED

Error 128 occurred at disk power-on lifetime: 6985 hours (291 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 c1 8f 93 08 0e  Error: ICRC, ABRT at LBA = 0x0e08938f = 235443087

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 50 94 08 40 00	  01:01:52.658  READ FPDMA QUEUED
  60 00 00 50 93 08 40 00	  01:01:52.657  READ FPDMA QUEUED
  60 00 08 50 92 08 40 00	  01:01:52.656  READ FPDMA QUEUED
  60 00 00 50 91 08 40 00	  01:01:52.655  READ FPDMA QUEUED
  60 88 08 c8 90 08 40 00	  01:01:52.654  READ FPDMA QUEUED

Error 127 occurred at disk power-on lifetime: 6984 hours (291 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 01 47 eb 48 06  Error: ICRC, ABRT at LBA = 0x0648eb47 = 105442119

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 48 eb 48 40 00	  00:08:58.855  READ FPDMA QUEUED
  60 00 00 48 ea 48 40 00	  00:08:58.855  READ FPDMA QUEUED
  60 00 08 48 e9 48 40 00	  00:08:58.854  READ FPDMA QUEUED
  60 00 00 48 e8 48 40 00	  00:08:58.853  READ FPDMA QUEUED
  60 00 08 48 e7 48 40 00	  00:08:58.852  READ FPDMA QUEUED

Error 126 occurred at disk power-on lifetime: 6982 hours (290 days + 22 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 21 ef 68 fe 02  Error: ICRC, ABRT at LBA = 0x02fe68ef = 50227439

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 10 69 fe 40 00   6d+03:14:08.439  READ FPDMA QUEUED
  60 00 08 10 68 fe 40 00   6d+03:14:08.438  READ FPDMA QUEUED
  60 00 00 10 67 fe 40 00   6d+03:14:08.437  READ FPDMA QUEUED
  60 00 08 10 66 fe 40 00   6d+03:14:08.436  READ FPDMA QUEUED
  60 00 00 10 65 fe 40 00   6d+03:14:08.434  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	 23878		 -
# 2  Extended offline	Completed without error	   00%	 23772		 -
# 3  Short offline	   Completed without error	   00%	 23762		 -
# 4  Extended offline	Completed without error	   00%	 23652		 -
# 5  Short offline	   Completed without error	   00%	 23642		 -
# 6  Short offline	   Completed without error	   00%	 23579		 -
# 7  Extended offline	Completed without error	   00%	 23469		 -
# 8  Short offline	   Completed without error	   00%	 23459		 -
# 9  Extended offline	Completed without error	   00%	 23349		 -
#10  Short offline	   Completed without error	   00%	 23339		 -
#11  Extended offline	Completed without error	   00%	 23230		 -
#12  Short offline	   Completed without error	   00%	 23219		 -
#13  Short offline	   Completed without error	   00%	 23100		 -
#14  Short offline	   Completed without error	   00%	 22983		 -
#15  Extended offline	Completed without error	   00%	 22969		 -
#16  Short offline	   Completed without error	   00%	 22959		 -
#17  Extended offline	Completed without error	   00%	 22848		 -
#18  Short offline	   Completed without error	   00%	 22839		 -
#19  Extended offline	Completed without error	   00%	 22730		 -
#20  Short offline	   Completed without error	   00%	 22719		 -
#21  Extended offline	Completed without error	   00%	 22609		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas] ~#

Kamal Soor · Oct 12, 2017

I maxed out on the message, here is the rest of the output from SMART on da2

Code:

=========================================================================================================================

@@@@@@@@@@@@


[root@freenas] ~# smartctl -a /dev/da2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 HGST Deskstar NAS
Device Model:	 HGST HDN724040ALE640
Serial Number:	PK1334PCHRERBS
LU WWN Device Id: 5 000cca 24cd84b2d
Firmware Version: MJAOA5E0
User Capacity:	4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	7200 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Thu Oct 12 15:41:23 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection:		 (   24) seconds.
Offline data collection
capabilities:			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 ( 569) minutes.
SCT capabilities:		   (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000b   100   100   016	Pre-fail  Always	   -	   0
  2 Throughput_Performance  0x0005   137   137   054	Pre-fail  Offline	  -	   77
  3 Spin_Up_Time			0x0007   139   139   024	Pre-fail  Always	   -	   581 (Average 529)
  4 Start_Stop_Count		0x0012   100   100   000	Old_age   Always	   -	   173
  5 Reallocated_Sector_Ct   0x0033   100   100   005	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000b   100   100   067	Pre-fail  Always	   -	   0
  8 Seek_Time_Performance   0x0005   121   121   020	Pre-fail  Offline	  -	   34
  9 Power_On_Hours		  0x0012   097   097   000	Old_age   Always	   -	   23913
 10 Spin_Retry_Count		0x0013   100   100   060	Pre-fail  Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   173
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   287
193 Load_Cycle_Count		0x0012   100   100   000	Old_age   Always	   -	   287
194 Temperature_Celsius	 0x0002   193   193   000	Old_age   Always	   -	   31 (Min/Max 18/38)
196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0022   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0008   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x000a   200   200   000	Old_age   Always	   -	   131

SMART Error Log Version: 1
ATA Error Count: 131 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 131 occurred at disk power-on lifetime: 6987 hours (291 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 61 3f ba 6d 0a  Error: ICRC, ABRT at LBA = 0x0a6dba3f = 174963263

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 a0 ba 6d 40 00	  03:25:12.453  READ FPDMA QUEUED
  60 00 00 a0 b9 6d 40 00	  03:25:12.453  READ FPDMA QUEUED
  60 00 08 a0 b8 6d 40 00	  03:25:12.452  READ FPDMA QUEUED
  60 00 00 a0 b7 6d 40 00	  03:25:12.451  READ FPDMA QUEUED
  60 00 08 a0 b6 6d 40 00	  03:25:12.450  READ FPDMA QUEUED

Error 130 occurred at disk power-on lifetime: 6987 hours (291 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 51 df 7e 77 0d  Error: ICRC, ABRT at LBA = 0x0d777edf = 225935071

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 30 7f 77 40 00	  02:55:16.508  READ FPDMA QUEUED
  60 00 08 30 7e 77 40 00	  02:55:16.508  READ FPDMA QUEUED
  60 00 00 30 7d 77 40 00	  02:55:16.507  READ FPDMA QUEUED
  60 00 08 30 7c 77 40 00	  02:55:16.506  READ FPDMA QUEUED
  60 00 00 20 7b 77 40 00	  02:55:16.505  READ FPDMA QUEUED

Error 129 occurred at disk power-on lifetime: 6987 hours (291 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 b1 bf 0f cd 04  Error: ICRC, ABRT at LBA = 0x04cd0fbf = 80547775

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 70 0f cd 40 00	  02:40:46.633  READ FPDMA QUEUED
  60 00 00 70 0e cd 40 00	  02:40:46.632  READ FPDMA QUEUED
  60 00 00 70 0d cd 40 00	  02:40:46.631  READ FPDMA QUEUED
  60 00 00 70 0c cd 40 00	  02:40:46.630  READ FPDMA QUEUED
  60 00 00 70 0b cd 40 00	  02:40:46.629  READ FPDMA QUEUED

Error 128 occurred at disk power-on lifetime: 6985 hours (291 days + 1 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 59 57 aa a5 07  Error: ICRC, ABRT at LBA = 0x07a5aa57 = 128297559

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 b8 08 b0 aa a5 40 00	  00:52:47.418  READ FPDMA QUEUED
  60 d8 00 d8 a9 a5 40 00	  00:52:47.417  READ FPDMA QUEUED
  60 c8 08 10 a9 a5 40 00	  00:52:47.416  READ FPDMA QUEUED
  60 00 00 10 a8 a5 40 00	  00:52:47.416  READ FPDMA QUEUED
  60 d8 08 30 a7 a5 40 00	  00:52:47.415  READ FPDMA QUEUED

Error 127 occurred at disk power-on lifetime: 6983 hours (290 days + 23 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 e1 d7 d2 38 03  Error: ICRC, ABRT at LBA = 0x0338d2d7 = 54055639

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 f0 00 b8 d3 38 40 00   6d+04:28:42.338  READ FPDMA QUEUED
  60 00 08 b8 d2 38 40 00   6d+04:28:42.337  READ FPDMA QUEUED
  60 00 00 b8 d1 38 40 00   6d+04:28:42.336  READ FPDMA QUEUED
  60 00 08 b8 d0 38 40 00   6d+04:28:42.336  READ FPDMA QUEUED
  60 f0 00 c8 cf 38 40 00   6d+04:28:42.335  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	 23880		 -
# 2  Extended offline	Completed without error	   00%	 23775		 -
# 3  Short offline	   Completed without error	   00%	 23765		 -
# 4  Extended offline	Completed without error	   00%	 23655		 -
# 5  Short offline	   Completed without error	   00%	 23645		 -
# 6  Short offline	   Completed without error	   00%	 23582		 -
# 7  Extended offline	Completed without error	   00%	 23472		 -
# 8  Short offline	   Completed without error	   00%	 23462		 -
# 9  Extended offline	Completed without error	   00%	 23352		 -
#10  Short offline	   Completed without error	   00%	 23342		 -
#11  Extended offline	Completed without error	   00%	 23233		 -
#12  Short offline	   Completed without error	   00%	 23222		 -
#13  Short offline	   Completed without error	   00%	 23103		 -
#14  Short offline	   Completed without error	   00%	 22985		 -
#15  Extended offline	Completed without error	   00%	 22971		 -
#16  Short offline	   Completed without error	   00%	 22961		 -
#17  Extended offline	Completed without error	   00%	 22851		 -
#18  Short offline	   Completed without error	   00%	 22841		 -
#19  Extended offline	Completed without error	   00%	 22733		 -
#20  Short offline	   Completed without error	   00%	 22721		 -
#21  Extended offline	Completed without error	   00%	 22612		 -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas] ~#

Kamal Soor · Oct 13, 2017

hello everyone,

After starting up the Long SMART test on da5 last night, this morning I woke up my Mac, and I discovered I had been logged out of my FreeNas box.

I think I was supposed to use smartctl -a /dev/da5 to see LONG results, is this right? or did I screw-up and I need to run the test again on da5?

Here is the output, looks clean to me

Code:


[root@freenas] ~# smartctl -a /dev/da5

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)

Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===

Model Family:	 HGST Deskstar NAS

Device Model:	 HGST HDN724040ALE640

Serial Number:	PK2334PCJSDPKB

LU WWN Device Id: 5 000cca 24ce6d333

Firmware Version: MJAOA5E0

User Capacity:	4,000,787,030,016 bytes [4.00 TB]

Sector Sizes:	 512 bytes logical, 4096 bytes physical

Rotation Rate:	7200 rpm

Form Factor:	  3.5 inches

Device is:		In smartctl database [for details use: -P show]

ATA Version is:   ATA8-ACS T13/1699-D revision 4

SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)

Local Time is:	Fri Oct 13 11:25:14 2017 EDT

SMART support is: Available - device has SMART capability.

SMART support is: Enabled


=== START OF READ SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED


General SMART Values:

Offline data collection status:  (0x82)	Offline data collection activity

					was completed without error.

					Auto Offline Data Collection: Enabled.

Self-test execution status:	  (   0)	The previous self-test routine completed

					without error or no self-test has ever 

					been run.

Total time to complete Offline 

data collection:		 (   24) seconds.

Offline data collection

capabilities:			 (0x5b) SMART execute Offline immediate.

					Auto Offline data collection on/off support.

					Suspend Offline collection upon new

					command.

					Offline surface scan supported.

					Self-test supported.

					No Conveyance Self-test supported.

					Selective Self-test supported.

SMART capabilities:			(0x0003)	Saves SMART data before entering

					power-saving mode.

					Supports SMART auto save timer.

Error logging capability:		(0x01)	Error logging supported.

					General Purpose Logging supported.

Short self-test routine 

recommended polling time:	 (   1) minutes.

Extended self-test routine

recommended polling time:	 ( 569) minutes.

SCT capabilities:		   (0x003d)	SCT Status supported.

					SCT Error Recovery Control supported.

					SCT Feature Control supported.

					SCT Data Table supported.


SMART Attributes Data Structure revision number: 16

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate	 0x000b   100   100   016	Pre-fail  Always	   -	   0

  2 Throughput_Performance  0x0005   123   123   054	Pre-fail  Offline	  -	   125

  3 Spin_Up_Time			0x0007   253   253   024	Pre-fail  Always	   -	   136 (Average 135)

  4 Start_Stop_Count		0x0012   100   100   000	Old_age   Always	   -	   1201

  5 Reallocated_Sector_Ct   0x0033   100   100   005	Pre-fail  Always	   -	   0

  7 Seek_Error_Rate		 0x000b   100   100   067	Pre-fail  Always	   -	   0

  8 Seek_Time_Performance   0x0005   119   119   020	Pre-fail  Offline	  -	   35

  9 Power_On_Hours		  0x0012   098   098   000	Old_age   Always	   -	   15719

 10 Spin_Retry_Count		0x0013   100   100   060	Pre-fail  Always	   -	   0

 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   143

192 Power-Off_Retract_Count 0x0032   099   099   000	Old_age   Always	   -	   1210

193 Load_Cycle_Count		0x0012   099   099   000	Old_age   Always	   -	   1210

194 Temperature_Celsius	 0x0002   200   200   000	Old_age   Always	   -	   30 (Min/Max 19/38)

196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0

197 Current_Pending_Sector  0x0022   100   100   000	Old_age   Always	   -	   0

198 Offline_Uncorrectable   0x0008   100   100   000	Old_age   Offline	  -	   0

199 UDMA_CRC_Error_Count	0x000a   200   200   000	Old_age   Always	   -	   0


SMART Error Log Version: 1

No Errors Logged


SMART Self-test log structure revision number 1

Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error

# 1  Extended offline	Completed without error	   00%	 15714		 -

# 2  Extended offline	Completed without error	   00%	 15561		 -

# 3  Extended offline	Completed without error	   00%	 15441		 -

# 4  Extended offline	Completed without error	   00%	 15258		 -

# 5  Extended offline	Completed without error	   00%	 15138		 -

# 6  Short offline	   Completed without error	   00%	 15101		 -

# 7  Short offline	   Completed without error	   00%	 15100		 -

# 8  Short offline	   Completed without error	   00%	 15077		 -

# 9  Short offline	   Completed without error	   00%	 15076		 -

#10  Extended offline	Completed without error	   00%	 15018		 -

#11  Extended offline	Completed without error	   00%	 14767		 -

#12  Extended offline	Completed without error	   00%	 14637		 -

#13  Extended offline	Completed without error	   00%	 14517		 -

#14  Extended offline	Completed without error	   00%	 14398		 -

#15  Short offline	   Completed without error	   00%	 14361		 -

#16  Short offline	   Completed without error	   00%	 14360		 -

#17  Short offline	   Completed without error	   00%	 14337		 -

#18  Short offline	   Completed without error	   00%	 14336		 -

#19  Extended offline	Completed without error	   00%	 14278		 -

#20  Extended offline	Completed without error	   00%	 14159		 -

#21  Extended offline	Completed without error	   00%	 14039		 -


SMART Selective self-test log data structure revision number 1

 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS

	1		0		0  Not_testing

	2		0		0  Not_testing

	3		0		0  Not_testing

	4		0		0  Not_testing

	5		0		0  Not_testing

Selective self-test flags (0x0):

  After scanning selected spans, do NOT read-scan remainder of disk.

If Selective self-test is pending on power-up, resume after 0 minute delay.


[root@freenas] ~#

Inxsible · Oct 13, 2017

yeah, looks OK.

You might want to run a manual SMART on every drive you have because it might not be da5 which had the issue in the first place. Also as @Chris Moore mentioned, you cannot rely on da# as that will change. gptId is the best way to pin-point a drive.

Chris Moore · Oct 13, 2017

There are actually a bunch of monitoring and reporting scripts available that you might want to look into running.

Here is the link to the place that tells you how to set them up, don't skip to the end, the important bit is at the beginning:
https://forums.freenas.org/index.ph...d-identification-and-backup-the-config.27365/

Here is the link to the Github Repository info:
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

Chris Moore · Oct 13, 2017

Kamal Soor said:
Here is the output

Sample output of one of the scripts I mentioned:

Code:

########## SMART status report summary for all drives ##########

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|Device|Serial		 |Temp|Power|Start|Spin |ReAlloc|Current|Offline |UDMA  |Seek  |High  |Command|Last|
|	  |			   |	|On   |Stop |Retry|Sectors|Pending|Uncorrec|CRC   |Errors|Fly   |Timeout|Test|
|	  |			   |	|Hours|Count|Count|	   |Sectors|Sectors |Errors|	  |Writes|Count  |Age |
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|da0   |WD-WCC1T087****| 32 | 9352|   70|	0|	  0|	  0|	   0|	 0|   N/A|   N/A|	N/A|   2|
|da1   |WD-WCC1T086****| 30 | 9352|   70|	0|	  0|	  0|	   0|	 0|   N/A|   N/A|	N/A|   2|
|da2   |W300****	   | 31 | 9357|   72|	0|	  0|	  0|	   0|	 0|	 0|	13|	  0|   2|
|da3   |W300****	   | 29 | 9357|   72|	0|	  0|	  0|	   0|	 0|	 0|	 9|	  0|   2|
|da4   |WD-WCC4N0HT****| 30 | 4472|   23|	0|	  0|	  0|	   0|	 0|   N/A|   N/A|	N/A|   2|
|da5   |WD-WCC4NEFD****| 30 | 4472|   23|	0|	  0|	  0|	   0|	 0|   N/A|   N/A|	N/A|   2|
|da6   |W730****	   | 30 | 4475|   23|	0|	  0|	  0|	   0|	 0|	 0|	 2|	  0|   2|
|da7   |W730****	   | 29 | 4475|   23|	0|	  0|	  0|	   0|	 0|	 0|	 6|	  0|   2|
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+

It is very concise and lists as many drives as you have.
I use it on one of my servers at work that has 62 drives in the list. It is much easier to scan down this list than to manually pull the report on each drive.

Kamal Soor · Oct 13, 2017

Chris, Thank you very much! I'm going to install and run the scripts over the weekend.

New Question: Testing New Drives
-----------------
Can someone point to the way I can SAFELY take all my existing vaults offline (plex, AFP shares etc) and unplug the drives, so that I can plug in the three new drives to make sure I didn't get some a duds, without screwing up my whole setup.

OR is there any better way to test the dives and my new M1015 card. Should I do a new install with just the 3 new drives with a black flash disk?

PS I'm a Mac user

Kamal Soor · Oct 15, 2017

bump...

Chris Moore · Oct 15, 2017

You could remove the boot drive along with the rest of the drives. That would ensure that your configuration is not affected.
Then make a temporary installation on another media to run the tests on the new drives.
I use dban boot and nuke to do a DOD wipe on my new drives. If they come through that with no errors, they are probably fine.
There are other testing procedures listed on the forum that other people have had success with.

Important Announcement for the TrueNAS Community.

SMART Error this morning!

Cadet

Guru

MVP

Hall of Famer

Guru

Hall of Famer

MVP

Hall of Famer

Sweet'NASty

Sweet'NASty

Guru

Cadet

Cadet

Cadet

Guru

Hall of Famer

Hall of Famer

Cadet

Cadet

Hall of Famer

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "SMART Error this morning!"

Similar threads