CRITICAL: Aug. 18, 2017, 8:03 a.m. - The volume trust state is ONLINE

dirkme · Aug 18, 2017

Just wondering i am moving data around to free 2 x 2 TB drives as I get another 2 x 2 TB and want to use them as raid ? 1 drive for redundancy.

However, I have 2x 1 TBdrives working a s a mirriopr and copy some more important data on them and got that message below, wondering if all is OK again?

I can't see any failure message at the volume or disks.

CRITICAL: Aug. 18, 2017, 8:03 a.m. - The volume trust state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

gpsguy · Aug 18, 2017

Please do a zpool status and post the results in code tags.

dirkme · Aug 18, 2017

Code:

root@voyager:~ # zpool status
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

   NAME  STATE  READ WRITE CKSUM
   freenas-boot  ONLINE  0  0  0
	da0p2  ONLINE  0  0  0

errors: No known data errors

  pool: trust
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
   attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
   using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
  scan: none requested
config:

   NAME  STATE  READ WRITE CKSUM
   trust  ONLINE  0  0  0
	mirror-0  ONLINE  0  0  0
	gptid/8866752a-7750-11e7-b015-bc5ff4ac2b1d.eli  ONLINE  0  0  0
	gptid/89a53332-7750-11e7-b015-bc5ff4ac2b1d.eli  ONLINE  0  0  1

errors: No known data errors

  pool: trust1
state: ONLINE
  scan: none requested
config:

   NAME  STATE  READ WRITE CKSUM
   trust1  ONLINE  0  0  0
	gptid/080fc0d9-7023-11e7-b8cd-bc5ff4ac2b1d.eli  ONLINE  0  0  0

errors: No known data errors

  pool: trust2
state: ONLINE
  scan: none requested
config:

   NAME  STATE  READ WRITE CKSUM
   trust2  ONLINE  0  0  0
	gptid/cb32ec73-700a-11e7-b8cd-bc5ff4ac2b1d.eli  ONLINE  0  0  0

errors: No known data errors

How do I actually clear the error-message?

Ericloewe · Aug 18, 2017

With zpool clear, but you'll want to look at that drive's SMART data to see what's going on.

gpsguy · Aug 18, 2017

Don't clear the message yet.

As you can see there's a checksum error on the 2nd drive in your "trust" volume.

Do you run regular SMART tests on your drives. If you map the gptid to the device name, like ada0 (or whatever), you can retrieve view the results using smartctl -a /dev/ada0 and post these results in code tags.

dirkme · Aug 18, 2017

Code:

root@voyager:~ # smartctl -a /dev/ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:  Seagate NAS HDD
Device Model:  ST2000VN000-1H3164
Serial Number:  W1H291V0
LU WWN Device Id: 5 000c50 06a7a4ec2
Firmware Version: SC42
User Capacity:  2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  5900 rpm
Form Factor:  3.5 inches
Device is:  In smartctl database [for details use: -P show]
ATA Version is:  ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Fri Aug 18 12:02:26 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)   Offline data collection activity
		   was completed without error.
		   Auto Offline Data Collection: Enabled.
Self-test execution status:  (  0)   The previous self-test routine completed
		   without error or no self-test has ever
		   been run.
Total time to complete Offline
data collection:	  (  97) seconds.
Offline data collection
capabilities:		 (0x7b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  1) minutes.
Extended self-test routine
recommended polling time:	 ( 264) minutes.
Conveyance self-test routine
recommended polling time:	 (  2) minutes.
SCT capabilities:	 (0x10bd)   SCT Status supported.
		   SCT Error Recovery Control supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000f  118  099  006  Pre-fail  Always  -  189289240
  3 Spin_Up_Time  0x0003  096  095  000  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  020  Old_age  Always  -  203
  5 Reallocated_Sector_Ct  0x0033  100  100  010  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000f  075  060  030  Pre-fail  Always  -  30339761117
  9 Power_On_Hours  0x0032  072  072  000  Old_age  Always  -  25199
 10 Spin_Retry_Count  0x0013  100  100  097  Pre-fail  Always  -  0
 12 Power_Cycle_Count  0x0032  100  100  020  Old_age  Always  -  198
184 End-to-End_Error  0x0032  100  100  099  Old_age  Always  -  0
187 Reported_Uncorrect  0x0032  100  100  000  Old_age  Always  -  0
188 Command_Timeout  0x0032  100  099  000  Old_age  Always  -  12
189 High_Fly_Writes  0x003a  001  001  000  Old_age  Always  -  335
190 Airflow_Temperature_Cel 0x0022  073  058  045  Old_age  Always  -  27 (Min/Max 26/28)
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  0
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  70
193 Load_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  203
194 Temperature_Celsius  0x0022  027  042  000  Old_age  Always  -  27 (0 16 0 0 0)
197 Current_Pending_Sector  0x0012  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0010  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x003e  200  200  000  Old_age  Always  -  0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description  Status  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline  Completed without error  00%  25117  -
# 2  Short offline  Completed without error  00%  24782  -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

gpsguy · Aug 18, 2017

I don't think this is the right drive. Please recheck, or just run it on all 4 of them and post the results.

When I said ada0, it was just an example, so you'd see the syntax.

btw, it looks like you've only done 2 short tests recently, in the last 3 years or so. You'll want to do a long test on it. And, set it up to do short and long tests on a regular schedule. sorry for going off on a tangent.

dirkme · Aug 18, 2017

gpsguy said:
I don't think this is the right drive. Please recheck, or just run it on all 4 of them and post the results.

When I said ada0, it was just an example, so you'd see the syntax.

btw, it looks like you've only done 2 short tests recently, in the last 3 years or so. You'll want to do a long test on it. And, set it up to do short and long tests on a regular schedule. sorry for going off on a tangent.

I just added those drives maybe 6 weeks ago, and now they are holding some important data until (25 August) until my other 2 x 2 TB arrive and i can rebuild a pool with 4 x 2 TB, after that i can play around with these 2 disks, don't want to risk much yet. I set a SMART test for 13:00 which is in 45 minutes, after that i should get some better out put and will post it.

It is just that the error message doesn't say which drive it was, sigh

dirkme · Aug 18, 2017

These are the 2 drives from that "trust" pool. I think ada3 is the culprit it says something with an error

Code:

198 Offline_Uncorrectable  0x0030  100  100  000  Old_age  Offline  -  0

Code:

root@voyager:~ # smartctl -a /dev/ada3
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:  Toshiba 2.5" HDD MQ01ABD...
Device Model:  TOSHIBA MQ01ABD100
Serial Number:  46CMT294T
LU WWN Device Id: 5 000039 6f1a82f9b
Firmware Version: AX1P2A
User Capacity:  1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  5400 rpm
Form Factor:  2.5 inches
Device is:  In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Fri Aug 18 12:22:54 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)   Offline data collection activity
		   was never started.
		   Auto Offline Data Collection: Disabled.
Self-test execution status:  (  0)   The previous self-test routine completed
		   without error or no self-test has ever
		   been run.
Total time to complete Offline
data collection:	  (  120) seconds.
Offline data collection
capabilities:		 (0x5b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   No Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  2) minutes.
Extended self-test routine
recommended polling time:	 ( 244) minutes.
SCT capabilities:	 (0x003d)   SCT Status supported.
		   SCT Error Recovery Control supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000b  100  100  050  Pre-fail  Always  -  0
  2 Throughput_Performance  0x0005  100  100  050  Pre-fail  Offline  -  0
  3 Spin_Up_Time  0x0027  100  100  001  Pre-fail  Always  -  1899
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  114
  5 Reallocated_Sector_Ct  0x0033  100  100  050  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000b  100  100  050  Pre-fail  Always  -  0
  8 Seek_Time_Performance  0x0005  100  100  050  Pre-fail  Offline  -  0
  9 Power_On_Hours  0x0032  081  081  000  Old_age  Always  -  7823
 10 Spin_Retry_Count  0x0033  102  100  030  Pre-fail  Always  -  0
 12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  113
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  1
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  18
193 Load_Cycle_Count  0x0032  093  093  000  Old_age  Always  -  78796
194 Temperature_Celsius  0x0022  100  100  000  Old_age  Always  -  33 (Min/Max 20/55)
196 Reallocated_Event_Count 0x0032  100  100  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
220 Disk_Shift  0x0002  100  100  000  Old_age  Always  -  0
222 Loaded_Hours  0x0032  096  096  000  Old_age  Always  -  1832
223 Load_Retry_Count  0x0032  100  100  000  Old_age  Always  -  0
224 Load_Friction  0x0022  100  100  000  Old_age  Always  -  0
226 Load-in_Time  0x0026  100  100  000  Old_age  Always  -  281
240 Head_Flying_Hours  0x0001  100  100  001  Pre-fail  Offline  -  0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@voyager:~ # smartctl -a /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:  Toshiba 2.5" HDD MQ01ABD...
Device Model:  TOSHIBA MQ01ABD100
Serial Number:  46CMT296T
LU WWN Device Id: 5 000039 6f1a82f9d
Firmware Version: AX1P2A
User Capacity:  1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  5400 rpm
Form Factor:  2.5 inches
Device is:  In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Fri Aug 18 12:23:08 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)   Offline data collection activity
		   was never started.
		   Auto Offline Data Collection: Disabled.
Self-test execution status:  (  0)   The previous self-test routine completed
		   without error or no self-test has ever
		   been run.
Total time to complete Offline
data collection:	  (  120) seconds.
Offline data collection
capabilities:		 (0x5b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   No Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  2) minutes.
Extended self-test routine
recommended polling time:	 ( 238) minutes.
SCT capabilities:	 (0x003d)   SCT Status supported.
		   SCT Error Recovery Control supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000b  100  100  050  Pre-fail  Always  -  0
  2 Throughput_Performance  0x0005  100  100  050  Pre-fail  Offline  -  0
  3 Spin_Up_Time  0x0027  100  100  001  Pre-fail  Always  -  1900
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  155
  5 Reallocated_Sector_Ct  0x0033  100  100  050  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000b  100  100  050  Pre-fail  Always  -  0
  8 Seek_Time_Performance  0x0005  100  100  050  Pre-fail  Offline  -  0
  9 Power_On_Hours  0x0032  081  081  000  Old_age  Always  -  7820
 10 Spin_Retry_Count  0x0033  102  100  030  Pre-fail  Always  -  0
 12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  155
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  1
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  28
193 Load_Cycle_Count  0x0032  092  092  000  Old_age  Always  -  88489
194 Temperature_Celsius  0x0022  100  100  000  Old_age  Always  -  32 (Min/Max 21/44)
196 Reallocated_Event_Count 0x0032  100  100  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  1
220 Disk_Shift  0x0002  100  100  000  Old_age  Always  -  0
222 Loaded_Hours  0x0032  097  097  000  Old_age  Always  -  1307
223 Load_Retry_Count  0x0032  100  100  000  Old_age  Always  -  0
224 Load_Friction  0x0022  100  100  000  Old_age  Always  -  0
226 Load-in_Time  0x0026  100  100  000  Old_age  Always  -  268
240 Head_Flying_Hours  0x0001  100  100  001  Pre-fail  Offline  -  0

SMART Error Log Version: 1
ATA Error Count: 1
   CR = Command Register [HEX]
   FR = Features Register [HEX]
   SC = Sector Count Register [HEX]
   SN = Sector Number Register [HEX]
   CL = Cylinder Low Register [HEX]
   CH = Cylinder High Register [HEX]
   DH = Device/Head Register [HEX]
   DC = Device Command Register [HEX]
   ER = Error register [HEX]
   ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 7423 hours (309 days + 7 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 a8 37 6b 49 40  Error: ICRC, ABRT at LBA = 0x00496b37 = 4811575

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC  Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 a8 38 6a 49 40 00  07:38:28.252  WRITE FPDMA QUEUED
  ef 02 00 00 00 00 40 00  07:38:28.252  SET FEATURES [Enable write cache]
  ef aa 00 00 00 00 40 00  07:38:28.252  SET FEATURES [Enable read look-ahead]
  c6 00 10 00 00 00 40 00  07:38:28.250  SET MULTIPLE MODE
  ef 10 02 00 00 00 40 00  07:38:28.249  SET FEATURES [Enable SATA feature]

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

gpsguy · Aug 18, 2017

ada1 is the problem drive.

Code:

199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 1

give me a minute to compose a fuller reply.

dirkme said:
I think ada3 is the culprit it says something with an error

gpsguy · Aug 18, 2017

Since you've never run SMART tests on these 2.5" Toshiba's, I would run a short test, followed by a long test. The short one only takes a few minutes, the long one, probably a few hours.

The syntax would be smartctl -t short /dev/ada1 For long tests, substitute long for short You can review the results, like you've already done.

One thing I noticed, is your Load_Cycle_Count is high. On WD Greens, one can run a DOS command to mitigate the issue. I don't know if Toshiba has something similar or not.

Do you plan to re-use these drives in your new pool?

Ericloewe · Aug 18, 2017

Yeah, you really have to setup regular SMART tests. Something like one short per day plus a long test every two weeks.

dirkme · Aug 18, 2017

gpsguy said:
Do you plan to re-use these drives in your new pool?

I hoped to have these 2 Toshiba drives still running as a mirrored zpool, do you think they still any good for holding my jails and some easy stuff?

Seems, I have to look into a case which can hold 6 x 3.5" drives, I just don't have the funds yet for the case or the extra 2 x 2 TB drives, sigh.

gpsguy · Aug 18, 2017

Run the short tests on them (only takes a few minutes), capture the results, run long tests (takes hours), capture the results.

We'll know more about them after the testing.

As @Ericloewe and I have said, you'll want to schedule SMART tests to run automatically. See: http://doc.freenas.org/11/tasks.html#s-m-a-r-t-tests

dirkme said:
I hoped to have these 2 Toshiba drives still running as a mirrored zpool

dirkme · Aug 18, 2017

ada1

if it says offline testing, what does that mean?

Code:

root@voyager:~ # smartctl -a /dev/ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:  Toshiba 2.5" HDD MQ01ABD...
Device Model:  TOSHIBA MQ01ABD100
Serial Number:  46CMT296T
LU WWN Device Id: 5 000039 6f1a82f9d
Firmware Version: AX1P2A
User Capacity:  1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  5400 rpm
Form Factor:  2.5 inches
Device is:  In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Fri Aug 18 13:47:37 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)   Offline data collection activity
		   was never started.
		   Auto Offline Data Collection: Disabled.
Self-test execution status:  (  0)   The previous self-test routine completed
		   without error or no self-test has ever
		   been run.
Total time to complete Offline
data collection:	  (  120) seconds.
Offline data collection
capabilities:		 (0x5b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   No Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  2) minutes.
Extended self-test routine
recommended polling time:	 ( 238) minutes.
SCT capabilities:	 (0x003d)   SCT Status supported.
		   SCT Error Recovery Control supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000b  100  100  050  Pre-fail  Always  -  0
  2 Throughput_Performance  0x0005  100  100  050  Pre-fail  Offline  -  0
  3 Spin_Up_Time  0x0027  100  100  001  Pre-fail  Always  -  1900
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  155
  5 Reallocated_Sector_Ct  0x0033  100  100  050  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000b  100  100  050  Pre-fail  Always  -  0
  8 Seek_Time_Performance  0x0005  100  100  050  Pre-fail  Offline  -  0
  9 Power_On_Hours  0x0032  081  081  000  Old_age  Always  -  7822
10 Spin_Retry_Count  0x0033  102  100  030  Pre-fail  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  155
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  1
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  28
193 Load_Cycle_Count  0x0032  092  092  000  Old_age  Always  -  88489
194 Temperature_Celsius  0x0022  100  100  000  Old_age  Always  -  32 (Min/Max 21/44)
196 Reallocated_Event_Count 0x0032  100  100  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  1
220 Disk_Shift  0x0002  100  100  000  Old_age  Always  -  0
222 Loaded_Hours  0x0032  097  097  000  Old_age  Always  -  1309
223 Load_Retry_Count  0x0032  100  100  000  Old_age  Always  -  0
224 Load_Friction  0x0022  100  100  000  Old_age  Always  -  0
226 Load-in_Time  0x0026  100  100  000  Old_age  Always  -  268
240 Head_Flying_Hours  0x0001  100  100  001  Pre-fail  Offline  -  0

SMART Error Log Version: 1
ATA Error Count: 1
   CR = Command Register [HEX]
   FR = Features Register [HEX]
   SC = Sector Count Register [HEX]
   SN = Sector Number Register [HEX]
   CL = Cylinder Low Register [HEX]
   CH = Cylinder High Register [HEX]
   DH = Device/Head Register [HEX]
   DC = Device Command Register [HEX]
   ER = Error register [HEX]
   ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 7423 hours (309 days + 7 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 41 a8 37 6b 49 40  Error: ICRC, ABRT at LBA = 0x00496b37 = 4811575

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC  Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 a8 38 6a 49 40 00  07:38:28.252  WRITE FPDMA QUEUED
  ef 02 00 00 00 00 40 00  07:38:28.252  SET FEATURES [Enable write cache]
  ef aa 00 00 00 00 40 00  07:38:28.252  SET FEATURES [Enable read look-ahead]
  c6 00 10 00 00 00 40 00  07:38:28.250  SET MULTIPLE MODE
  ef 10 02 00 00 00 40 00  07:38:28.249  SET FEATURES [Enable SATA feature]

SMART Self-test log structure revision number 1
Num  Test_Description  Status  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline  Completed without error  00%  7821  -
# 2  Short offline  Completed without error  00%  7821  -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Ericloewe · Aug 18, 2017

That is runs in the background, while the drive continues to service requests normally. Odd nomenclature.

dirkme · Aug 18, 2017

ada3

Code:

root@voyager:~ # smartctl -a /dev/ada3
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:  Toshiba 2.5" HDD MQ01ABD...
Device Model:  TOSHIBA MQ01ABD100
Serial Number:  46CMT294T
LU WWN Device Id: 5 000039 6f1a82f9b
Firmware Version: AX1P2A
User Capacity:  1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  5400 rpm
Form Factor:  2.5 inches
Device is:  In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Fri Aug 18 14:15:17 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)   Offline data collection activity
		   was never started.
		   Auto Offline Data Collection: Disabled.
Self-test execution status:  (  37)   The self-test routine was interrupted
		   by the host with a hard or soft reset.
Total time to complete Offline
data collection:	  (  120) seconds.
Offline data collection
capabilities:		 (0x5b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   No Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  2) minutes.
Extended self-test routine
recommended polling time:	 ( 244) minutes.
SCT capabilities:	 (0x003d)   SCT Status supported.
		   SCT Error Recovery Control supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000b  100  100  050  Pre-fail  Always  -  0
  2 Throughput_Performance  0x0005  100  100  050  Pre-fail  Offline  -  0
  3 Spin_Up_Time  0x0027  100  100  001  Pre-fail  Always  -  1899
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  114
  5 Reallocated_Sector_Ct  0x0033  100  100  050  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000b  100  100  050  Pre-fail  Always  -  0
  8 Seek_Time_Performance  0x0005  100  100  050  Pre-fail  Offline  -  0
  9 Power_On_Hours  0x0032  081  081  000  Old_age  Always  -  7824
 10 Spin_Retry_Count  0x0033  102  100  030  Pre-fail  Always  -  0
 12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  113
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  1
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  18
193 Load_Cycle_Count  0x0032  093  093  000  Old_age  Always  -  78796
194 Temperature_Celsius  0x0022  100  100  000  Old_age  Always  -  33 (Min/Max 20/55)
196 Reallocated_Event_Count 0x0032  100  100  000  Old_age  Always  -  0
197 Current_Pending_Sector  0x0032  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0030  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
220 Disk_Shift  0x0002  100  100  000  Old_age  Always  -  0
222 Loaded_Hours  0x0032  096  096  000  Old_age  Always  -  1832
223 Load_Retry_Count  0x0032  100  100  000  Old_age  Always  -  0
224 Load_Friction  0x0022  100  100  000  Old_age  Always  -  0
226 Load-in_Time  0x0026  100  100  000  Old_age  Always  -  281
240 Head_Flying_Hours  0x0001  100  100  001  Pre-fail  Offline  -  0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description  Status  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline  Interrupted (host reset)  50%  7824  -
# 2  Short offline  Interrupted (host reset)  90%  7823  -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Stux · Aug 18, 2017

Code:

199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 1

This is your error. It normally indicates a cabling problem. Ie, there was an error between the sata port and the hd.

Are you using ECC RAM?

dirkme · Aug 18, 2017

Stux said:
Code:
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 1

This is your error. It normally indicates a cabling problem. Ie, there was an error between the sata port and the hd.

My cable is quite old, I can change it after my days of copying data are done ;-)

Are you using ECC RAM?

Nope, just 16 GB gaming RAM, one day I build a top FreeNAS Box :)

For no there isn't a real disk error, right? and if I change for a better cable, this error might not happen anymore, also right?

dirkme · Aug 18, 2017

Build FreeNAS-11.0-U2 (e417d8aa5)

Platform AMD Phenom(tm) II X4 965 Processor

Memory 16319MB

System Time Fri Aug 18 15:21:58 CDT 2017

Uptime 3:21PM up 20:36, 1 users

Load Average 0.56, 0.63, 0.63

Important Announcement for the TrueNAS Community.

CRITICAL: Aug. 18, 2017, 8:03 a.m. - The volume trust state is ONLINE

Contributor

Active Member

Contributor

Server Wrangler

Active Member

Contributor

Active Member

Contributor

Contributor

Active Member

Active Member

Server Wrangler

Contributor

Active Member

Contributor

Server Wrangler

Contributor

MVP

Contributor

Contributor

Similar threads