ZPOOL Error after Scrub

[Mike Bruns]

Hi all,

Could someone help me interpret this zpool error after a scrub? The pool is functional, just giving a corruption error in a non-critical file. It's a new Freenas install.

My config is a: Dell Poweredge T110-ii Server, 16GB ECC RAM, 5x6TB drives, RaidZ2, Current 9.3.1 stable software.

Note: One of the drives shows a smartctl error "in the past" but appears fine now. I'm waiting to get the RMA replacement and will replace the questionable one after burn-in. Is it better to run a RaidZ2 with 1 questionable drive, or remove the questionable drive and run a RaidZ1

Code:

########## ZPool status report summary for all pools ##########

+--------------+--------+------+------+------+----+--------+------+-----+
|Pool Name     |Status  |Read  |Write |Cksum |Used|Scrub   |Scrub |Last |
|              |        |Errors|Errors|Errors|    |Repaired|Errors|Scrub|
|              |        |      |      |      |    |Bytes   |      |Age  |
+--------------+--------+------+------+------+----+--------+------+-----+
|freenas-boot  |ONLINE  |     0|     0|     0|  7%|       0|     0|    1|
|fullvolume   !|ONLINE  |     0|     0|    28| 30%|     84K|     7|    0|
+--------------+--------+------+------+------+----+--------+------+-----+



########## ZPool status report for freenas-boot ##########

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Wed Dec 30 02:24:26 2015
config:

    NAME                                            STATE     READ WRITE CKSUM
    freenas-boot                                    ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/c97814ea-a44e-11e5-b1d0-f8db88ffc155  ONLINE       0     0     0
        gptid/c9a28995-a44e-11e5-b1d0-f8db88ffc155  ONLINE       0     0     0

errors: No known data errors



########## ZPool status report for fullvolume ##########

  pool: fullvolume
state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 84K in 6h47m with 7 errors on Wed Dec 30 08:48:23 2015
config:

    NAME                                            STATE     READ WRITE CKSUM
    fullvolume                                      ONLINE       0     0     7
      raidz2-0                                      ONLINE       0     0    14
        gptid/89d13f2b-aba0-11e5-be4a-f8db88ffc155  ONLINE       0     0     1
        gptid/8a83b206-aba0-11e5-be4a-f8db88ffc155  ONLINE       0     0     0
        gptid/8b271ed3-aba0-11e5-be4a-f8db88ffc155  ONLINE       0     0     1
        gptid/8bd76287-aba0-11e5-be4a-f8db88ffc155  ONLINE       0     0     3
        gptid/8c8582df-aba0-11e5-be4a-f8db88ffc155  ONLINE       0     0     2

errors: Permanent errors have been detected in the following files:

        /var/db/system/cores/python2.7.core


=========================

########## SMART status report summary for all drives ##########

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|Device|Serial         |Temp|Power|Start|Spin |ReAlloc|Current|Offline |UDMA  |Seek  |High  |Command|Last|
|      |               |    |On   |Stop |Retry|Sectors|Pending|Uncorrec|CRC   |Errors|Fly   |Timeout|Test|
|      |               |    |Hours|Count|Count|       |Sectors|Sectors |Errors|      |Writes|Count  |Age |
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+
|ada0 ?|WOL240326574   | 35 |  141|    8|    0|      0|      0|       0|     0|   N/A|   N/A|    N/A|   5|
|ada1 ?|WOL240327198   | 35 |  284|   12|    0|      0|      0|       0|     0|   N/A|   N/A|    N/A|   5|
|ada2 ?|WOL240327200   | 35 |  284|   12|    0|      0|      0|       0|     0|   N/A|   N/A|    N/A|   5|
|ada3 ?|WOL240327207   | 34 |  284|   12|    0|      0|      0|       0|     0|   N/A|   N/A|    N/A|   5|
|ada4 ?|WOL240327210   | 36 |  265|   14|    0|      0|      0|       0|     0|   N/A|   N/A|    N/A|   5|
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+------+------+-------+----+



########## SMART status report for ada0 drive (: WOL240326574) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   201   197   021    Pre-fail  Always       -       8941
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       8
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       141
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       8
16 Unknown_Attribute       0x0022   000   200   000    Old_age   Always       -       17197295533
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       4
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       3
194 Temperature_Celsius     0x0022   117   112   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Extended offline    Completed without error       00%        12         -



########## SMART status report for ada1 drive (: WOL240327198) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   211   207   021    Pre-fail  Always       -       8433
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       12
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       284
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       12
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       63
194 Temperature_Celsius     0x0022   117   111   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Conveyance offline  Completed without error       00%       164         -



########## SMART status report for ada2 drive (: WOL240327200) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   210   206   021    Pre-fail  Always       -       8483
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       12
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       284
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       12
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       61
194 Temperature_Celsius     0x0022   117   109   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Conveyance offline  Completed without error       00%       164         -



########## SMART status report for ada3 drive (: WOL240327207) ##########

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   209   206   021    Pre-fail  Always       -       8508
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       12
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       284
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       12
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       62
194 Temperature_Celsius     0x0022   118   112   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Conveyance offline  Completed without error       00%       164         -



########## SMART status report for ada4 drive (: WOL240327210) ##########

SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   001   051    Pre-fail  Always   In_the_past 0
  3 Spin_Up_Time            0x0027   209   206   021    Pre-fail  Always       -       8533
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       14
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       265
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       10
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       6
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       21
194 Temperature_Celsius     0x0022   116   110   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   198   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

No Errors Logged

Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
Short offline       Completed without error       00%       147         -

 

[Bidule0hm]

Something is wrong and it's definitely not the drives (well, it can, but 4 drives out of 5? yeah, very unlikely...).

How the drives are connected? (to the MB? through a HBA?)

What's the power rating of the PSU?

Is it better to run a RaidZ2 with 1 questionable drive, or remove the questionable drive and run a RaidZ1

Always leave the drive connected unless it hangs the system.

 

[Mike Bruns]

Drives are connected directly to the MB, no HBA involved. The MB has 5 SATA ports, I am using one that was connected to the CDROM for one of the drives. The PSU is 305w. I have an older (but slightly beefier) Dell T310 that I'll be moving to shortly. The CPU is slower, but it has more RAM (32GB) and expandability. Waiting for the burn-in to complete before I move the drives.

The "corrupted" file is interesting:

1. errors: Permanent errors have been detected in the following files:
2. 
   
3. /var/db/system/cores/python2.7.core

Is that file even on the drives?

Code:

[root@agnas1] /var/db/system# df -h
Filesystem                                                     Size    Used   Avail Capacity  Mounted on
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201512121950               13G    529M     12G     4%    /
devfs                                                          1.0k    1.0k      0B   100%    /dev
tmpfs                                                           32M    5.3M     26M    17%    /etc
tmpfs                                                          4.0M    8.0k      4M     0%    /mnt
tmpfs                                                          2.7G    105M    2.6G     4%    /var
freenas-boot/grub                                               12G    6.8M     12G     0%    /boot/grub
fullvolume                                                      10T    227k     10T     0%    /mnt/fullvolume
fullvolume/business                                             11T    644G     10T     6%    /mnt/fullvolume/business
fullvolume/jails                                                10T    220k     10T     0%    /mnt/fullvolume/jails
fullvolume/jails/.warden-template-pluginjail                    10T    548M     10T     0%    /mnt/fullvolume/jails/.warden-template-pluginjail
fullvolume/personal                                             10T     15G     10T     0%    /mnt/fullvolume/personal
fullvolume/personal/mike                                        13T    2.3T     10T    18%    /mnt/fullvolume/personal/mike
fullvolume/personal/pat                                         11T    327G     10T     3%    /mnt/fullvolume/personal/pat
fullvolume/public                                               10T     45G     10T     0%    /mnt/fullvolume/public
fullvolume/public/video                                         12T    1.5T     10T    13%    /mnt/fullvolume/public/video
fullvolume/sort                                                 10T    142G     10T     1%    /mnt/fullvolume/sort
fullvolume/.system                                              10T    184k     10T     0%    /var/db/system
fullvolume/.system/cores                                        10T     30M     10T     0%    /var/db/system/cores
fullvolume/.system/samba4                                       10T    8.0M     10T     0%    /var/db/system/samba4
fullvolume/.system/syslog-5ece5c906a8f4df886779fae5cade8a5      10T     15M     10T     0%    /var/db/system/syslog-5ece5c906a8f4df886779fae5cade8a5
fullvolume/.system/rrd-5ece5c906a8f4df886779fae5cade8a5         10T    170k     10T     0%    /var/db/system/rrd-5ece5c906a8f4df886779fae5cade8a5
fullvolume/.system/configs-5ece5c906a8f4df886779fae5cade8a5     10T    568k     10T     0%    /var/db/system/configs-5ece5c906a8f4df886779fae5cade8a5
fullvolume/jails/plexmediaserver_1                              10T    918M     10T     0%    /mnt/fullvolume/jails/plexmediaserver_1
devfs                                                          1.0k    1.0k      0B   100%    /mnt/fullvolume/jails/plexmediaserver_1/dev
procfs                                                         4.0k    4.0k      0B   100%    /mnt/fullvolume/jails/plexmediaserver_1/proc
/mnt/fullvolume/public                                          10T     45G     10T     0%    /mnt/fullvolume/jails/plexmediaserver_1/media
[root@agnas1] /var/db/system#

/var/db/system/cores/python2.7.core

 

[Mike Bruns]

Always leave the drive connected unless it hangs the system.

Thanks, that was my instinct also.

 

[DrKK]

Let's see

Code:

smartctl -x /dev/{whatever}

output for these drives. Let's evaluate how serious this problem is. It's likely to be only mild, but we'll need a look.

 

[Mike Bruns]

Here's the output for the bad drive:

Code:

[root@agnas1] /var/db/system# smartctl -x /dev/ada4
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WL6000GSA6457
Serial Number:    WOL240327210
LU WWN Device Id: 5 0014ee 0592b84a2
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Dec 30 19:05:53 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                ( 8624) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 740) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   001   051    Past 0
  3 Spin_Up_Time            POS--K   209   206   021    -    8533
  4 Start_Stop_Count        -O--CK   100   100   000    -    14
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   100   253   000    -    0
  9 Power_On_Hours          -O--CK   100   100   000    -    267
10 Spin_Retry_Count        -O--CK   100   253   000    -    0
11 Calibration_Retry_Count -O--CK   100   253   000    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    10
192 Power-Off_Retract_Count -O--CK   200   200   000    -    6
193 Load_Cycle_Count        -O--CK   200   200   000    -    22
194 Temperature_Celsius     -O---K   117   110   000    -    35
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   198   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   100   253   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      40  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 18815 (device log contains only the most recent 24 errors)
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 18815 [22] occurred at disk power-on lifetime: 137 hours (5 days + 17 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 02 97 90 91 00 40 00  Error: IDNF at LBA = 0x297909100 = 11132768512

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 01 00 00 70 00 02 97 90 78 00 40 00     12:00:00.949  WRITE FPDMA QUEUED
  61 01 00 00 68 00 02 97 90 77 00 40 00     12:00:00.949  WRITE FPDMA QUEUED
  61 01 00 00 60 00 02 97 90 76 00 40 00     12:00:00.948  WRITE FPDMA QUEUED
  61 01 00 00 58 00 02 97 90 75 00 40 00     12:00:00.947  WRITE FPDMA QUEUED
  61 01 00 00 50 00 02 97 90 74 00 40 00     12:00:00.947  WRITE FPDMA QUEUED

Error 18814 [21] occurred at disk power-on lifetime: 105 hours (4 days + 9 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 02 97 90 b8 00 40 00  Error: IDNF at LBA = 0x29790b800 = 11132778496

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 01 00 00 c0 00 02 97 90 b8 00 40 08  4d+09:39:25.580  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 b7 00 40 08  4d+09:39:25.579  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 b6 00 40 08  4d+09:39:25.579  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 b5 00 40 08  4d+09:39:25.578  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 b4 00 40 08  4d+09:39:25.577  WRITE FPDMA QUEUED

Error 18813 [20] occurred at disk power-on lifetime: 105 hours (4 days + 9 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 02 97 90 a8 00 40 00  Error: IDNF at LBA = 0x29790a800 = 11132774400

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 01 00 00 c0 00 02 97 90 a8 00 40 08  4d+09:39:18.507  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 a7 00 40 08  4d+09:39:18.506  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 a6 00 40 08  4d+09:39:18.506  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 a5 00 40 08  4d+09:39:18.505  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 a4 00 40 08  4d+09:39:18.504  WRITE FPDMA QUEUED

Error 18812 [19] occurred at disk power-on lifetime: 105 hours (4 days + 9 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 02 97 90 98 00 40 00  Error: IDNF at LBA = 0x297909800 = 11132770304

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 01 00 00 c0 00 02 97 90 98 00 40 08  4d+09:39:11.422  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 97 00 40 08  4d+09:39:11.421  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 96 00 40 08  4d+09:39:11.421  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 95 00 40 08  4d+09:39:11.420  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 94 00 40 08  4d+09:39:11.419  WRITE FPDMA QUEUED

Error 18811 [18] occurred at disk power-on lifetime: 105 hours (4 days + 9 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 02 97 90 88 00 40 00  Error: IDNF at LBA = 0x297908800 = 11132766208

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 01 00 00 c0 00 02 97 90 88 00 40 08  4d+09:39:04.344  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 87 00 40 08  4d+09:39:04.343  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 86 00 40 08  4d+09:39:04.343  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 85 00 40 08  4d+09:39:04.342  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 84 00 40 08  4d+09:39:04.341  WRITE FPDMA QUEUED

Error 18810 [17] occurred at disk power-on lifetime: 105 hours (4 days + 9 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 02 97 90 78 00 40 00  Error: IDNF at LBA = 0x297907800 = 11132762112

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 01 00 00 c0 00 02 97 90 78 00 40 08  4d+09:38:57.268  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 77 00 40 08  4d+09:38:57.268  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 76 00 40 08  4d+09:38:57.267  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 75 00 40 08  4d+09:38:57.267  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 74 00 40 08  4d+09:38:57.266  WRITE FPDMA QUEUED

Error 18809 [16] occurred at disk power-on lifetime: 105 hours (4 days + 9 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 02 97 90 68 00 40 00  Error: IDNF at LBA = 0x297906800 = 11132758016

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 01 00 00 c0 00 02 97 90 68 00 40 08  4d+09:38:50.187  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 67 00 40 08  4d+09:38:50.186  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 66 00 40 08  4d+09:38:50.186  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 65 00 40 08  4d+09:38:50.185  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 64 00 40 08  4d+09:38:50.184  WRITE FPDMA QUEUED

Error 18808 [15] occurred at disk power-on lifetime: 105 hours (4 days + 9 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  10 -- 51 00 00 00 02 97 90 58 00 40 00  Error: IDNF at LBA = 0x297905800 = 11132753920

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  61 01 00 00 c0 00 02 97 90 58 00 40 08  4d+09:38:43.111  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 57 00 40 08  4d+09:38:43.110  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 56 00 40 08  4d+09:38:43.110  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 55 00 40 08  4d+09:38:43.104  WRITE FPDMA QUEUED
  61 01 00 00 c0 00 02 97 90 54 00 40 08  4d+09:38:43.104  WRITE FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       147         -
# 2  Short offline       Completed: unknown failure    90%       144         -
# 3  Short offline       Completed: unknown failure    90%       125         -
# 4  Extended offline    Completed: unknown failure    90%       121         -
# 5  Conveyance offline  Completed: unknown failure    90%       121         -
# 6  Short offline       Completed: unknown failure    90%       121         -
# 7  Conveyance offline  Completed without error       00%         0         -
# 8  Short offline       Completed without error       00%         0         -
# 9  Short offline       Completed without error       00%         0         -
#10  Conveyance offline  Interrupted (host reset)      90%         0         -
#11  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    35 Celsius
Power Cycle Min/Max Temperature:     33/41 Celsius
Lifetime    Min/Max Temperature:     17/42 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (99)

Index    Estimated Time   Temperature Celsius
100    2015-12-30 11:08    37  ******************
...    ..( 75 skipped).    ..  ******************
176    2015-12-30 12:24    37  ******************
177    2015-12-30 12:25    36  *****************
...    ..( 40 skipped).    ..  *****************
218    2015-12-30 13:06    36  *****************
219    2015-12-30 13:07    37  ******************
...    ..( 16 skipped).    ..  ******************
236    2015-12-30 13:24    37  ******************
237    2015-12-30 13:25    38  *******************
...    ..( 20 skipped).    ..  *******************
258    2015-12-30 13:46    38  *******************
259    2015-12-30 13:47    39  ********************
...    ..( 76 skipped).    ..  ********************
336    2015-12-30 15:04    39  ********************
337    2015-12-30 15:05    38  *******************
...    ..(125 skipped).    ..  *******************
463    2015-12-30 17:11    38  *******************
464    2015-12-30 17:12    37  ******************
...    ..( 19 skipped).    ..  ******************
   6    2015-12-30 17:32    37  ******************
   7    2015-12-30 17:33    36  *****************
...    ..( 69 skipped).    ..  *****************
  77    2015-12-30 18:43    36  *****************
  78    2015-12-30 18:44    35  ****************
...    ..( 20 skipped).    ..  ****************
  99    2015-12-30 19:05    35  ****************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            8  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            9  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       220849  Vendor specific

The same model good drive:

Code:

[root@agnas1] /var/db/system# smartctl -x /dev/ada2
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WL6000GSA6457
Serial Number:    WOL240327200
LU WWN Device Id: 5 0014ee 0ae8b62ec
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Dec 30 19:13:53 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                ( 7964) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 733) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   210   206   021    -    8483
  4 Start_Stop_Count        -O--CK   100   100   000    -    12
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   100   253   000    -    0
  9 Power_On_Hours          -O--CK   100   100   000    -    286
10 Spin_Retry_Count        -O--CK   100   253   000    -    0
11 Calibration_Retry_Count -O--CK   100   253   000    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    12
192 Power-Off_Retract_Count -O--CK   200   200   000    -    7
193 Load_Cycle_Count        -O--CK   200   200   000    -    61
194 Temperature_Celsius     -O---K   118   109   000    -    34
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      40  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%       164         -
# 2  Extended offline    Completed without error       00%       116         -
# 3  Conveyance offline  Completed without error       00%         0         -
# 4  Short offline       Completed without error       00%         0         -
# 5  Short offline       Completed without error       00%         0         -
# 6  Conveyance offline  Interrupted (host reset)      90%         0         -
# 7  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    34 Celsius
Power Cycle Min/Max Temperature:     32/39 Celsius
Lifetime    Min/Max Temperature:     19/43 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (450)

Index    Estimated Time   Temperature Celsius
451    2015-12-30 11:16    36  *****************
...    ..( 41 skipped).    ..  *****************
  15    2015-12-30 11:58    36  *****************
  16    2015-12-30 11:59    35  ****************
...    ..( 71 skipped).    ..  ****************
  88    2015-12-30 13:11    35  ****************
  89    2015-12-30 13:12    36  *****************
...    ..( 23 skipped).    ..  *****************
113    2015-12-30 13:36    36  *****************
114    2015-12-30 13:37    37  ******************
...    ..( 86 skipped).    ..  ******************
201    2015-12-30 15:04    37  ******************
202    2015-12-30 15:05    36  *****************
...    ..(122 skipped).    ..  *****************
325    2015-12-30 17:08    36  *****************
326    2015-12-30 17:09    35  ****************
...    ..( 42 skipped).    ..  ****************
369    2015-12-30 17:52    35  ****************
370    2015-12-30 17:53    34  ***************
...    ..( 79 skipped).    ..  ***************
450    2015-12-30 19:13    34  ***************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            8  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            9  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       221330  Vendor specific

[root@agnas1] /var/db/system#

 

[Mike Bruns]

Just wanted to update the thread:

I moved the drives to another server (Dell T-310, 32GB ECC Ram), deleted the bad file, then ran a full scrub. The scrub came back clean, but didn't remove the error message.

I ran a "zpool scrub" again, then a "zpool scrub -s" immediately to cancel it and it cleared the error message. Ran another scrub for good measure and it also came back clean.

I'm still going to replace the suspected-bad drive once I get the RMA replacement and burn-in the new drive with the badblocks burn-in script. But things are (and have been) very stable

Thanks all!