After 9.1.0 GUI Upgrade: Bad Message in Logs?

[ewhac]

I have an adorable little HP N54L microserver with 8GiB of RAM on which I originally installed FreeNAS 8.3.1. The machine comes with a 250GB HP drive (rebranded Seacrate), and I added three 2TB Hitachi 5K3000-series drives in a RAID-Z1 configuration.

After upgrading from 8.3.1 to 9.1.0 via the GUI upgrade image, a new message is appearing in the boot logs that I've never seen before:

(aprobe0:ahcich0:0:0:0): SETFEATURES ENABLE SATA FEATURE. ACB: ef 10 00 00 00 40 00 00 00 00 02 00
(aprobe0:ahcich0:0:0:0): CAM status: ATA Status Error
(aprobe0:ahcich0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
(aprobe0:ahcich0:0:0:0): RES: 51 04 00 00 00 40 00 00 00 02 00
(aprobe0:ahcich0:0:0:0): Retrying command
(aprobe0:ahcich0:0:0:0): SETFEATURES ENABLE SATA FEATURE. ACB: ef 10 00 00 00 40 00 00 00 00 02 00
(aprobe0:ahcich0:0:0:0): CAM status: ATA Status Error
(aprobe0:ahcich0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
(aprobe0:ahcich0:0:0:0): RES: 51 04 00 00 00 40 00 00 00 02 00
(aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted

...And then later (curious part bolded)...

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <VB0250EAVER HPG7> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
da0: <SanDisk Cruzer Fit 1.26> Removable Direct Access SCSI-5 device
da0: 40.000MB/s transfers
da0: 3819MB (7821312 512 byte sectors: 255H 63S/T 486C)
da0: quirks=0x2<NO_6_BYTE>
ada1: <Hitachi HDS5C3020ALA632 ML6OAA10> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <Hitachi HDS5C3020ALA632 ML6OAA10> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <Hitachi HDS5C3020ALA632 ML6OAA10> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad10

These messages only appear once at boot, and then everything comes up as before.

Googling for the SETFEATURES message in particular suggests that this is a dire problem. However, to all outward appearances, the machine is operating fine, and no data is lost.

Googling for the "Previously was known as" messages suggests these only appear if you move the drives around behind ZFS's back. However, the hardware was not touched before, during, or after the upgrade to 9.1.0.

The drive throwing the SATA error appears to be the 250GB Seacrate that came with the machine. However, this drive has been acting suspiciously since day one (SMART reports an excessive number of corrected read errors), and it has never earned my trust. As such, it's being treated as a junk drive -- it's currently formatted ZFS, is in a pool all by itself, and contains no important data.

So: What's going on? Is there anything I need to worry about?

 

[dlavigne]

How does "zpool status" look?

 

[ewhac]

Code:

$ zpool status
  pool: faulty
state: ONLINE
status: The pool is formatted using a legacy on-disk format.  The pool can
    still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
    pool will no longer be accessible on software that does not support feature
    flags.
  scan: scrub repaired 0 in 0h13m with 0 errors on Sun Aug  4 05:13:44 2013
config:
 
    NAME                                          STATE    READ WRITE CKSUM
    faulty                                        ONLINE      0    0    0
      gptid/01ac11b2-9b72-11e2-a6df-28924a2d54d5  ONLINE      0    0    0
 
errors: No known data errors
 
  pool: uranus
state: ONLINE
status: The pool is formatted using a legacy on-disk format.  The pool can
    still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
    pool will no longer be accessible on software that does not support feature
    flags.
  scan: scrub repaired 0 in 0h40m with 0 errors on Sun Aug 11 05:40:36 2013
config:
 
    NAME                                            STATE    READ WRITE CKSUM
    uranus                                          ONLINE      0    0    0
      raidz1-0                                      ONLINE      0    0    0
        gptid/404ee341-82fa-11e2-a2c3-28924a2d54d5  ONLINE      0    0    0
        gptid/411c1069-82fa-11e2-a2c3-28924a2d54d5  ONLINE      0    0    0
        gptid/41e4e425-82fa-11e2-a2c3-28924a2d54d5  ONLINE      0    0    0
 
errors: No known data errors

Yes, the first pool's name is 'faulty' -- the suspicious 250GB Seacrate drive.

 

[dlavigne]

zpool and scrub looks good. Seeing that smart is complaining, it wouldn't hurt to keep an eye on that disk and have a backup plan and replacement disk handy.

 

[cyberjock]

Few things in order from top to bottom of the thread.

1. The abort command is a pretty common problem with Seagates(I have 16 of them that have sat in a box for 3 years because I just don't trust them anymore). The ABRT command is supposed to be used by a hard drive when it is having a read error. The problem "seems" to be that Seagates issue ABRT commands without actually trying to perform the read, or some other unknown situation. But whatever the case, it actually didn't have a problem reading the sector, or the error is intermittent. I tend to dismiss the intermittent reason because this issue is widespread for many families and generations of Seagates but not with other companies. Nobody really understands why Seagates are so liberal with issuing ABRT commands for a disk in perfect health that passes every single test you could run on it. So those can kind of be ignored. I say kind of because there's more to it than that. If you get enough ABRT commands many SATA controllers will drop the hard drive from the machine, which will then be redetected a few seconds later(check out the brand new Seagate hard drive.. never mind that it looks like the old Seagate). This can cause havoc with your SATA controller and on rare occasions cause other drives to have their own problems because the SATA controller is deal with the ABRT commands regularly.

Some people have theorized that this deliberate issuing of ABRT commands is to make their desktop drives unsuitable for servers. In my case(and alot of other people's cases) it did accomplish exactly that. There's no evidence that Seagate has actually done that. But lets face it, Seagate would much rather sell you the 3TB enterprise class drive for the much higher price than the desktop equivalent. So I wouldn't put it past them that they either deliberately created the issue or they are choosing to not fix it because many desktop users won't ever care.

Sorry, no RMA for you on this one since Seagate will say that the disk is in perfect health(I know.. my 90 day old hard drives were dropping ABRT commands like crazy and they wouldn't acknowledge there was a problem). About $2000 went down the drain on that mistake. So I had to turn around and replace all 16 drives that were 90 days old with WD drives. I had sworn by Seagates for more than 10 years until this "repurchasing". Now I won't trust Seagate anymore.

2. If a disk is disconnected from the computer and then reattached most hardware I've played with will assign it a new device. This makes sense since you don't want ada4 to suddenly be a different actual disk but still be labeled ada4. So if you have 4 disks and replace ada0 you'll have a new ada4 device created when you plug in the new drive(assuming your hardware support AHCI and you are doing a hotswap). The fact that every one of your disks was a different ada last bootup gives me cause for concern. It could be related to the Seagate or it could be a problem with the SATA controller itself. If the SATA controller is having its own problems drives that are in perfect working order can drop out and come back. Obviously, this isn't a great situation to be in since any data that is "supposed" to be written to a given disk will be lost when the drive is assigned a new device 5-10 seconds later. The system discards the lost data since it has no expectation that the disk will be reattached seconds later. Normally I'd agree with your searches that this is cause to be very concerned. So either you've been very lucky and had no data loss or something is going on that just happens to not interrupt the flow of data to/from the pool.

The fact that your zpool has been scrubbed recently and shows no errors is cause for a sigh of relief. But because of #1 and #2 I'd be a little wary.

#1 is easy to solve. You don't use the disk, so remove it.

#2 is easy to solve too. Buy an additional SATA controller and plug the drives into it if the issue continues. Remember that #2 might be related to #1, so don't buy a new controller until you've tried #1(even if only temporarily).

Personally, since so many people have your exact hardware and aren't having your problems, I'd be a little worried. There's something up with your drives or your hardware. But let's check your drives.

Can you post the output of the following commands in CODE so we can see what your disk health is for all disks?

smartctl -a -q noserial /dev/ada0
smartctl -a -q noserial /dev/ada1
smartctl -a -q noserial /dev/ada2
smartctl -a -q noserial /dev/ada3

On a totally unrelated note, RAIDZ1 isn't very safe. If a disk fails and you have to resilver your pool and another disk has even 1 defect you will have data loss. Drive error rates aren't decreasing as fast as disk sizes are increasing, so many people considered RAID5/RAIDZ1 "dead" all the way back in 2009(link in my sig if you want to read more). So definitely keep religious backups. If your situation is like most other people's you will be having to restore from backup the first time you have a failed disk with RAIDZ1. :/

 

[cyberjock]

I can't edit posts right now(forum issues) but if you do find that you want to purchase a SATA controller, the M1015 reflashed to IT mode is an excellent choice. It's the cheapest solution you are going to find and allows you to add 8 drives to the system(more if you want to use a SAS expander).

The M1015 is used by a lot of people on the forum as well as a lot of ESXi, FreeBSD and Linux users. So its a very safe bet that it will be supported thoroughly for a very long time to come.

 

[ewhac]

As requested:

/dev/ada0 (Seagate). Check out the lovely Raw_Read_Error_Rate:

Code:

$ smartctl -a -q noserial /dev/ada0
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     HP 250GB SATA disk VB0250EAVER
Device Model:     VB0250EAVER
Firmware Version: HPG7
User Capacity:    250,059,350,016 bytes [250 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 2.6, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Aug 24 18:52:00 2013 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (  617) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (  45) minutes.
SCT capabilities:            (0x1039)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   118   100   006    Pre-fail  Always       -       183835903
  3 Spin_Up_Time            0x0023   097   097   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   098   098   020    Old_age   Always       -       2604
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002f   100   253   030    Pre-fail  Always       -       863190
  9 Power_On_Hours          0x0032   095   095   000    Old_age   Always       -       4393
 10 Spin_Retry_Count        0x0033   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       17
180 Unknown_HDD_Attribute   0x002b   100   100   000    Pre-fail  Always       -       730297775
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   097    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   067   059   045    Old_age   Always       -       33 (Min/Max 29/38)
194 Temperature_Celsius     0x0022   033   041   000    Old_age   Always       -       33 (0 18 0 0 0)
195 Hardware_ECC_Recovered  0x003a   044   037   000    Old_age   Always       -       183835903
196 Reallocated_Event_Count 0x0032   100   100   036    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3995         -
# 2  Short offline       Completed without error       00%      3827         -
# 3  Short offline       Completed without error       00%      3755         -
# 4  Short offline       Completed without error       00%      3587         -
# 5  Short offline       Completed without error       00%      3419         -
# 6  Short offline       Completed without error       00%      3251         -
# 7  Short offline       Completed without error       00%      3083         -
# 8  Short offline       Completed without error       00%      3035         -
# 9  Short offline       Completed without error       00%      2867         -
#10  Short offline       Completed without error       00%      2699         -
#11  Short offline       Completed without error       00%      2531         -
#12  Short offline       Completed without error       00%      2363         -
#13  Short offline       Completed without error       00%      2291         -
#14  Short offline       Completed without error       00%      2123         -
#15  Short offline       Completed without error       00%      1955         -
#16  Short offline       Completed without error       00%      1787         -
#17  Short offline       Completed without error       00%      1619         -
#18  Short offline       Completed without error       00%      1571         -
#19  Short offline       Completed without error       00%      1417         -
#20  Short offline       Completed without error       00%      1249         -
#21  Short offline       Completed without error       00%      1081         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

/dev/ada1:

Code:

$ smartctl -a -q noserial /dev/ada1
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Deskstar 5K3000
Device Model:     Hitachi HDS5C3020ALA632
Firmware Version: ML6OAA10
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5940 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Aug 24 18:53:33 2013 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (   28) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 394) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   134   134   054    Pre-fail  Offline      -       101
  3 Spin_Up_Time            0x0007   140   140   024    Pre-fail  Always       -       393 (Average 391)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       1231
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   144   144   020    Pre-fail  Offline      -       30
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       4411
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       9
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1336
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1336
194 Temperature_Celsius     0x0002   176   176   000    Old_age   Always       -       34 (Min/Max 19/43)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      4013         -
# 2  Short offline       Completed without error       00%      3845         -
# 3  Short offline       Completed without error       00%      3773         -
# 4  Short offline       Completed without error       00%      3605         -
# 5  Short offline       Completed without error       00%      3437         -
# 6  Short offline       Completed without error       00%      3269         -
# 7  Short offline       Completed without error       00%      3101         -
# 8  Short offline       Completed without error       00%      3053         -
# 9  Short offline       Completed without error       00%      2885         -
#10  Short offline       Completed without error       00%      2717         -
#11  Short offline       Completed without error       00%      2549         -
#12  Short offline       Completed without error       00%      2381         -
#13  Short offline       Completed without error       00%      2309         -
#14  Short offline       Completed without error       00%      2141         -
#15  Short offline       Completed without error       00%      1973         -
#16  Short offline       Completed without error       00%      1805         -
#17  Short offline       Completed without error       00%      1637         -
#18  Short offline       Completed without error       00%      1589         -
#19  Short offline       Completed without error       00%      1421         -
#20  Short offline       Completed without error       00%      1253         -
#21  Short offline       Completed without error       00%      1085         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

/dev/ada2:

Code:

$ smartctl -a -q noserial /dev/ada2
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Deskstar 5K3000
Device Model:     Hitachi HDS5C3020ALA632
Firmware Version: ML6OAA10
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5940 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Aug 24 18:54:16 2013 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (   28) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 372) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   135   135   054    Pre-fail  Offline      -       98
  3 Spin_Up_Time            0x0007   137   137   024    Pre-fail  Always       -       400 (Average 400)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       876
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   146   146   020    Pre-fail  Offline      -       29
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       4243
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       7
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       877
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       877
194 Temperature_Celsius     0x0002   171   171   000    Old_age   Always       -       35 (Min/Max 19/42)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3846         -
# 2  Short offline       Completed without error       00%      3678         -
# 3  Short offline       Completed without error       00%      3606         -
# 4  Short offline       Completed without error       00%      3438         -
# 5  Short offline       Completed without error       00%      3270         -
# 6  Short offline       Completed without error       00%      3102         -
# 7  Short offline       Completed without error       00%      2934         -
# 8  Short offline       Completed without error       00%      2886         -
# 9  Short offline       Completed without error       00%      2718         -
#10  Short offline       Completed without error       00%      2550         -
#11  Short offline       Completed without error       00%      2382         -
#12  Short offline       Completed without error       00%      2214         -
#13  Short offline       Completed without error       00%      2142         -
#14  Short offline       Completed without error       00%      1974         -
#15  Short offline       Completed without error       00%      1806         -
#16  Short offline       Completed without error       00%      1638         -
#17  Short offline       Completed without error       00%      1470         -
#18  Short offline       Completed without error       00%      1422         -
#19  Short offline       Completed without error       00%      1254         -
#20  Short offline       Completed without error       00%      1086         -
#21  Short offline       Completed without error       00%       918         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

/dev/ada3:

Code:

$ smartctl -a -q noserial /dev/ada3
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi Deskstar 5K3000
Device Model:     Hitachi HDS5C3020ALA632
Firmware Version: ML6OAA10
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5940 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Aug 24 18:54:46 2013 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (   28) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 369) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   134   134   054    Pre-fail  Offline      -       100
  3 Spin_Up_Time            0x0007   139   139   024    Pre-fail  Always       -       394 (Average 394)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       875
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   144   144   020    Pre-fail  Offline      -       30
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       4244
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       7
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       876
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       876
194 Temperature_Celsius     0x0002   171   171   000    Old_age   Always       -       35 (Min/Max 19/42)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      3846         -
# 2  Short offline       Completed without error       00%      3678         -
# 3  Short offline       Completed without error       00%      3606         -
# 4  Short offline       Completed without error       00%      3438         -
# 5  Short offline       Completed without error       00%      3270         -
# 6  Short offline       Completed without error       00%      3102         -
# 7  Short offline       Completed without error       00%      2934         -
# 8  Short offline       Completed without error       00%      2886         -
# 9  Short offline       Completed without error       00%      2718         -
#10  Short offline       Completed without error       00%      2550         -
#11  Short offline       Completed without error       00%      2382         -
#12  Short offline       Completed without error       00%      2214         -
#13  Short offline       Completed without error       00%      2142         -
#14  Short offline       Completed without error       00%      1974         -
#15  Short offline       Completed without error       00%      1806         -
#16  Short offline       Completed without error       00%      1638         -
#17  Short offline       Completed without error       00%      1470         -
#18  Short offline       Completed without error       00%      1422         -
#19  Short offline       Completed without error       00%      1254         -
#20  Short offline       Completed without error       00%      1086         -
#21  Short offline       Completed without error       00%       918         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The fact that the ABRT didn't happen with 8.3.1 makes me wonder what the 9.1-STABLE kernel is doing differently. Is it more paranoid? Is it probing/initializing devices differently?

Won't pulling the Seagate out of the rack cause the remaining drives to get new device names (again)?

I read your post concerning RAIDZ1 being unsafe. However, the HP N54L only has four drive bays (five if you count the empty CD-ROM bay), and near as I can tell RAIDz2 requires a minimum of four drives (which effectively makes it equivalent to mirroring). I'm further given to understand that the next step up would be six drives, and there's no room in this box for that. (Five drives will evidently work, but have performance issues.)

 

[titan_rw]

As requested:

/dev/ada0 (Seagate). Check out the lovely Raw_Read_Error_Rate:

The raw read error rate looks fine to me. Do you understand how those attributes work?

 

[Yatti420]

Run the suspect drive under HDAT2 or similiar (spinrite since its 250gb if you own it - I like it for small drives).. Id' use Hirens.. Hirens 15.1 has hdd regenerator I think.. 15.2 comes with Dr. Revitalize (trial/free) I don't like the new change.. It's got all the hdd manufacturer dos diag tools aswell.. Run the drive a few times under hdat2 then run appropriate dos diag from hirens etc..

** Becareful in HDAT2.. There are destructive "write" tests.. **


That drive id be backing up / disabling soon.. The Hitachis were purchased all together? - look good.. I see the HP has recommended firmware thats good.. ada0 smart value1/value65/value73 on ada0.. The hardware ECC count.. Get rid of that drive.. Always move to IDE / compatible mode for SATA do not have AHCI enabled when running spinrite.. I usually turn it off in hdat2 aswell..

If SMART begins failing RMA if you can?

** Looking at the #s again ada0 looks ok.. I'd still run it stress it check again with dos level tools.. It's a small drive so I don't know if its worth it to keep..

 

[gpsguy]

On a Seagate hard disk, I wouldn't be concerned about it.

/dev/ada0 (Seagate). Check out the lovely Raw_Read_Error_Rate:

Since it looks like you've never run a long test, I'd do one.

smartctl -t long /dev/ada0

It will probably take a few hours to run.

# 1 Short offline Completed without error 00% 3995

cyberjock's mantra is to push RAIDz2, for good reason. We hate to see someone have a drive go bad, replace it, and have another one fail during the resilvering process. If you have sufficient backups and are willing to take that gamble, don't worry about it.

There are some brackets (can't find a link) that would allow you to squeeze 2 - 3.5" drives in the ODD bay. Since that cable and the eSATA connection don't support AHCI, you'd need to install a hacked BIOS version. So, you could do RAIDz2 on your box. You'd need to re-route an e-SATA cable from the back of the unit, to the internal hard disk.

I read your post concerning RAIDZ1 being unsafe. However, the HP N54L only has four drive bays (five if you count the empty CD-ROM bay), ...

BTW, you can put 16Gb of RAM in your N54L. I'm using the RAM listed below, in mine (for ESXi).

Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM DDR3 1333
ECC Unbuffered Server Memory Model KVR1333D3E9SK2/16G

 

[gpsguy]

As an owner of a N40L and N54L, I put my 250's in the closet for spare parts.

It's a small drive so I don't know if its worth it to keep

 

[cyberjock]

I know 9.1 does something differently with how it reads the disks. I can't remember the name, but it's a 9.x thing that can make pools faster than 8.x. Mine is about 30% faster. :)

I watch and see a lot of people lose data in the forums. The 4 most common reasons are usually broken down into a few small categories:
1. Server neglect-as in rarely/never checking its status for months or years, then shocked when it "suddenly" dies. But the truth is that disks had been failing for months and they didn't know because they didn't check(ignorance is bliss, right?)
2. Insufficient redundancy- For example, a RAIDZ1 that has 1 disk fail and when you are trying to resilver a second disk starts through errors.
3. Bad RAM-this one is only when you are using non-ECC RAM.
4. Human error/ignorance-you destroy the pool, offline the good disks and leave bad ones and can't resilver, or add a single disk to a zpool thinking it will add the disk to the vdev but it doesn't; then that disk fails.

About 90% of all of our data loss falls into #2. The other three choices are various small fractions. Some are a combination of #1 and #2(such as a RAIDZ2 that they also neglected). But by a long shot the vast majority of data loss also involved a RAIDZ1 or a stripped array with 5+ disks(idiots...).

So yeah, its not like some users happen to be unlucky, it's that RAIDZ1 really isn't reliable. It's just the freakin' truth whether people want to admit it or not. And you can't play ignorant and expect that things will just turn out for the better.

So now you know why I'm not a fan of those HP microservers except in small server environments. They have limited expandability at the cost of size. Choose your size and choose your fate. They do have the word "micro" in them.. what more do you want?

 

[ewhac]

The raw read error rate looks fine to me. Do you understand how those attributes work?

Apparently not. I've just started reading Seagate's page on the subject. ( http://forums.seagate.com/t5/Deskto...-Read-Error-Rate-and-Hardware-ECC/td-p/122382 )

However, it's probably also worth mentioning that, when I first got the machine, I 'dd'ed the Seagate full of zeros (poor man's surface test), and immediately noticed the drive chattering to itself, as if it was seeking madly across the platter and relocating sectors. So, no, even if the SMART numbers turn out okay, I still don't trust the disk.

Since it looks like you've never run a long test, I'd do one.

smartctl -t long /dev/ada0

In progress...

BTW, you can put 16Gb of RAM in your N54L. I'm using the RAM listed below, in mine (for ESXi).

Kingston 16GB (2 x 8GB) 240-Pin DDR3 SDRAM DDR3 1333
ECC Unbuffered Server Memory Model KVR1333D3E9SK2/16G

I noticed that some people had done that, but since 8GiB should be enough to handle 6GB of spindles, I wasn't prepared to worry about it just yet.

I watch and see a lot of people lose data in the forums. The 4 most common reasons are usually broken down into a few small categories:
1. Server neglect-as in rarely/never checking its status for months or years, then shocked when it "suddenly" dies. But the truth is that disks had been failing for months and they didn't know because they didn't check(ignorance is bliss, right?)

Every day, I look at the status emails it sends me. I actually look forward to the scrub results.

2. Insufficient redundancy- For example, a RAIDZ1 that has 1 disk fail and when you are trying to resilver a second disk starts through errors.

Do drives often fail in groups like that? (I realize that's probably a silly question, since it's clearly happened often enough to people that you actively push RAIDZ2.) I've only ever had that happen once, about ten years ago. It was with a pair of IBM Ultrastar SCSI drives, which died within days of each other. One instant it was fine; the next, thousands of unreadable blocks...

3. Bad RAM-this one is only when you are using non-ECC RAM.

Since this is (intended to be) a server-class machine, I gave it ECC RAM.

4. Human error

I suspect that will strike me long before a failing disk .

So now you know why I'm not a fan of those HP microservers except in small server environments. They have limited expandability at the cost of size. Choose your size and choose your fate. They do have the word "micro" in them.. what more do you want?

That's exactly my use case -- a small server for home use. The fact that it's small and quiet is a nice bonus. I'll keep a closer eye on it, however.

 

[cyberjock]

Do drives often fail in groups like that? (I realize that's probably a silly question, since it's clearly happened often enough to people that you actively push RAIDZ2.) I've only ever had that happen once, about ten years ago. It was with a pair of IBM Ultrastar SCSI drives, which died within days of each other. One instant it was fine; the next, thousands of unreadable blocks...

Well, statistically if you have a disk fail you have something like a 90% chance chance of a second disk failing within 4 hours and something like 40% within 24 hours.

The real issue is that resilvering is hard on disks. They get warmer because they start working real hard. For many people, this starts overheating them. But you are also making the nasty assumption that your disks are pristine. Not going to have any UREs, no bad sectors are going to develop that you don't know about yet, etc. It's kind of like driving 1000 miles and noting the condition of the roads, then a few weeks later going back over that same path and actually expecting that there's no new cracks, no potholes, no construction, etc. It's just not realistic and something will have happened. Same thing with hard drives.

Then you also have to factor in the fact that the hard drives are getting older(so they are more susceptible to failure), your PSU is getting older(so they are more susceptible to not putting out ideal voltage), all of your disks have roughly the same "wear and tear"(so one disk that is worn out is a really good indicator another is coming soon), etc.

That's exactly my use case -- a small server for home use. The fact that it's small and quiet is a nice bonus. I'll keep a closer eye on it, however.

And that's fine. I just don't always understand why people demand that they have a small machine they just "have" to put on their desk. You don't have to put it on your desk. You can shove it in the corner of the room. Once its built and working you won't have to touch it until you need to replace something. Many people probably could go with a larger server and shove it in their basement, spare room, or whatever and had the option for expandability. Instead, they buy those Microservers, then get upset when they didn't really think ahead and 6 months later they have outgrown their server. I prefer to leave myself open to as many upgrade options as possible and look for places to shove the machine that is out of the way.