Long SMART self test failed [unsupported field in scsi command]

[oldcripple]

I've had this server for a couple of months and its working fine but from the beginning I've gotten the above error on all smartctl long tests.
I've tried everything I could think of, and tried some of the different options on the smartmontools wiki. All drives have this error, short tests return no errors but I'd like to at least know whats causing this if not how to fix it.

Show : smartctl -a /dev/da0

Code:

root@MyNAS:~ # smartctl -a /dev/da0          
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HITACHI
Product:              HUS72302CLAR2000
Revision:             C1D6
Compliance:           SPC-4
User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca01c2b6e00
Serial number:        YGGSWE9D
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Mon Mar  4 01:19:30 2019 EST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     29 C
Drive Trip Temperature:        85 C

Manufactured in week 10 of year 2012
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  21
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2242
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 2719892699955068928

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0   652877         0    652877   47065833     219628.551           0
write:         0  3041096         0   3041096     220326      74384.445           0
verify:        0     6755         0      6755     201970      69394.434           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   53278                 - [-   -    -]
# 2  Background short  Completed                   -   53254                 - [-   -    -]
# 3  Background short  Completed                   -   53186                 - [-   -    -]
# 4  Background short  Completed                   -   53185                 - [-   -    -]
# 5  Background short  Completed                   -   53017                 - [-   -    -]
# 6  Background short  Completed                   -   52849                 - [-   -    -]
# 7  Background short  Completed                   -   52681                 - [-   -    -]
# 8  Background short  Completed                   -   52441                 - [-   -    -]
# 9  Background short  Completed                   -   52283                 - [-   -    -]
#10  Background short  Completed                   -   52115                 - [-   -    -]
#11  Background short  Completed                   -   51947                 - [-   -    -]
#12  Background short  Completed                   -   51707                 - [-   -    -]
#13  Background short  Completed                   -   51539                 - [-   -    -]
#14  Background short  Completed                   -   51371                 - [-   -    -]
#15  Background short  Completed                   -   51203                 - [-   -    -]
#16  Background short  Completed                   -   50987                 - [-   -    -]
#17  Background short  Completed                   -   50789                 - [-   -    -]
#18  Background short  Completed                   -   50621                 - [-   -    -]
#19  Background short  Completed                   -   50453                 - [-   -    -]
#20  Background short  Completed                   -   50435                 - [-   -    -]

Long (extended) Self Test duration: 6 seconds [0.1 minutes]

Show : smartctl -r ioctl -t long /dev/da0

Code:

root@MyNAS:~ # smartctl -r ioctl -t long /dev/da0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

[inquiry: 12 00 00 00 24 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[inquiry: 12 01 00 00 fc 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0xeb
  status=0x0
[inquiry: 12 00 00 00 24 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[request sense: 03 00 00 00 12 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[send diagnostic: 1d 40 00 00 00 00 ]
  CAM status=0x8c, SCSI status=0x2, resid=0x0
  sense_len=0x20, sense_resid=0x0
  status=0x2: sense_key=0x5 asc=0x24 ascq=0x0
Long (extended) offline self test failed [unsupported field in scsi command]

Show : smartctl -x /dev/da0

Code:

root@MyNAS:~ # smartctl -x /dev/da0  
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HITACHI
Product:              HUS72302CLAR2000
Revision:             C1D6
Compliance:           SPC-4
User Capacity:        2,000,398,934,016 bytes [2.00 TB]
Logical block size:   512 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca01c2b6e00
Serial number:        YGGSWE9D
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Mon Mar  4 02:23:36 2019 EST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported
Read Cache is:        Enabled
Writeback Cache is:   Disabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     29 C
Drive Trip Temperature:        85 C

Manufactured in week 10 of year 2012
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  21
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  2242
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 9608429804994953216

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0   652885         0    652885   47065962     219633.978           0
write:         0  3041792         0   3041792     220355      74393.732           0
verify:        0     6755         0      6755     201973      69394.434           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   53278                 - [-   -    -]
# 2  Background short  Completed                   -   53254                 - [-   -    -]
# 3  Background short  Completed                   -   53186                 - [-   -    -]
# 4  Background short  Completed                   -   53185                 - [-   -    -]
# 5  Background short  Completed                   -   53017                 - [-   -    -]
# 6  Background short  Completed                   -   52849                 - [-   -    -]
# 7  Background short  Completed                   -   52681                 - [-   -    -]
# 8  Background short  Completed                   -   52441                 - [-   -    -]
# 9  Background short  Completed                   -   52283                 - [-   -    -]
#10  Background short  Completed                   -   52115                 - [-   -    -]
#11  Background short  Completed                   -   51947                 - [-   -    -]
#12  Background short  Completed                   -   51707                 - [-   -    -]
#13  Background short  Completed                   -   51539                 - [-   -    -]
#14  Background short  Completed                   -   51371                 - [-   -    -]
#15  Background short  Completed                   -   51203                 - [-   -    -]
#16  Background short  Completed                   -   50987                 - [-   -    -]
#17  Background short  Completed                   -   50789                 - [-   -    -]
#18  Background short  Completed                   -   50621                 - [-   -    -]
#19  Background short  Completed                   -   50453                 - [-   -    -]
#20  Background short  Completed                   -   50435                 - [-   -    -]

Long (extended) Self Test duration: 6 seconds [0.1 minutes]

Background scan results log
  Status: waiting until BMS interval timer expires
    Accumulated power on time, hours:minutes 53280:48 [3196848 minutes]
    Number of background scans performed: 313,  scan progress: 0.00%
    Number of background medium scans performed: 313

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 7
  number of phys = 1
  phy identifier = 0
    attached device type: SAS or SATA device
    attached reason: power on
    reason: unknown
    negotiated logical link rate: phy enabled; 6 Gbps
    attached initiator port: ssp=1 stp=1 smp=1
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca01c2b6e01
    attached SAS address = 0x500605b005f80e90
    attached phy identifier = 0
    Invalid DWORD count = 267
    Running disparity error count = 256
    Loss of DWORD synchronization = 67
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 267
     Running disparity error count: 256
     Loss of dword synchronization count: 67
     Phy reset problem count: 0
relative target port id = 2
  generation code = 7
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca01c2b6e02
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

System

Code:

SuperMicro X9DRi-LN4+ 2x, Intel E5-2620
2x LSI9210-8i HBA Controllers
32GB SATADOM FreeNAS 11.2-U2 boot drive
32GB RAM (8x4GB)
12x2TB HDD (2x6HDD RAIDZ2 Pools)

Edited to move specs to body for mobile users

 

[Chris Moore]

Sorry you're having trouble.
A note about the forum first. If someone is looking at the forum from a mobile device like a tablet or phone, as I am this morning, they can't see the signature information. If you have hardware details in the signature, would you mind putting those details in the body of a post?
Here's a link to some guidance on what is helpful to share.

Updated Forum Rules 12/5/18
https://forums.freenas.org/index.php?threads/updated-forum-rules-12-5-18.45124/

I am pretty sure that you have a hardware problem.

 

[Johnny Fartpants]

smartctl -r ioctl -t long /dev/da0

I've not used this command before for long smart tests.

Do you get the same results with: smartctl -t long /dev/da0

 

[Chris Moore]

I've had this server for a couple of months and its working fine but from the beginning I've gotten the above error

If you have gotten an error, it has not been working fine. There is a problem with your configuration. The controller card you are using, LSI9210-8i, can be configured in Initiator Target (IT) mode, but the default configuration is RAID (IR) mode. If it is in IR mode, it needs to be crossflashed to IT mode firmware and you may need to reconfigure your storage pool. The way you have it working may appear to be working, but it should not be giving you errors. The fact that you have errors, indicates that there is a problem. That is why there is a lengthy burn-in process that is supposed to be completed before data is put on the NAS.

 

[oldcripple]

Here's the firmware stuff for the cards both controllers are flashed to IT mode. Like I said, I've combed through the forums of every place I could think of and read all the docs I could think to before I broke down and posted something.

Its not a huge deal if something "destructive" has to occur to fix this, I've really only been goofing around with this thing so far and have yet to put anything of worth on it. I'd just like to solve the mystery for my own peace of mind, if I've got to tear this thing apart to do it then so be it.

sas2flash -list

Code:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2008(B2)   

        Controller Number              : 0
        Controller                     : SAS2008(B2)   
        PCI Address                    : 00:02:00:00
        SAS Address                    : 500605b-0-05f8-0e90
        NVDATA Version (Default)       : 14.01.00.06
        NVDATA Version (Persistent)    : 14.01.00.06
        Firmware Product ID            : 0x2213 (IT)
        Firmware Version               : 20.00.02.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9210-8i
        BIOS Version                   : 07.39.00.00
        UEFI BSD Version               : N/A
        FCODE Version                  : N/A
        Board Name                     : SAS9210-8i
        Board Assembly                 : H3-25329-01C
        Board Tracer Number            : SP24916574

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

sas2flash -listall returns:

Code:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2008(B2)   

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2008(B2)     20.00.02.00    14.01.00.06    07.39.00.00     00:02:00:00
1  SAS2008(B2)     20.00.02.00    14.01.00.06    07.39.00.00     00:83:00:00

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

@johnny
This is the result of: smartctl -t long /dev/da*

Code:

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

Long (extended) offline self test failed [unsupported field in scsi command]

 

[Chris Moore]

before I broke down and posted something.

Please don't think of it that way.
How can we help if you don't ask?

Firmware Version : 20.00.02.00

It is IT mode, but out of date, current version should be: 20.00.07.00
That might make a difference. I know there were a lot of bugs being fixed. Every new firmware was fixing something. For a while, it felt like we were updating firmware every couple months.

There is a guide here that is about crossflashing, which is already done, but it should also give you the information to update the firmware to the latest version:
https://www.ixsystems.com/community...shing-lsi-9211-9300-9311-hba-and-variants.54/

Are all the drives the same model:

Vendor: HITACHI
Product: HUS72302CLAR2000

 

[oldcripple]

All the drives are identical model/firmware. I'll flash the upgrade this evening and let you know what happens.

 

[oldcripple]

Same error, new firmware.

Code:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2008(B2)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2008(B2)     20.00.07.00    14.01.00.06    07.39.00.00     00:02:00:00
1  SAS2008(B2)     20.00.07.00    14.01.00.06    07.39.00.00     00:83:00:00

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

Code:

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

Long (extended) offline self test failed [unsupported field in scsi command]

Code:

root@MyNAS:~ # smartctl -H -r ioctl -t long /dev/da0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org



[inquiry: 12 00 00 00 24 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[inquiry: 12 01 00 00 fc 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0xeb
  status=0x0
[inquiry: 12 00 00 00 24 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
=== START OF READ SMART DATA SECTION ===
[log sense: 4d 00 40 00 00 00 00 00 04 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[log sense: 4d 00 40 00 00 00 00 00 14 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x1
  status=0x0
[log sense: 4d 00 6f 00 00 00 00 00 04 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[log sense: 4d 00 6f 00 00 00 00 00 0c 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[request sense: 03 00 00 00 12 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[log sense: 4d 00 4d 00 00 00 00 00 04 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[log sense: 4d 00 4d 00 00 00 00 00 10 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
SMART Health Status: OK

[request sense: 03 00 00 00 12 00 ]
  CAM status=0x1, SCSI status=0x0, resid=0x0
  status=0x0
[send diagnostic: 1d 40 00 00 00 00 ]
  CAM status=0x8c, SCSI status=0x2, resid=0x0
  sense_len=0x20, sense_resid=0x0
  status=0x2: sense_key=0x5 asc=0x24 ascq=0x0
Long (extended) offline self test failed [unsupported field in scsi command]

Whatever "[send diagnostic: 1d 40 00 00 00 00 ]" is trying to do isnt working.

Edit: Poked around and found some sense, ASC, and CAMstatus keys in the freebsd kernel code: sense_key=0x5 is "illegal request", asc=0x24 is "invalid field in CDB", and CAM status=0x8c is "scsi status error", SCSI Status = 0x2 is "Check Condition"

This means nothing to me, unfortunately.

Edit again: did a dmesg and looked for "mps" based on a similar issue someone had and never got sorted.
It returned:

Code:

mps0: <Avago Technologies (LSI) SAS2008> port 0x8000-0x80ff mem 0xdf600000-0xdf603fff,0xdf580000-0xdf5bffff irq 27 at device 0.0 numa-domain 0 on pci2
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps1: <Avago Technologies (LSI) SAS2008> port 0xf000-0xf0ff mem 0xfbe00000-0xfbe03fff,0xfbd80000-0xfbdbffff irq 64 at device 0.0 numa-domain 1 on pci12
mps1: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>

Someone said a mismatching HBA firmware to freebsd driver version could cause this but I didn't see any v21 firmware on broadcoms site so I dont know if this even applies.

 

[Johnny Fartpants]

Can you tell us more about the chassis and how the drives are connected?

 

[oldcripple]

Its a SuperMicro 12-bay CSE-826 case with a BPN-SAS-826A backplane.

Two cables from BP into one HBA and one cable into the other HBA.

 

[oldcripple]

I've been reading a SCSI command reference manual, seems like the command freenas is sending for a extended background self-test(40h) is being read by my devices as an abort background test command. The result of that is the exact response I'm getting when I look at -r ioctl output for long tests and compare them to successful short tests.

No idea how to fix this...

 

[SweetAndLow]

Why are you using the -r flag?

 

[oldcripple]

I'm trying to figure out whats being sent to the drive that would cause the issue I'm having, -r ioctl is just showing the ioctl requests/responses.

More and more I'm starting to think this is a HDD smartctl capability/firmware issue, the HDDs just don't recognize the command being given for long test for whatever reason, unfortunately all my disks are the same model & firmware rev. so I have no way of testing this theory until I need to upgrade them. There aren't any good sources on these drives SCSI command breakdown and smartmontools

The whole smart model# not matching the label model# should've been a stronger indication to me that the issue was going to be a PITA, apparently these proprietary HGST SAS drives are full of surprises.

It is what it is, I guess. The positive is that this is just a goof around project/media server, so any failures are mostly inconsequential. The negative is that the solution is 'no known/feasible solution'.

 

[CheeryFlame]

I'm trying to figure out whats being sent to the drive that would cause the issue I'm having, -r ioctl is just showing the ioctl requests/responses.

More and more I'm starting to think this is a HDD smartctl capability/firmware issue, the HDDs just don't recognize the command being given for long test for whatever reason, unfortunately all my disks are the same model & firmware rev. so I have no way of testing this theory until I need to upgrade them. There aren't any good sources on these drives SCSI command breakdown and smartmontools

The whole smart model# not matching the label model# should've been a stronger indication to me that the issue was going to be a PITA, apparently these proprietary HGST SAS drives are full of surprises.

It is what it is, I guess. The positive is that this is just a goof around project/media server, so any failures are mostly inconsequential. The negative is that the solution is 'no known/feasible solution'.

I know this is an old thread but I just recently bought 4x HGST SAS drives and added them in my server and they're outputting the same error when trying to do a smartctl long test. Have you ever figured it out? My server and HBA controller aren't the issue since I've been able to successfully perform long tests on my 14 other drives.

 

[RapidScampi]

bump. Same issue.

I've just set up a disk shelf using a Supermicro chassis with the same backplane connected to the same model of HBA. I've got 4 x 8TB Seagate Exynos Enterprise SAS 12Gb/s drives connected and three work fine with one reporting this issue. Freakishly similar to OP. Would love to know if this is a disk issue or something to do with the setup.

 

[CheeryFlame]

Pretty sure this has to do with HGST itself.

 

[isopropyl]

Likely encountering the same issue it seems like if there is any info there that might help

Are 2 of My Drives Failed? (See Edit: Moving Data Onto To New Vdev, To Remove Old)

Normally I wouldn't suggest weekly scrubs, but we are outside of standard procedures. You could try to reach to HGST (or WD?) in order to see if they have a way to fix this but I wouldn't be very optimistic. I suggest you to update the firmware of your HBA anyway. Edit: found this. It has a...

www.truenas.com