LSI 9207-8E -> Dell MD1280, console full of ILLEGAL REQUEST asc:20,0 when accessing the disks

[mije]

Hi,

Got a ESXi 5.5 host where FreeNAS is installed (8 vCPUs / 16GB memory), I'm passing through a LSI 9207-8E controller (the only one MD1280 seems to be supported with), the MD1280 has 42x4TB disks (Toshiba MG04SCA**EN Series).

If I look in /dev, there is 84 daX disks listed (so doesn't seem to recognize the SAS multipathing properly?).

I initially tried them out on a Windows 2016 box with storage spaces where they were are working fine / MPIO worked.

But something seems weird, every time i reboot freenas, it's randomly switching between which 42 disks it shows under Storage -> View Disks in the Web UI, some times it lists da01 -> da42, some times it lists da43 -> da84.

When installing / booting freenas its throwing stuff like (for all the daXX disks):
(da50:mps0:0:65:0): CAM status: SCSI Status Error
(da50:mps0:0:65:0): SCSI status: Check Condition
(da50:mps0:0:65:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
(da50:mps0:0:65:0): Error 22, Unretryable error
(da50:mps0:0:65:0): READ(10). CDB: 28 00 00 00 02 00 00 01 00 00
(da50:mps0:0:65:0): CAM status: SCSI Status Error
(da50:mps0:0:65:0): SCSI status: Check Condition

But it eventually boots up, if i try to wipe the disks from the Web UI it just throws:
Error: Failed to wipe da43: dd: /dev/da43: Operation not permitted

If I try to manually clear it with dd:
[root@freenas] /var/log# dd if=/dev/zero of=/dev/da2 bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes transferred in 9.880610 secs (51819 bytes/sec)

It will clear the first GPT entry, but if I try to remove GPT entry at the end, it just fails (and starts throwing the same SCSI Sense/CAM/Error 22 as above in the console), and dd fails with:
[root@freenas] /var/log# dd if=/dev/zero of=/dev/da2 bs=512 seek=7814035120
dd: /dev/da2: Invalid argument
1+0 records in
0+0 records out
0 bytes transferred in 0.008593 secs (0 bytes/sec)

The controller is running:
[root@freenas] /var/log# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

Adapter Selected is a LSI SAS: SAS2308_2(D1)

Controller Number : 0
Controller : SAS2308_2(D1)
PCI Address : 00:03:00:00
SAS Address : 500605b-0-0b10-f5f0
NVDATA Version (Default) : 14.01.00.06
NVDATA Version (Persistent) : 14.01.00.06
Firmware Product ID : 0x2214 (IT)
Firmware Version : 20.00.07.00
NVDATA Vendor : LSI
NVDATA Product ID : SAS9207-8e
BIOS Version : N/A
UEFI BSD Version : 07.25.01.00
FCODE Version : N/A
Board Name : SAS9207-8e
Board Assembly : 03-25658-00A
Board Tracer Number : SP54933419


The disks are:
[root@freenas] /var/log# diskinfo -v /dev/da2
/dev/da2
512 # sectorsize
4000787030016 # mediasize in bytes (3.6T)
7814037168 # mediasize in sectors
0 # stripesize
0 # stripeoffset
486401 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
47K0A0E9FVNC # Disk ident.
id1,enc@n50050cc11ac0092d/type@0/slot@18 # Physical path

FreeNAS version:
FreeBSD freenas.local 10.3-STABLE FreeBSD 10.3-STABLE #0 r295946+21897e6695f(HEAD): Thu May 25 17:14:15 UTC 2017 root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64

Tried sg_format, to do a low level format, that didn't help either, tried setting sysctl kern.geom.debugflags=0x10 and rerunning dd but still no luck, gpart destroy, complains that the device isnt configured.

Really puzzled what's going on, even tried disconnecting one of the SAS chains to the MD1280, but it still throws the above errors when doing anything with the disks.

Does anyone have any ideas on what to try further? (There is no data stored on it, so that helps a bit at least :))

Maybe the MD1280 is just a bad disk enclousure to use with FreeNAS (couldn't find any mention of anyone successfully using it with FreeNAS at least).

If all else fails I'll throw it back to Storage Spaces where its working, but would really like FreeNAS/ZFS on it instead.

 

[Johnny Fartpants]

I had a very similar issue using the same card with a Dell PowerVault MD3060e. I tried multipathing and singlepathing but it was never happy. I never had these issues with Supermicro, AIC or Quanta kit and after long chats with Dell we couldn't solve the issue. Luckily it was just some test kit.

PS: I have seen another similar scenario on a 45bay Supermicro JBOD and it transpired the internal wiring needed to be different depending on operating systems.

https://forums.freenas.org/index.php?threads/server-unavailable-all-of-the-sudden.35194/

 

[nomad-fr]

Hi, you shoud have a look at this :
http://talesinit.blogspot.fr/2015/11/formatted-with-type-2-protection-huh.html
It's because Dell sell drives formatted with type 2 protection.

You have to install sg3_utils then check with :

Code:

$ sg_readcap -l /dev/da28
Read Capacity results:
Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
Logical block provisioning: lbpme=0, lbprz=0
Last logical block address=15628053167 (0x3a3812aaf), Number of logical blocks=15628053168
Logical block length=512 bytes
Logical blocks per physical block exponent=3 [so physical block length=4096 bytes]
Lowest aligned logical block address=0
Hence:
Device size: 8001563222016 bytes, 7630885.3 MiB, 8001.56 GB
$

a normal format disk should said :

Code:

Protection: prot_en=0, p_type=0, p_i_exponent=0

You have to format the disks low-level with NO protection information.

Code:

sg_format --format --fmtpinfo=0 /dev/yourdevice