Hi,
Got a ESXi 5.5 host where FreeNAS is installed (8 vCPUs / 16GB memory), I'm passing through a LSI 9207-8E controller (the only one MD1280 seems to be supported with), the MD1280 has 42x4TB disks (Toshiba MG04SCA**EN Series).
If I look in /dev, there is 84 daX disks listed (so doesn't seem to recognize the SAS multipathing properly?).
I initially tried them out on a Windows 2016 box with storage spaces where they were are working fine / MPIO worked.
But something seems weird, every time i reboot freenas, it's randomly switching between which 42 disks it shows under Storage -> View Disks in the Web UI, some times it lists da01 -> da42, some times it lists da43 -> da84.
When installing / booting freenas its throwing stuff like (for all the daXX disks):
(da50:mps0:0:65:0): CAM status: SCSI Status Error
(da50:mps0:0:65:0): SCSI status: Check Condition
(da50:mps0:0:65:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
(da50:mps0:0:65:0): Error 22, Unretryable error
(da50:mps0:0:65:0): READ(10). CDB: 28 00 00 00 02 00 00 01 00 00
(da50:mps0:0:65:0): CAM status: SCSI Status Error
(da50:mps0:0:65:0): SCSI status: Check Condition
But it eventually boots up, if i try to wipe the disks from the Web UI it just throws:
Error: Failed to wipe da43: dd: /dev/da43: Operation not permitted
If I try to manually clear it with dd:
[root@freenas] /var/log# dd if=/dev/zero of=/dev/da2 bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes transferred in 9.880610 secs (51819 bytes/sec)
It will clear the first GPT entry, but if I try to remove GPT entry at the end, it just fails (and starts throwing the same SCSI Sense/CAM/Error 22 as above in the console), and dd fails with:
[root@freenas] /var/log# dd if=/dev/zero of=/dev/da2 bs=512 seek=7814035120
dd: /dev/da2: Invalid argument
1+0 records in
0+0 records out
0 bytes transferred in 0.008593 secs (0 bytes/sec)
The controller is running:
[root@freenas] /var/log# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2308_2(D1)
Controller Number : 0
Controller : SAS2308_2(D1)
PCI Address : 00:03:00:00
SAS Address : 500605b-0-0b10-f5f0
NVDATA Version (Default) : 14.01.00.06
NVDATA Version (Persistent) : 14.01.00.06
Firmware Product ID : 0x2214 (IT)
Firmware Version : 20.00.07.00
NVDATA Vendor : LSI
NVDATA Product ID : SAS9207-8e
BIOS Version : N/A
UEFI BSD Version : 07.25.01.00
FCODE Version : N/A
Board Name : SAS9207-8e
Board Assembly : 03-25658-00A
Board Tracer Number : SP54933419
The disks are:
[root@freenas] /var/log# diskinfo -v /dev/da2
/dev/da2
512 # sectorsize
4000787030016 # mediasize in bytes (3.6T)
7814037168 # mediasize in sectors
0 # stripesize
0 # stripeoffset
486401 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
47K0A0E9FVNC # Disk ident.
id1,enc@n50050cc11ac0092d/type@0/slot@18 # Physical path
FreeNAS version:
FreeBSD freenas.local 10.3-STABLE FreeBSD 10.3-STABLE #0 r295946+21897e6695f(HEAD): Thu May 25 17:14:15 UTC 2017 root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64
Tried sg_format, to do a low level format, that didn't help either, tried setting sysctl kern.geom.debugflags=0x10 and rerunning dd but still no luck, gpart destroy, complains that the device isnt configured.
Really puzzled what's going on, even tried disconnecting one of the SAS chains to the MD1280, but it still throws the above errors when doing anything with the disks.
Does anyone have any ideas on what to try further? (There is no data stored on it, so that helps a bit at least :))
Maybe the MD1280 is just a bad disk enclousure to use with FreeNAS (couldn't find any mention of anyone successfully using it with FreeNAS at least).
If all else fails I'll throw it back to Storage Spaces where its working, but would really like FreeNAS/ZFS on it instead.
Got a ESXi 5.5 host where FreeNAS is installed (8 vCPUs / 16GB memory), I'm passing through a LSI 9207-8E controller (the only one MD1280 seems to be supported with), the MD1280 has 42x4TB disks (Toshiba MG04SCA**EN Series).
If I look in /dev, there is 84 daX disks listed (so doesn't seem to recognize the SAS multipathing properly?).
I initially tried them out on a Windows 2016 box with storage spaces where they were are working fine / MPIO worked.
But something seems weird, every time i reboot freenas, it's randomly switching between which 42 disks it shows under Storage -> View Disks in the Web UI, some times it lists da01 -> da42, some times it lists da43 -> da84.
When installing / booting freenas its throwing stuff like (for all the daXX disks):
(da50:mps0:0:65:0): CAM status: SCSI Status Error
(da50:mps0:0:65:0): SCSI status: Check Condition
(da50:mps0:0:65:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
(da50:mps0:0:65:0): Error 22, Unretryable error
(da50:mps0:0:65:0): READ(10). CDB: 28 00 00 00 02 00 00 01 00 00
(da50:mps0:0:65:0): CAM status: SCSI Status Error
(da50:mps0:0:65:0): SCSI status: Check Condition
But it eventually boots up, if i try to wipe the disks from the Web UI it just throws:
Error: Failed to wipe da43: dd: /dev/da43: Operation not permitted
If I try to manually clear it with dd:
[root@freenas] /var/log# dd if=/dev/zero of=/dev/da2 bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes transferred in 9.880610 secs (51819 bytes/sec)
It will clear the first GPT entry, but if I try to remove GPT entry at the end, it just fails (and starts throwing the same SCSI Sense/CAM/Error 22 as above in the console), and dd fails with:
[root@freenas] /var/log# dd if=/dev/zero of=/dev/da2 bs=512 seek=7814035120
dd: /dev/da2: Invalid argument
1+0 records in
0+0 records out
0 bytes transferred in 0.008593 secs (0 bytes/sec)
The controller is running:
[root@freenas] /var/log# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved
Adapter Selected is a LSI SAS: SAS2308_2(D1)
Controller Number : 0
Controller : SAS2308_2(D1)
PCI Address : 00:03:00:00
SAS Address : 500605b-0-0b10-f5f0
NVDATA Version (Default) : 14.01.00.06
NVDATA Version (Persistent) : 14.01.00.06
Firmware Product ID : 0x2214 (IT)
Firmware Version : 20.00.07.00
NVDATA Vendor : LSI
NVDATA Product ID : SAS9207-8e
BIOS Version : N/A
UEFI BSD Version : 07.25.01.00
FCODE Version : N/A
Board Name : SAS9207-8e
Board Assembly : 03-25658-00A
Board Tracer Number : SP54933419
The disks are:
[root@freenas] /var/log# diskinfo -v /dev/da2
/dev/da2
512 # sectorsize
4000787030016 # mediasize in bytes (3.6T)
7814037168 # mediasize in sectors
0 # stripesize
0 # stripeoffset
486401 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
47K0A0E9FVNC # Disk ident.
id1,enc@n50050cc11ac0092d/type@0/slot@18 # Physical path
FreeNAS version:
FreeBSD freenas.local 10.3-STABLE FreeBSD 10.3-STABLE #0 r295946+21897e6695f(HEAD): Thu May 25 17:14:15 UTC 2017 root@gauntlet:/freenas-9.10-releng/_BE/objs/freenas-9.10-releng/_BE/os/sys/FreeNAS.amd64 amd64
Tried sg_format, to do a low level format, that didn't help either, tried setting sysctl kern.geom.debugflags=0x10 and rerunning dd but still no luck, gpart destroy, complains that the device isnt configured.
Really puzzled what's going on, even tried disconnecting one of the SAS chains to the MD1280, but it still throws the above errors when doing anything with the disks.
Does anyone have any ideas on what to try further? (There is no data stored on it, so that helps a bit at least :))
Maybe the MD1280 is just a bad disk enclousure to use with FreeNAS (couldn't find any mention of anyone successfully using it with FreeNAS at least).
If all else fails I'll throw it back to Storage Spaces where its working, but would really like FreeNAS/ZFS on it instead.