FreeNAS using disks from a direct attached FC enclosure

[TheSmoker]

Hello,

I am trying to convince FreeNAS into using the disks from an EMC DAE Enclosure, connected through a cable directly the the FreeNAS Server.
FreeNAS config:
- Xeon 4 cores;
- 64G RAM;
- No HDDs;
- 2 SLC USB Flash Drives for boot.
Controller:
- QLogic QLE2464
DAE:
- EMC KTN-STL4 4Gbps FC direct attached enclosure http://www.ebay.com/itm/EMC-EMC2-DE...204590?hash=item1c836f772e:g:IhYAAOSwuLZY1~e3

QLE2564 does see all the disks.

Most of the cases FNAS does not see the disks, actually it sees them sa daX to daY but in the end of the boot process it detaches them, so there will no longer be visible.
After reading the thread: https://forums.freenas.org/index.php?threads/fc-target-support-in-freenas-9-1-0.14139/
1 or 2 times I've managed to make them visible to FNAS (still trying to replicate that) but instead of seeing 4TB disks is sees only 500GB disks?!? Like WTF?!?

Does anyone have any experience with this kind of setup? How should I do it?

Thanks and regards,

The Smoker

 

[mav@]

It could help if you provided any logs showing the error.

 

[TheSmoker]

I will come back with full set of logs in the evening when I get the chance to access the server. Thanks!

 

[TheSmoker]

Hello,

FreeNAS 9.10

Code:

Build FreeNAS-9.10.2-U3 (e1497f269)
Platform Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz
Memory 16241MB

QLE2564

Code:

isp0: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0xc100-0xc1ff mem 0xf7d84000-0xf7  d87fff,0xf7c00000-0xf7cfffff irq 18 at device 0.0 on pci3
isp0: Chan 0 setting role to 0x0
isp1: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0xc000-0xc0ff mem 0xf7d80000-0xf7  d83fff,0xf7b00000-0xf7bfffff irq 19 at device 0.1 on pci3
isp1: Chan 0 setting role to 0x2
pcib4: <PCI-PCI bridge> at device 3.0 on pci2
pci4: <PCI bus> on pcib4
isp2: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0xb100-0xb1ff mem 0xf7a84000-0xf7  a87fff,0xf7900000-0xf79fffff irq 19 at device 0.0 on pci4
isp2: Chan 0 setting role to 0x2
isp3: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0xb000-0xb0ff mem 0xf7a80000-0xf7  a83fff,0xf7800000-0xf78fffff irq 16 at device 0.1 on pci4
isp3: Chan 0 setting role to 0x2

ISP sysctls

Code:

[root@freenas] ~# sysctl -a|grep isp
device  isp
net.isr.dispatch: direct
dev.vgapci.0.%desc: VGA-compatible display
dev.isp.3.topo: 0
dev.isp.3.loopstate: 0
dev.isp.3.fwstate: 0
dev.isp.3.linkstate: 0
dev.isp.3.speed: 0
dev.isp.3.role: 3
dev.isp.3.gone_device_time: 30
dev.isp.3.loop_down_limit: 60
dev.isp.3.wwpn: 2377900762155458239
dev.isp.3.wwnn: 2305843168117530303
dev.isp.3.%parent: pci4
dev.isp.3.%pnpinfo: vendor=0x1077 device=0x2532 subvendor=0x1077 subdevice=0x015e class=0x0c0400
dev.isp.3.%location: slot=0 function=1 dbsf=pci0:4:0:1
dev.isp.3.%driver: isp
dev.isp.3.%desc: Qlogic ISP 2532 PCI FC-AL Adapter
dev.isp.2.topo: 0
dev.isp.2.loopstate: 0
dev.isp.2.fwstate: 0
dev.isp.2.linkstate: 0
dev.isp.2.speed: 0
dev.isp.2.role: 3
dev.isp.2.gone_device_time: 30
dev.isp.2.loop_down_limit: 60
dev.isp.2.wwpn: 2377900762155458238
dev.isp.2.wwnn: 2305843168117530302
dev.isp.2.%parent: pci4
dev.isp.2.%pnpinfo: vendor=0x1077 device=0x2532 subvendor=0x1077 subdevice=0x015e class=0x0c0400
dev.isp.2.%location: slot=0 function=0 dbsf=pci0:4:0:0
dev.isp.2.%driver: isp
dev.isp.2.%desc: Qlogic ISP 2532 PCI FC-AL Adapter
dev.isp.1.topo: 0
dev.isp.1.loopstate: 0
dev.isp.1.fwstate: 0
dev.isp.1.linkstate: 0
dev.isp.1.speed: 0
dev.isp.1.role: 3
dev.isp.1.gone_device_time: 30
dev.isp.1.loop_down_limit: 60
dev.isp.1.wwpn: 2377900762155458237
dev.isp.1.wwnn: 2305843168117530301
dev.isp.1.%parent: pci3
dev.isp.1.%pnpinfo: vendor=0x1077 device=0x2532 subvendor=0x1077 subdevice=0x015e class=0x0c0400
dev.isp.1.%location: slot=0 function=1 dbsf=pci0:3:0:1
dev.isp.1.%driver: isp
dev.isp.1.%desc: Qlogic ISP 2532 PCI FC-AL Adapter
dev.isp.0.topo: 0
dev.isp.0.loopstate: 10
dev.isp.0.fwstate: 3
dev.isp.0.linkstate: 1
dev.isp.0.speed: 4
dev.isp.0.role: 1
dev.isp.0.gone_device_time: 30
dev.isp.0.loop_down_limit: 60
dev.isp.0.wwpn: 2377900762155458236
dev.isp.0.wwnn: 2305843168117530300
dev.isp.0.%parent: pci3
dev.isp.0.%pnpinfo: vendor=0x1077 device=0x2532 subvendor=0x1077 subdevice=0x015e class=0x0c0400
dev.isp.0.%location: slot=0 function=0 dbsf=pci0:3:0:0
dev.isp.0.%driver: isp
dev.isp.0.%desc: Qlogic ISP 2532 PCI FC-AL Adapter
dev.isp.%parent:

As of now dev.isp0.role is 1 (target). Also, no disks are present in FNAS GUI. If I switch it to 2 (initiator) the following thing happens:

Code:

da2 at isp0 bus 0 scbus0 target 0 lun 0
da2: < TOSHIBA FP2A> Fixed Direct Access SPC-2 SCSI device
da2: Serial Number TOSHIBA MG04ACA400A  273DK024FK8A
da2: 400.000MB/s transfers WWNN 0x200000218650884f WWPN 0x220000218650884f PortID 0xce
da2: Command Queueing enabled
da2: 476930MB (961727650 520 byte sectors)
(da2:isp0:0:0:0): READ(10). CDB: 28 00 39 52 cc a1 00 00 01 00
(da2:isp0:0:0:0): CAM status: SCSI Status Error
(da2:isp0:0:0:0): SCSI status: Check Condition
(da2:isp0:0:0:0): SCSI sense: ABORTED COMMAND asc:8,3 (Logical unit communication CRC error (Ultra-DMA/32))
(da2:isp0:0:0:0): Info: 0x3952cca1
(da2:isp0:0:0:0): Field Replaceable Unit: 1
(da2:isp0:0:0:0): Command Specific Info: 0x4184
(da2:isp0:0:0:0): Retrying command (per sense data)
(da2:isp0:0:0:0): READ(10). CDB: 28 00 39 52 cc a1 00 00 01 00
(da2:isp0:0:0:0): CAM status: SCSI Status Error
(da2:isp0:0:0:0): SCSI status: Check Condition
(da2:isp0:0:0:0): SCSI sense: ABORTED COMMAND asc:8,3 (Logical unit communication CRC error (Ultra-DMA/32))
(da2:isp0:0:0:0): Info: 0x3952cca1
(da2:isp0:0:0:0): Field Replaceable Unit: 1
(da2:isp0:0:0:0): Command Specific Info: 0x4184
(da2:isp0:0:0:0): Retrying command (per sense data)
(da2:isp0:0:0:0): READ(10). CDB: 28 00 39 52 cc a1 00 00 01 00
(da2:isp0:0:0:0): CAM status: SCSI Status Error
(da2:isp0:0:0:0): SCSI status: Check Condition
(da2:isp0:0:0:0): SCSI sense: ABORTED COMMAND asc:8,3 (Logical unit communication CRC error (Ultra-DMA/32))
(da2:isp0:0:0:0): Info: 0x3952cca1
(da2:isp0:0:0:0): Field Replaceable Unit: 1
(da2:isp0:0:0:0): Command Specific Info: 0x4184
(da2:isp0:0:0:0): Retrying command (per sense data)
(da2:isp0:0:0:0): READ(10). CDB: 28 00 39 52 cc a1 00 00 01 00
(da2:isp0:0:0:0): CAM status: SCSI Status Error
(da2:isp0:0:0:0): SCSI status: Check Condition
(da2:isp0:0:0:0): SCSI sense: ABORTED COMMAND asc:8,3 (Logical unit communication CRC error (Ultra-DMA/32))
(da2:isp0:0:0:0): Info: 0x3952cca1
(da2:isp0:0:0:0): Field Replaceable Unit: 1
.........
(da2:isp0:0:0:0): CAM status: SCSI Status Error
(da2:isp0:0:0:0): SCSI status: Check Condition
(da2:isp0:0:0:0): SCSI sense: ABORTED COMMAND asc:8,3 (Logical unit communication CRC error (Ultra-DMA/32))
(da2:isp0:0:0:0): Field Replaceable Unit: 1
(da2:isp0:0:0:0): Command Specific Info: 0x4184
(da2:isp0:0:0:0): Error 5, Retries exhausted

The errors are repeating on and on until probably the command times out.
Now the disk is visible:

Code:

[root@freenas] ~# camcontrol devlist
< TOSHIBA FP2A>  at scbus0 target 0 lun 0 (da2,pass6)

Code:

[root@freenas] ~# diskinfo -c da2
da2
  520  # sectorsize
  500098378000  # mediasize in bytes (466G)
  961727650  # mediasize in sectors
  0  # stripesize
  0  # stripeoffset
  59864  # Cylinders according to firmware.
  255  # Heads according to firmware.
  63  # Sectors according to firmware.
  TOSHIBA MG04ACA400A  273DK024FK8AFP2A  # Disk ident.
I/O command overhead:
diskinfo: read: Input/output error

The drive is visible now in GUI but with a size of 500G. The disks are actually SATA disks Toshiba MG04ACA400A 4TB. Also the disks are being correctly identified by the controller.

Any ideas are greatly appreciated!

Regards,

The Smoker

 

[mav@]

I am not sure why it cause exactly those errors, but here I see one clue: "520 byte sectors". ZFS supports only devices with power-of-2 sectors. If it would be SAS/SCSI disks, I would recommend you to reformat them to different sector size with sg_format tool. But how to do it for SATA disks I am not sure.

 

[TheSmoker]

Mav, from your experience those disks should be visible without issues?

 

[mav@]

I don't have experience with 520-byte SATA drives. SAS drives with 520 byte sectors should be correctly detected, but they can't be used with ZFS without being reformatted.