SOLVED Fibre channel domains

[Brezlord]

I have 2 single port Qlogic QLE2560 one sitting on each CPU in a Dell T620. I have a LUN setup and mounted on an ESXi 7 but only path is visible by the host, isp.0 domain 0. When I use a 2 port card both paths are visible. I'm wandering if this is because when using to separate cards the system creates a separate domain for each card.

TrueNAS-12.0-U5.1

Code:

root@nas1:~ # sysctl dev.isp.0
dev.isp.0.use_gff_id: 1
dev.isp.0.use_gft_id: 1
dev.isp.0.topo: 3
dev.isp.0.loopstate: 10
dev.isp.0.fwstate: 3
dev.isp.0.linkstate: 1
dev.isp.0.speed: 8
dev.isp.0.role: 1
dev.isp.0.gone_device_time: 30
dev.isp.0.loop_down_limit: 60
dev.isp.0.wwpn: 2377900762149597914
dev.isp.0.wwnn: 2305843168111669978
dev.isp.0.%domain: 0
dev.isp.0.%parent: pci1
dev.isp.0.%pnpinfo: vendor=0x1077 device=0x2532 subvendor=0x1077 subdevice=0x015c class=0x0c0400
dev.isp.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.PEX1.NDX0
dev.isp.0.%driver: isp
dev.isp.0.%desc: Qlogic ISP 2532 PCI FC-AL Adapter
root@nas1:~ # sysctl dev.isp.1
dev.isp.1.use_gff_id: 1
dev.isp.1.use_gft_id: 1
dev.isp.1.topo: 3
dev.isp.1.loopstate: 10
dev.isp.1.fwstate: 3
dev.isp.1.linkstate: 1
dev.isp.1.speed: 8
dev.isp.1.role: 1
dev.isp.1.gone_device_time: 30
dev.isp.1.loop_down_limit: 60
dev.isp.1.wwpn: 2377900762149386453
dev.isp.1.wwnn: 2305843168111458517
dev.isp.1.%domain: 1
dev.isp.1.%parent: pci15
dev.isp.1.%pnpinfo: vendor=0x1077 device=0x2532 subvendor=0x1077 subdevice=0x015c class=0x0c0400
dev.isp.1.%location: slot=0 function=0 dbsf=pci0:66:0:0
dev.isp.1.%driver: isp
dev.isp.1.%desc: Qlogic ISP 2532 PCI FC-AL Adapter

Is there a hint that I can use to make both cards the same domain? Or am i barking up the wrong tree?

 

[HoneyBadger]

Is this a point-to-point setup? FC domain IDs are assigned to switches so it would almost seem like it's assuming a separate domain for some reason.

 

[Brezlord]

I'm using a Cisco DS-C9148-16p-K9 with 2 VSAN's so I have to SANs, SAN-A and SAN-B with one HBA in each.

Code:

SAN-MDS-1(config)# show zone
zone name SAN-A-R720-HOST-01-TrueNAS-01 vsan 100
  fcalias name R720-HOST-01-Slot4-Port1 vsan 100
    pwwn 21:00:00:24:ff:37:0a:80

  fcalias name TrueNAS-01-Slot1-Port1 vsan 100
    pwwn 21:00:00:24:ff:0e:b2:da

zone name SAN-A-R720-HOST-02-TrueNAS-01 vsan 100
  fcalias name R720-HOST-02-Slot4-Port1 vsan 100

  fcalias name TrueNAS-01-Slot1-Port1 vsan 100
    pwwn 21:00:00:24:ff:0e:b2:da

zone name SAN-B-R720-HOST-01-TrueNAS-01 vsan 200
  fcalias name R720-HOST-01-Slot4-Port2 vsan 200
    pwwn 21:00:00:24:ff:37:0a:81

  fcalias name TrueNAS-01-Slot7-Port1 vsan 200
    pwwn 21:00:00:24:ff:0b:78:d5

zone name SAN-B-R720-HOST-02-TrueNAS-01 vsan 200
  fcalias name R720-HOST-02-Slot4-Port2 vsan 200

  fcalias name TrueNAS-01-Slot7-Port1 vsan 200
    pwwn 21:00:00:24:ff:0b:78:d5

 

[Brezlord]

Going direct HBA to HBA works but through the switch it does not. Below is was get printed to the TrueNAS console when you plugin the fibre.

Code:

Oct  8 19:50:30 nas1.#####.com ctlfeasync: WWPN 0x2511547feedf7338 port 0xfffc08 path 10 target 0 arrived
Oct  8 19:51:01 nas1.#####.com ctlfeasync: WWPN 0x2511547feedf7338 port 0xfffc08 path 10 target 0 left
Oct  8 19:51:05 nas1.#####.com isp1: Chan 255 LINK FAILED
Oct  8 19:51:13 nas1.#####.com isp1: isp_handle_notify_24xx: Received INOT with status 2a on VP ff

 

[Brezlord]

All the hardware checks out good and I could not see and errors in the FC zoning so I just deleted all of the zones and recreated them and low and behold everything works now as it should.