SOLVED FreeNAS 9.3 FC Fibre Channel Target Mode DAS & SAN

[Holt Andrei Tiberiu]

QLE2462 is a 4 GBS card, 400 mb+ is a maximum, so my question is, what is the problem? or you are using 2 x QLE2462 ?

1 Gigabit is 125 megabyte.

On FC you can use 80% of the total bandwith, the rest 20% is rezerved for comunication.

Your values are correct for 1 x 4GBS connection.

 

[mav@]

~400MB/s from 4Gbps FC HBAs is an expected maximal speed due to 8b/10b line coding. See table "Fibre Channel variants" here: https://en.wikipedia.org/wiki/Fibre_Channel

 

[Letni]

Hi Mav,
Thanks again for helping me setup fiber channel storage years ago. I'm back with a quick question.
The QLE2462 4Gb HBAs I have are maxing out on FC at 410.91MB/s read consistently.
Using iSCSI vdev extent with two striped SSDs. (Testing on initiator with Crystal Disk Mark on bare-metal Windows 7)

Created a jail to run Bonnie++ which outputs...
610m on Sequential Output and 945m on Sequential Input

I had a spare mobo so I swapped it out - CPU and RAM power now doubled and I'm getting the same results. Phenom 980 3.7GHz 16GB RAM.

Also seeing someone else online with this HBA model get maxed out at 400MBps too.

Could this be a driver issue or just a bad HBA?

As others have stated, 400 MByte/Second is the MAX of a single FC connection.. Luckily FC is NOT like IP/Ethernet in you can easily "combine" the connection and gain speed directly for the same data channel.. This is known as multi-pathing (ie adding paths for redundancy and speed).

Look into Multipathing drivers on your host (the machine receiving the disks from FreeNAS). If you connect 2 x cables with the same volumes being presented down to the host, and properly multi path, you will easily increase the disk transfer speeds..

In VMware (there is native multipathing software and configuration built in), I use the QLE2462 and plug in both FC ports from FreeNAS to my ESXi. As long as I configure the round-robin (a parameter of the multipathing software) on each volume, I get effectively 800 MByte/Second transfer to each disk.. It scales up (though realistically 4-8 paths is the practical limit)

 

[Holt Andrei Tiberiu]

I confirm what Letni wrote.
Also for 8 GBs, i used it in the past. Achieved over 1400 meg/sec with 2 x 8 GBS QLE Cards.

 

[sfcredfox]

Anyone using FC target mode with version 11.3?

I can't find any specific posts about 11.3 and FC. One would assume that it still supports ISP2432 based cards, but I always like to see if anyone has personal experience.

My system specifically is providing shared storage to ESXi 6.5, so I'm curious if anyone is personally using 11.3 with FC to ESXi 6.5?

 

[Holt Andrei Tiberiu]

It works, tha same way as in the other versions.

I have a few clients using it, i can confirm 11.3u1 also

 

[shistory]

FreeNAS-11.3-U3.2
I can't find Qlogic ISP 2532，What shall I do?


root@freenas[~]# dmesg | grep isp
vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
isp0: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0x6000-0x60ff mem 0xfd2fc000-0xfd2fffff irq 16 at device 0.0 on pci5
isp0: Chan 0 setting role to 0x2
isp1: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0x6400-0x64ff mem 0xfd2f8000-0xfd2fbfff irq 17 at device 0.1 on pci5
isp1: Chan 0 setting role to 0x2

 

[shistory]

FreeNAS-11.3-U3.2
I can't find Qlogic ISP 2532，What shall I do?


root@FreeNAS[~]# dmesg | grep isp
vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
isp0: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0x6000-0x60ff mem 0xfd2fc000-0xfd2fffff irq 16 at device 0.0 on pci5
isp0: Chan 0 setting role to 0x2
isp1: <Qlogic ISP 2532 PCI FC-AL Adapter> port 0x6400-0x64ff mem 0xfd2f8000-0xfd2fbfff irq 17 at device 0.1 on pci5
isp1: Chan 0 setting role to 0x2

@mav@

 

[Holt Andrei Tiberiu]

you should reset the cards in thei'r bios to factory default.
Also do a FW update to the latest Version..

 

[mav@]

I can't find Qlogic ISP 2532，What shall I do?

My crystal ball is not functioning to guess what problem you may have or even what are you trying to achieve without any input.
The role 0x2 means initiator. The recommendation for target is to set role to 0 (disabled) via loader.conf tunable and then enable target mode through the `ctladm` when system is booted. At least first step seems like not done. So please, help everybody and read available materials first.

 

[orion88]

Hi,

I made the configuration from the guide on the TrueNas 12.0-U4 version and I find that everything works as planned.

I shared the disk space with FC cards for hosts running vmware esxi 6.5 and from vcenter I configured the resource shared with FC technology and everything worked fine.

Thanks for the instructions.

Greetings

 

[sfcredfox]

@orion88,
You sir, read my mind. Thanks.

 

[orion88]

Hi,

I made the configuration from the guide on the TrueNas 12.0-U6 version and I find that everything works as planned.

I used an FC QLogic QLE2564 8G card and it works fine.
I shared the disk space with FC cards for hosts running vmware esxi 6.5 and from vcenter I configured the resource shared with FC technology and everything worked fine.
I also plan to test the configuration on Proxmox.

Greetings

 

[sfcredfox]

Hi,

I made the configuration from the guide on the TrueNas 12.0-U6 version and I find that everything works as planned.

I used an FC QLogic QLE2564 8G card and it works fine.
I shared the disk space with FC cards for hosts running vmware esxi 6.5 and from vcenter I configured the resource shared with FC technology and everything worked fine.
I also plan to test the configuration on Proxmox.

Greetings

Has anyone tried the same scenario on TrueNAS scale? I have not come across any guides yet. Tunables turn into something in Debian I assume?

 

[LearnLearnLearn]

I enjoyed FC on freenas and trusnas for many years without problems.
I've moved on to 10GbE instead but I'd love to keep the configuration.
Is there a way to backup just that config?

 

[philxor]

Sorry for bringing up an old thread, but trying to get this working and having some difficulty. I have a QLE 2562 based card in my server with 4 8G ports. I'm using ESXI and passing just one of the ports to a TrueNAS

Most of the output on TrueNAS seems okay and I believe the SAN switches are setup correctly, but I can't see the LUN I've created on another ESX host connected via FCoE.

ctlfe_onoffline: isp0 current WWNN 0x20000024ff780ec0
ctlfe_onoffline: isp0 current WWPN 0x21000024ff780ec0
ctlfe_onoffline: isp0 (path id 3) target role enable succeeded

root@truenas[~]# ctladm portlist -i
Port Online Frontend Name pp vp
0 NO camsim camsim 0 0 naa.5000000d8a27f701
Target: naa.5000000d8a27f700
1 YES ioctl ioctl 0 0
2 YES tpc tpc 0 0
3 YES camtgt isp0 0 0 naa.21000024ff780ec0
Target: naa.20000024ff780ec0
4 YES iscsi iscsi 1 1 iqn.2005-10.org.freenas.ctl:ronfc,t,0x0001
Target: iqn.2005-10.org.freenas.ctl:ronfc

root@truenas[~]# ctladm portlist -l
Port Online Frontend Name pp vp
0 NO camsim camsim 0 0 naa.5000000d8a27f701
All LUNs mapped
1 YES ioctl ioctl 0 0
All LUNs mapped
2 YES tpc tpc 0 0
All LUNs mapped
3 YES camtgt isp0 0 0 naa.21000024ff780ec0
All LUNs mapped
4 YES iscsi iscsi 1 1 iqn.2005-10.org.freenas.ctl:ronfc,t,0x0001
No LUNs mapped

I noticed on the SAN switch (Cisco MDS) the port shows up like this, but thought it would be scsi-fcp:target?
0x790000 N 21:00:00:24:ff:78:0e:c3 (Qlogic) scsi-fcp:init

I can't seem to get into the QLogic BIOS on this UCS server to check any potential settings there.

Any help is appreciated.

 

[Simon Sparks]

@philxor have you followed all the screenshots in my post

SOLVED - FreeNAS 9.3 FC Fibre Channel Target Mode DAS & SAN

And, just so everyone knows, this is wicked fast. I was using iSCSI for so long, I forgot how fast external storage can actually be.

www.truenas.com

You need to add tunables and also an init script.

 

[Simon Sparks]

sorry for hijacking the post a little bit. I have no previous experiences of FC and FC switch. I have did what Aran said in the post but i have no idea what i should do with my brocade switch.
can anyone tell the basic flows/steps ? such as setup zone? and etc
I believe I can look up the cli guides online.
I couldn't use the ezswitch software because some java related issue.

Download the OLD java then use the JAVA control panel to disable your NEWest JAVA version and enable the old one then launch the FC switch management application.

 

[philxor]

@philxor have you followed all the screenshots in my post

SOLVED - FreeNAS 9.3 FC Fibre Channel Target Mode DAS & SAN

And, just so everyone knows, this is wicked fast. I was using iSCSI for so long, I forgot how fast external storage can actually be.

www.truenas.com

You need to add tunables and also an init script.

Hi, yes I've followed all the instructions in the post and that seems to be working okay. Thanks for the detailed post. If I reboot TrueNAS I see the same output via ctladm. If I change the role in the tunables it reflects the change in role on the card. I'm still going through the SAN switch configuration to make sure I'm not missing something and also directly attaching another port on the server to the QLogic HBA to rule that out.

 

[Simon Sparks]

Which switch do you have?

I have the Cisco MDS9148 so I can help with the config of that if that helps you.

In a perfect world you have a zone with contains a SINGLE fibre channel target device and a SINGLE fibre channel initiator.

However it is acceptable to have a zone with contains a SINGLE fibre channel target device and a MULTIPLE fibre channel initiators.

e.g. assuming you have a QLC2694 fibre channel card in your TrueNAS

Zone 1 - TrueNAS - FC WWPort 1 and 4 x ESXi Hosts x 2 FC Ports
Zone 2 - TrueNAS - FC WWPort 2 and 4 x ESXi Hosts x 2 FC Ports
Zone 3 - TrueNAS - FC WWPort 3 and 4 x ESXi Hosts x 2 FC Ports
Zone 4 - TrueNAS - FC WWPort 4 and 4 x ESXi Hosts x 2 FC Ports

therefore each zone will contain 9 entries, 1 FC target and 8 FC initiators.