External Fibre SAN as FN targets

[LearnLearnLearn]

Our main storage is always fibre channel SAN based where almost all storage devices are configured as one large pool/drive and in turn, a device such as BlueArc or OnStor is used to manage and slice up the storage to users and groups via various protocols, most being NFS and CIFS.

I have a number of IBM DS3524 devices that came offline but they are still usable and I've been using FreeNAS for a while now so wondered if I might be able to use FreeNAS as a storage manager.

So basically, rather than having internal storage servers running FreeNAS, I would connect FC storage to one of the FreeNAS boxes and slice it up from there.
Meaning, I would attach storage to FreeNAS then manage it as if all of the external devices were drives in one chassis running FN.

I assume this is possible but I've only been able to find articles talking about USB sticks and not much about SAN devices so thought I would ask here since this is where the pros are :).

 

[HoneyBadger]

Correct me if I'm wrong, but doesn't the IBM DS3500 series have SAS host ports on all of its controllers? A quick search of the Redbooks seems to confirm this, the iSCSI and FC controllers still appear to have 2xSFF-8088 ports on them.

If so, I would just connect those to an external HBA (or multiple HBAs if needed) in the FreeNAS head, and treat the IBM devices as a giant JBOD.

 

[LearnLearnLearn]

It does but I've got it connected to a freenas box using fibre channel. It definitely linked up, I just can't see the storage yet.
Maybe I need to reboot fn.
If seen you are saying that it can be managed by fn then? That's exactly what I was wondering about.

iSCSI requires a daughter card and is not a default option.

 

[HoneyBadger]

You likely need to configure the IBM array itself to see the FC HBA in the FreeNAS system as an authorized initiator, and present LUNs to it.

 

[LearnLearnLearn]

Sorry, I should have added that they do see each other hardware wize, it's only in the GUI that I'm not seeing the storage. That said, the IBM storage is still sliced up as multiple LUNS at the moment so I'll connect another one that has only one large array and see what happens and report back.

 

[LearnLearnLearn]

I can see the link between the storage and the FN server is up but I cannot find a way to import/see the LUN that would be the IBM storage device.
Yes, I think the LUN is mapped/allowed from the IBM because I've set it to default group so no specific ID needed.
I can see the local FC card ports and what appears to be the IBM storage (see #3 below).

# ctladm port -l
Port Online Frontend Name pp vp
0 YES tpc tpc 0 0
1 NO camsim camsim 0 0 naa.5000000f296ddf02
2 YES ioctl ioctl 0 0
3 YES camtgt isp0 0 0 naa.5001438004c37d08 (This seems to be the IBM. At least I can see this ID on the IBM)
4 YES camtgt isp1 0 0 naa.5001438004c37d0a
5 YES iscsi iscsi 257 1 iqn.2005-10.org.freenas.ctl:nas01target,t,0x0101
6 YES iscsi iscsi 258 2 iqn.2005-10.org.freenas.ctl:nas0targetclients,t,0x0102

BTW, I do have an IBM BladeCenter and blades using the freenas storage over fibre channel. I got that to work using iSCSI block which I think was a work around when folks like me were trying to use fibre channel with freenas.

Since we're talking an FC LUN, I assume that's why I cannot see it using the GUI tools or maybe the fact that I'm exporting FC is causing a conflict with using FC based storage as well?

The volume manager does not see the storage.

 

[HoneyBadger]

Since we're talking an FC LUN, I assume that's why I cannot see it using the GUI tools or maybe the fact that I'm exporting FC is causing a conflict with using FC based storage as well?

That's a fairly important distinction; yes, if your port personalities are all set to behave as targets, you won't be able to have them act as initiators to connect to another system. I assume you used the qlogic isp driver hint to put them into initiator mode.

You'll want a second FC HBA to use as your "back end" drive connection to the IBM unit, and then leave them in initiator mode.

Still going to say exporting SAS from the IBM is preferred though.

 

[LearnLearnLearn]

Hi, sorry for not getting back to this.
If I understand your confirmation then, I could use one HBA to connect to external storage which I could then share out using FN.
Since I'm using the single HBA to offer FN as a target to another server, that HBA, even though it has two ports is basically now in target mode.
A second HBA would/could be used as an initiator or a single HBA could be used as one or the other but not both.

 

[HoneyBadger]

It depends on how you've set the hint.isp.N.role tunables ... N will be the portnumber, so if you've only set that for port 0 you should still have port 1 available to act as an initiator, unless it's somehow been defaulted to a target another way.

A separate card would be a good idea, but you'd also want to ensure that your cards are enumerated properly based on slot position - you don't want to have the port numbers change and suddenly be binding to/serving from different FC WWNs.

 

[LearnLearnLearn]

In terms of tunables, this is what I have.
BTW, there are no switches in play, ports are connected directly to storage/servers.

 

[HoneyBadger]

In terms of tunables, this is what I have.
BTW, there are no switches in play, ports are connected directly to storage/servers.

View attachment 34955

You've configured both ports to act as targets. You'll need to make the one connected to the IBM array act as an initiator (or add an extra HBA and not set the hint for the role)

 

[LearnLearnLearn]

What I'm showing you is what I set up a long time ago and never looked back until you mentioned it :).
I've no idea where all those notes have gone. Do you think you could share what I need to do. I've found some fibre channel info related to FN but not all that much that seems consistent, mostly questions and solutions for bits and pieces.

 

[LearnLearnLearn]

Ah, I see there is a lot more FC information these days.

 

[HoneyBadger]

The core config with regards to the tunables should be the same even in the more recent 11.x FC guidelines (although they say to use role: 0 which according to the manpage tells the card to use its own defaults rather than explicitly forcing it to target mode?)

See the manpage for the QLogic isp driver here:

isp(4)

www.freebsd.org

You could try setting your ports to behave as init and target simultaneously by setting hint.isp.N.role: 3 but then you're mixing your "back-end" and "front-end" traffic together. I'd use separate ports (and/or separate HBAs if you want to maintain two paths to each)

 

[LearnLearnLearn]

The card is actually an Emulex 42D0500 but yes, I'll look it up.

 

[LearnLearnLearn]

I've tried changing modes and it seems the HBA itself may have a problem since it never gets light. I think I'll have to order another one then get back to this.

 

[HoneyBadger]

Your signature says it's a QLE2462 though? Unless you mean the card in the IBM unit.

Obviously setting tunables for the qlogic card won't help you since your Emulex will probably be on the ocs_fc driver.

ocs_fc(4)

www.freebsd.org

 

[LearnLearnLearn]

I should update that since I changed it out to an 8GB Emulex which is why I've been going through this to start with.

 

[HoneyBadger]

Honestly, I'd just take the easy route and buy a pair of QLE2562 cards if you're after 8Gb FC. QLogic driver support is more mature under FreeBSD compared to Emulex.

 

[LearnLearnLearn]

I ordered a couple. A replacement Emulex and a Qlogic.