Can I use a fibre optic connection to ZFS sync 2 servers?

[tak21]

Dear all, since some time I have set up a mirror system as a "cold" stand by / backup to my main FreeNAS system. The main problem is the 1gbit connection. My last full sync took >36h (complete rebuld of the 2nd system).

Now I had the opportunity to get two 4gbit fibre channel PCIe adapters including a 30m fiber cable - but: I have no clue what to do now... (the cards are QLOGIC QLE2460, flashed to the newest firmware, connected and I have a nice orange light) I started reading and am getting more and more information ... and as more I read as more I am getting confused and by now I fear I can not achive what I really want:
freeNAS main-server (pool@snapshotxyz) --ZFS send-->--(fiber)--> freenas(pool/mainpool)-->zfs receive
The backup-server shall be online once per month and then offline again, the send/receive process will be started manually.

Is it possible to set up a fibre connection to achive the zfs sync? If yes, can I please ask for advice?

Thanks in advance!!!

 

[Nick2253]

There's nothing special about fiber. It's just a different networking interface. So yes, you can sync across the fiber connection.

The challenge for you is going to be setting up the fiber cards. I don't know if FreeBSD (and by extension FreeNAS) supports those cards. You're going to have to do you own due diligence on that point.

If they are supported, you'll have to set up network in FreeNAS. The easiest way would be to pick IPs in a different subnet from your ethernet network. Then you can replicate using the fiber interface IP of the destination computer.

A full step-by-step guide is beyond the scope of what anyone here can provide, but if you run in to specific issues, we can probably provide some advice.

Depending on your hardware, your 4Gb fiber may not make that much of a difference over 1Gb ethernet. It really depends on what your maximum array speed is. Also, your total sync time depends on how much data is changing. The first sync is always going to be the longest. By my calculation, your initial sync moved around 13TB of data. If you have less data change, then you'll only have to replicate those changes, which will drastically speed up the process.

 

[errmatt]

Those are not "network" cards, they are Fiber Channel cards. Fiber channel support is there, but not readily apparent in the gui. I followed this post: https://forums.freenas.org/index.php?threads/fc-target-support-in-freenas-9-1-0.14139/page-2 to get a pair of QLE2462 cards set up to link up my esxi host and freenas. I have a volume presented to the esxi host and have created a data store on it and run several VM's from it with decent performance. I'm not sure the exact setup you would want to use, but I suppose it would involve presenting a LUN from each host to the other and configuring whatever sync you need on each of those.

 

[tak21]

Thank you both. My current unserstanding: with these cards I can not replicate ZFS but I can use them to populate "disc space" to an other mashine where it can be used as if it was local storage. Hmm... this does not solve the original problem but now some more ideas came up. So thank you again (and I am now looking forward for my next holiday :)

 

[Bidule0hm]

I wonder if we can use these fiber cards as raw fd where you can just read/write data like on a normal fd?

 

[HoneyBadger]

As @errmatt states these are FC HBAs, not network interface cards with a fiber transceiver. Native zfs send/receive is impossible as they won't pass IP packets. IP over FC used to be a thing but isn't anymore.

I'm not sure the exact setup you would want to use, but I suppose it would involve presenting a LUN from each host to the other and configuring whatever sync you need on each of those.

You could configure your card in the "primary FreeNAS" machine as an initiator, the card in the "secondary" machine as a target, present raw/device LUNs from secondary to primary, and then mount and use them to build a pool named "remotebackup" or similar. Your primary machine would think that it's doing a local send/recv operation, but it would actually be writing across the glass to the other machine. This strikes me as a particularly ugly hack though.

 

[Darren Myers]

Or get 2 Chelsio T420-CR Cards + SFP+'s an some MM fiber and be done, give each one a static IP and let them talk to each other. Point to Point.