Hardware iSCSI offload and FreeNAS?

[Stilez]

In my sdetup, I'm using dual-port Chelsio 10G cards throughout my home lab (Chelsio T420-CR and a few T420-LL-CR's, the low latency version, specifically). FreeNAS 9.10.2U3 - soon to be 11RC - is on its own physical server and shares datasets and zvols on Samba and iSCSI respectively. The main iSCSI initiator is ESXi (ESXi 5.5 but upgrading to ESXi 6.0 or 6.5 if my Quadro 6000 + vSGA can unofficially still work on that version). The FreeNAS iSCSI target hosts the VM store, and the samba shares contain data files used by physical as well as VM LAN clients. The FreeNAS hardware spec is Supermicro + Xeon + 96GB 2400 ECC + Intel P3700 ZIL + 250GB NVMe L2ARC, and the ESXi server has an 8 core Xeon, 96GB ECC, and 250GB NVMe cache (both are specced to handle fast and large volume sync traffic as best I can figure out what's needed).

My question is, at the moment I'm newish to both ESXi and iSCSI, so I'm using ESXi's software iSCSI initiator. The T420 is described as having "Full iSCSI initiator and target mode stack" and "Full HBA offload" so it looks like they can be used to offload iSCSI from the CPU to the NIC on both boxes. If the Chelsio cards need a different driver, or a dedicated iSCSI NIC port or 2nd card for hardware offload, as a dedicated iSCSI target, then that's not a problem, I can do those things.

Is it worth setting up iSCSI offload on my FreeNAS box or should I carry on using the ordinary driver and iSCSI in software as I am presently doing (would I notice much difference and is Chelsio iSCSI offload firmware reliable)? If it's worth switching to hardware iSCSI offload (on just the target, just the initiator, or both), how do I make it happen (in FreeNAS and ESXi) and will it be easy to reverse back to non-offloaded sharing if I have issues?

 

[Stux]

Does FreeNAS/FreeBSD even have support for full iSCSI offload?

 

[Stilez]

According to Chelsio, yes, or at least, yes through their driver: http://service.chelsio.com/beta/dri...0.2/Chelsio-UnifiedWire-FreeBSD-UserGuide.pdf

#1"The adapters [include a protocol handling ASIC and] full offload of a complete Unified Wire solution comprising NIC, TOE, iWARP RDMA, iSCSI, FCoE and NAT support. It scales to true 100Gb line rate operation from a single TCP connection to thousands of connections"

#1.1: "Chelsio Unified Wire driver package for FreeBSD contains . . . Network driver, iWARP driver, iSCSI PDU Offload Target driver, Offload IPv6 driver."

#1.3: "The Chelsio Unified Wire software has been developed to run on FreeBSD based platforms."

Section #3.1 describes technical points within FreeBSD 10/11 and section 3.3 notes the changes to these instructions for the T5 and T4 series adapters (e.g., T420). Chelsio's README for the 3.1.0.0 FreeBSD driver (Feb 2017) confirms that the release package provides all features (including iSCSI target offload + RDMA) to FreeBSD 10.3 + 11 and the T420 is listed as compatible.

The T4 adapters also have native support for offloading SMB ("SMB Direct") using RDMA over ethernet too, for Microsoft but also presumably other supported OS clients: http://www.chelsio.com/chelsio-demo...b-direct-over-ethernet-on-windows-server-2012. (Updated later to "line rate")

Hence my OP questions, given what seems appropriate hardware, is full iSCSI offload (or target offload at least) to ESXi safe for my data and likely to be visibly beneficial, if anyone here uses it, and if so, how do I go about it.

 

[mav@]

FreeBSD does support TCP offload (TOE) and iSCSI offload (which is an iSCSI-specific extension of TOE) with Chelsio NICs. All required parts of that are now present in FreeNAS, but they are not enabled by default yet. While those offload technique indeed may provide performance improvements when it goes about hundreds of thousand IOPS and dozens of gigabits, they may have some side effects and are not deeply tested yet.

 

[Stilez]

FreeBSD does support TCP offload (TOE) and iSCSI offload (which is an iSCSI-specific extension of TOE) with Chelsio NICs. All required parts of that are now present in FreeNAS, but they are not enabled by default yet. While those offload technique indeed may provide performance improvements when it goes about hundreds of thousand IOPS and dozens of gigabits, they may have some side effects and are not deeply tested yet.

When you say "not deeply tested", do you mean just by iX and its users (fringe interest for this community)? Or by the FreeBSD community generally (fringe interest on FreeBSD)? Or by industry overall (reliability not really assumed by the datacentre/IT world at large)?

What sort of usage or testing is needed, and what sort has it had, I mean?

Is anyone out there using (or has tested/trialled) Chelsio's TOE/iSCSI/RDMA/SMB direct offload on FreeNAS (or FreeBSD generally)? What's your assessment?

 

[HoneyBadger]

When you say "not deeply tested", do you mean just by iX and its users (fringe interest for this community)? Or by the FreeBSD community generally (fringe interest on FreeBSD)? Or by industry overall (reliability not really assumed by the datacentre/IT world at large)?

ToE is leveraged where possible in the enterprise world because that can bring your iSCSI-based SAN's CPU overhead down to the point of an FC HBA; it's pretty well validated there.

I can't speak for iX, but from my perspective the CPU is almost never the bottleneck in a ZFS appliance; it's been easier and safer to throw hardware at the problem and worry about ToE existing on the initiator side where the CPU cycles are needed for actual workloads. And since most initiators aren't FreeBSD at the bare-metal level, there's not a lot of testing being done.

 

[Scampicfx]

Regarding this ticket: https://bugs.freenas.org/issues/16473
Are there any plans to make option "toe" applied by default when there is a chelsio network adapter detected? I remember I tested this option "toe" a few months ago and it triggered lots of errors, although chelsio nic is installed. I will update firmware of chelsio cards right now and then test it again!