TOE/iSCSI Acceleration Hidden? FreeNAS 11.1 U4

Jason Keller

Explorer
Joined
Apr 2, 2015
Messages
61
Hi guys - it's me again. Was just checking back to see if anyone else had tried playing around with TOE/iSCSI acceleration on the Chelsio cards lately. I am attempting to do some tinkering around with it, however I'm finding that even TOE functionality looks like it's disabled somewhere out of reach now...on a T520-CR. Last I tried it initially was with a T420-CR.

Code:
cxl0: <port 0> on t5nex0
cxl0: Ethernet address: 00:07:43:2b:70:60
cxl0: 16 txq, 8 rxq (NIC)
cxl1: <port 1> on t5nex0
cxl1: Ethernet address: 00:07:43:2b:70:68
cxl1: 16 txq, 8 rxq (NIC)

Code:
dev.t5nex.0.toecaps: 0

Code:
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
		description: connected to   (17)

Code:
options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
		ether 00:07:43:2b:70:60
		hwaddr 00:07:43:2b:70:60
		inet 111.11.1.111 netmask 0xffffff00 broadcast 111.11.1.111
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet 10Gbase-Twinax <full-duplex>
		status: active


I added the 'toe' option in the interface configuration, but no dice, and I haven't found anywhere else to enable it (and it isn't taking effect post reboot). I was also going to look at the full iSCSI offload, but I'm not sure what the option tag is for that (iSCSI?). If anyone has more info on these, I'd appreciate it.
 
Last edited:

Jason Keller

Explorer
Joined
Apr 2, 2015
Messages
61
And just after I post...I realized it might be disabled in a loader tunable...and I was right!
Code:
# Disable Chelsio TCP offload until T580-SO-CR 4x10G is fixed.
hw.cxgbe.toecaps_allowed=0
hw.cxgbe.rdmacaps_allowed=0
hw.cxgbe.iscsicaps_allowed=0


Still don't know if iSCSI PDU offload is ready for testing but...does anyone know how to enable it/use it?
 
Last edited by a moderator:

Jason Keller

Explorer
Joined
Apr 2, 2015
Messages
61
@Jason Keller - Were you able to figure this out? I have similar questions and would like to test these features out myself!

Apologies for the late reply - at that time I did PM @mav@ and I did try it (and did see increased performance that I'm eager for), however attempting this under 11.2-U1 causes CTLD to unceremoniously exit, after which point when it comes back up the config file is reset with the acceleration line removed. I've not been able to figure out yet how to try acceleration again, but plan to attempt to with U2 which just dropped today.
 
Last edited:

someone1

Dabbler
Joined
Jun 17, 2013
Messages
37
Thanks for the reply!

I found some instructions here: https://service.chelsio.com/beta/dr...0.1/Chelsio-UnifiedWire-FreeBSD-UserGuide.pdf

I haven't tested it myself, but here are my findings for 11.2:

  • The following bits were removed in 11.2 -
    Code:
    # Disable Chelsio TCP offload until T580-SO-CR 4x10G is fixed.
    hw.cxgbe.toecaps_allowed=0
    hw.cxgbe.rdmacaps_allowed=0
    hw.cxgbe.iscsicaps_allowed=0
  • t4_tom is part of the default loader config (which loads toecore as well)
  • I believe if_cxgbe is compiled as part of the FreeNAS kernel
    1550590042997.png
  • You can enable DDP and the cxgbei module via tuneables
    1550589461902.png
  • You can add the "toe" option to the NIC options:
    1550590318434.png

The only thing the FreeNAS GUI does not allow is setting the offload driver option for iSCSI portals, so manually modifying the ctl.conf file to add "offload cxgbei" to the portal configuration and restarting ctld is required on every reboot (or through some init script).

1550590378905.png


It'd be great if FreeNAS provided this via the UI as an advanced option when setting up the Portal settings.

I'll update here if I get a chance to test all this out - feel free to share if you get a chance as well!
 

Jason Keller

Explorer
Joined
Apr 2, 2015
Messages
61
So...this ended badly with my T520-CR...

When trying to enable with these directions, I noted that while the listening port opened up as usual on the tcp4 path, no additional connections on the toe4 path nor the tcp4 path for client connections. Just dead silence.

I should see...
Code:
# netstat -an | grep 3260
toe4       0      0 172.31.0.112.3260      172.31.0.2.61892       ESTABLISHED
toe4       0      0 172.31.0.112.3260      172.31.0.1.45779       ESTABLISHED
toe4       0      0 172.31.0.112.3260      172.31.0.3.36359       ESTABLISHED
toe4       0      0 172.31.1.112.3260      172.31.1.3.40218       ESTABLISHED
toe4       0      0 172.31.1.112.3260      172.31.1.4.35431       ESTABLISHED
toe4       0      0 172.31.0.112.3260      172.31.0.4.16030       ESTABLISHED
toe4       0      0 172.31.1.112.3260      172.31.1.2.45068       ESTABLISHED
toe4       0      0 172.31.1.112.3260      172.31.1.1.18614       ESTABLISHED
tcp4       0      0 *.3260                 *.*                    LISTEN


Instead I just see...
Code:
# netstat -an | grep 3260
tcp4       0      0 *.3260                 *.*                    LISTEN



Then I got to looking at ifconfig...

I should see...
Code:
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: connected to   (17)
options=ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>



Instead I see the following...note after LRO how TOE is missing?
Code:
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: connected to   (18)
options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>



I double checked all my sysctls, and they're the same on both boxes, save for a few more cxgbei ones on the latter since the module is loaded.
Code:
# sysctl -a | grep cxgbe
device  cxgbe
kern.icl.cxgbei.recvspace: 1048576
kern.icl.cxgbei.sendspace: 1048576
kern.icl.cxgbei.partial_receive_len: 131072
kern.icl.cxgbei.coalesce: 1
kern.icl.offloads: cxgbei none



Then there's this...
Code:
# kldstat -v | grep -E "toe|*_tom|cxgbe"
                124 cxgbe/vcxgbe
                121 t4nex/cxgbe
 6    1 0xffffffff8283d000 a7d8     cxgbei.ko (/boot/kernel/cxgbei.ko)
                10 cxgbei
                11 icl_cxgbei
 7    2 0xffffffff82848000 1c548    t4_tom.ko (/boot/kernel/t4_tom.ko)
 8    3 0xffffffff82865000 2f68     toecore.ko (/boot/kernel/toecore.ko)
                 8 toecore
24    1 0xffffffff82e1d000 7340     t3_tom.ko (/boot/kernel/t3_tom.ko)
                491 t3_tom



Undoing those additions brings my toe4 paths/caps back and everything is merry. @mav@ ?
 

someone1

Dabbler
Joined
Jun 17, 2013
Messages
37
Hmmmm.... I've gotten up to the last step (adding `offload cxgbei` to ctl.conf) and here's how my ifconfig output looks for my T520-CR:

Code:
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    options=ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:07:43:3a:cb:90
    hwaddr 00:07:43:3a:cb:90
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
    status: active
cxl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
    options=ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:07:43:3a:cb:98
    hwaddr 00:07:43:3a:cb:98
    inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
    status: active


Does adding the option change the capabilities on the cards? Do you have any other tuneables set except for the ones I've listed? I have iSCSI running without offload right now and its running fine - do you know if the iSCSI offload configures with its own IP independent of the NIC IP (I think the cards are capable of it - though not sure how to do it on FreeBSD)?

I may get a chance to try this later this week or sometime next.
 

Jason Keller

Explorer
Joined
Apr 2, 2015
Messages
61
@someone1 Have you rebooted with all these options present? Or did you hot load the module? I have no other tunables present on the system, and I'm even currently doing TOE with these adapters. Curiously I noticed you have DDP enabled, but I don't know that it's necessary (if I read their limited literature correctly, that feature is for placing data into other adapters directly like GPUs, not main memory).
 

Jason Keller

Explorer
Joined
Apr 2, 2015
Messages
61
Just verified - if booted with the cxgbei module loaded, TOE functionality no longer appears in the capability list of the adapters. That is the only change made in the configuration. Any suggestions?
 

someone1

Dabbler
Joined
Jun 17, 2013
Messages
37
That's odd - I think I had these settings during my last reboot though I can't remember. I guess maybe you can have the module load as part of an postinit script instead (which could also add modify ctl.conf and reload ctld for the offload option)?

Ideally FreeNAS makes a UI option when creating portal groups with a "Offload" dropdown populated by the result from `systctl kern.icl.offloads` split by spaces.

I'm waiting for new parts before doing another reboot. I have a semi-production load on my server so I taking it down is somewhat of a PITA. Hopefully they'll arrive within the next week or two - but hopefully we get a response from someone with more knowledge before then!
 

someone1

Dabbler
Joined
Jun 17, 2013
Messages
37
I had a power event and the box rebooted - not ideally how I'd like this to happen but I was able to see that toe did not load correctly. Oddly - there were logs complaining that t4_tom needs to be loaded before trying to enable "toe" on my Chelsio cards though it was already loaded. I manually unloaded cxgbei and t4_tom, then reloaded t4_tom and my cards came back up as expected.

Looking through `dmesg -a`, I see this:

Code:
module_register_init: MOD_LOAD (t4_tom, 0xffffffff82851f20, 0) error 42
...
You must kldload t4_tom.ko before trying to enable TOE on a cxgbe interface.
ifconfig: toe: Resource temporarily unavailable
You must kldload t4_tom.ko before trying to enable TOE on a cxgbe interface.
ifconfig: toe: Resource temporarily unavailable

And a whole bunch of failures due to the lack of my Chelsio adapters coming up with their assigned IP/MTU

Looking into this, `t4_tom` is loaded as part of the rc.conf, which is done after loader.conf, so really `cxgbei` should be added to the kldlist in the rc.conf. I'm not sure how adding this as a tuneable would affect things - do we need to duplicate the line as a tunable? The original line:
Code:
kld_list="dtraceall hwpmc t3_tom t4_tom"


Maybe try removing the loader tunable adding the following one (I'll be testing this on my next SCHEDULED reboot):
Variable = kld_list
Value = toecore t4_tom cxgbei
(Duplicating toecore & t4_tom in case its removed in a future FreeNAS update)
Type: rc.conf


There's still the post-init script that adds the offload option to the ctl.conf and reloading ctld...
 

someone1

Dabbler
Joined
Jun 17, 2013
Messages
37
I actually think the entire line needs to be duplicated, so it should be:

Variable = kld_list
Value = dtraceall hwpmc t3_tom t4_tom cxgbei
Type: rc.conf

I'm still very confused as to how to the offload driver is supposed to work. The instructions don't really specify if the host NIC address needs to be shared with the portal group listening address, and if multiple listening addresses are ok on the portal group. Do you have any experience on this, @Jason Keller ?
 

Jason Keller

Explorer
Joined
Apr 2, 2015
Messages
61
Well that didn't go so well either...so I got the module loaded, TOE was still enabled, everything connected to the correct TOE4 path...all good there. The moment that my vSphere hosts attempted to connect, instant panic/reboot of the FreeNAS unit.

So something majorly broke with this somewhere between 11.1 U4 and 11.2 U2.
 

someone1

Dabbler
Joined
Jun 17, 2013
Messages
37
Would it be worth opening a ticket over? I guess I can try and see if it works on a vanilla FreeBSD 11.2 installation but this may take some time for me to get setup/tested.
 

Jason Keller

Explorer
Joined
Apr 2, 2015
Messages
61
Would it be worth opening a ticket over? I guess I can try and see if it works on a vanilla FreeBSD 11.2 installation but this may take some time for me to get setup/tested.

Haven't the time at the moment, and @mav@ seems to be ignoring all posts regarding this, so I'd say optimistically it's low priority at best for them.
 
Top