Network loses connectivity when iscsi target connected

[Jacob Fox]

I am testing TrueNAS 12.0 Core Beta2 and I have been attempting to test iscsi connectivity with a Windows 7 virtual machine. I created a 6GB zvol and setup the iscsi extents and target. The windows 7 VM successfully finds the target and connects. However, immediately after connecting, the network interface in TrueNAS that the connection is established on losses connectivity. Web interface, ping, everything. I have attempted to use a second virtual network interface for the purpose of testing as well. In that case, only the interface that the connection is made on loses connectivity. I can see from the UI and command line that the interface thinks it's connected, but no traffic can be seen in or out. 5 secdonds later the following message will be seen on the command line:

"no ping reply (NOP-Out) after 5 seconds" and the name of the iscsi initiator.

The condition can be cleared by virtually unplugging and plugging in the interface from XenServer. Disabling HW Offloading also had no affect. When the interface is "unplugged" in XenServer, the following message is seen on the TrueNAS command line:

gnttab_end_foreign_access_ref: WARNING: g.e. still in use!
gnttab_sen_foreign_access: WARNING: leaking g.e. and page still in use!
gnttab_end_foreign_access_ref: WARNING: g.e. still in use!
gnttab_sen_foreign_access: WARNING: leaking g.e. and page still in use!
gnttab_end_foreign_access_ref: WARNING: g.e. still in use!
gnttab_sen_foreign_access: WARNING: leaking g.e. and page still in use!
gnttab_end_foreign_access_ref: WARNING: g.e. still in use!
gnttab_sen_foreign_access: WARNING: leaking g.e. and page still in use!
gnttab_end_foreign_access_ref: WARNING: g.e. still in use!
gnttab_sen_foreign_access: WARNING: leaking g.e. and page still in use!
gnttab_end_foreign_access_ref: WARNING: g.e. still in use!
gnttab_sen_foreign_access: WARNING: leaking g.e. and page still in use!
gnttab_end_foreign_access_ref: WARNING: g.e. still in use!
gnttab_sen_foreign_access: WARNING: leaking g.e. and page still in use!
gnttab_end_foreign_access_ref: WARNING: g.e. still in use!
gnttab_sen_foreign_access: WARNING: leaking g.e. and page still in use!
xn2: detached

 

[jenksdrummer]

I've seen that with VMWare guests that are configured with an iSCSI LUN that resides on a FreeNAS. I recall we tried a few tunable settings (based on google results) without much success. We ended up dropping the in-guest LUN connection in favor of CIFS as the application supported it. Our use case was a backup repository for an AppAssure VM. FreeNAS 11.x for then; so this isn't a "12" issue, IMO

Sorry no help here, but can confirm that it happens elsewhere.

 

[cods4]

I'm having the same problem and I am also running TrueNAS on xcp-ng.
I had the same sort of problem on BETA1, BETA2 and now RC1.

As soon as I connect from either windows or xcp-ng to an iSCSI extent, I lose all network connectivity on that interface.
And the only way to bring it back up is to unplug and plug the VIF via xcp-ng, or reboot the VM

I'm thinking I might need to buy a new NIC and pass it through to the TrueNAS VM rather than using the xcp-ng network stack.

If anyone has any ideas I'll be keen to try them. I guess I'll be going back to 11.3 for now though.

The screenshot just shows my windows machine pinging Truenas. The connection comes up as soon as I unplug and plug in the VIF, and then it goes down again as soon as I try to connect to the iSCSI drive in windows. Windows shows it is connected to the iSCSI, but it doesnt work at all.

 

[jenksdrummer]

FWIW, this is considered a VMWare problem; something about it not responding properly to a request from the FreeNAS (FreeBSD...) asking if the iSCSI Initiator is still online...VMWare doesn't reply, FreeBSD says it's offline and drops the connection.

At work; this happened often. At home, I've not been able to replicate it. I run a slightly different version of VMWare, and my workloads aren't as high, and my network hardware is different...

Relating to connection drop; not sure if you're using it, but jumbo frames can do some odd things until you have it sorted out from end to end....

Wait...screenshot - are you routing iSCSI traffic?

 

[cods4]

I'm running Truenas on xcp-ng, but I guess the virtual hardware is all very similar between different hypervisors?

I'm not sure if I'm following you, but if it was a issue with whatever is connecting to the iSCSI drive (in my case xcp-ng and windows), then isnt it strange that both xcp-ng and windows are causing the same issue? Or are they both not responding correctly to the request from FreeNAS?

No not routing iSCSI traffic, its just connecting over my local 1gb network.

Edit: And no I'm not using jumbo frames.

 

[cods4]

I didnt find a solution to this, ended up having to install an additional NIC and pass it through to FreeNAS. Now when I upgraded to 12.0 RC1 I dont have any issues when using iscsi.

 

[sbatman]

I'm also experiencing this issue that as soon as any iscsi initiator connects to the iscsi target hosted by Truenas12 (running as a VM on XEN) Truenas network stack appears to fall down. it appears as if all routes are cleared from truenas network stack, manually injecting the default route fixes the issue until the iscsi initiator reconnects again a second or two later.

Ive seen this with a number of initiators
with a clean install of xen and trunas
bonded and non bonded nic's
both ethernet and 10g fibre
wasn't present in the most recent build of freenas

 

[cods4]

I don't have anything more to offer sorry, on my main machine I have installed an additional NIC so I can pass through my 10g nic to the Truenas VM and I use the new NIC for xcp-ng. This seems to be working fine, although it is not an ideal solution.

On my other machine which only has the one NIC, I am still running Freenas 11.3.