TrueNAS 13.0-U1 WireGuard jail - network traffic stops after some time

[Cellobita]

After setting up two TrueNAS 12-U8.1 WireGuard jails successfully, using the step-by-step guide found on this forum, I have attempted the exact same procedure on a TrueNAS 13-RELEASE system, with a FreeBSD 13 based jail, and it doesn't work; I have checked and rechecked everything (it's basically a copy and paste procedure, after all), to no avail.

What I have noticed is that the

ipfw nat show config

command returns a blank response on the 13.0-RELEASE jail (instead of the correct line I get on the 12.3-RELEASE one). In fact, I have to flush the firewall rules to be able to ping anything outside the jail, either by name or by IP, so it seems that the issue is with the NAT part of the ipfw config on 13-RELEASE.

Anyone has ideas on this?

 

[Cellobita]

Upon further investigation, I have tentatively narrowed the problem down to the following ipfw command

ipfw -q nat 1 config if epair0b same_ports unreg_only reset

Either via the config script or input manually, I get the following error:

ipfw: setsockopt(IP_FW_NAT44_XCONFIG): Invalid argument

Apparently, there are issues with this on FreeBSD 13: https://forums.freebsd.org/threads/...rade-from-12-2-release-to-13-0-release.81813/

Edit: FWIW, I have created another jail inside TN13 but using 12.3-RELEASE as the underlying OS, and everything is working as it should. For now, I'm convinced that there is a bug with ipfw NAT support on FreeBSD 13.0. When the OS gets a version bump, I'll revisit the issue.

 

[Volts]

I think you're right.

For now, copy /sbin/ipfw from the host into the jail.

And open a ticket.

 

[Volts]

It looks like a 13.1-RELEASE jail Release is now available.
I spun up a test jail, and /sbin/ipfw behaves with your NAT commands.

 

[Cellobita]

It looks like a 13.1-RELEASE jail Release is now available.
I spun up a test jail, and /sbin/ipfw behaves with your NAT commands.

I can confirm that the instructions on the original link work with a 13.1-RELEASE jail.

It is now quite obvious that there was a bug on 13.1-RC6 - thankfully, it got sorted out. My WireGuard 13.1-RELEASE jail is now working, and will be released to my users tomorrow.

Thank you for the heads up!

 

[Volts]

was a bug on 13.1-RC6

I don't think this was a bug, per se. Certainly not a FreeBSD 13.1-RC6 bug.

Old /sbin userland components may not always be upward-compatible.

Glad it's working!

 

[Cellobita]

I don't think this was a bug, per se. Certainly not a FreeBSD 13.1-RC6 bug.

Old /sbin userland components may not always be upward-compatible.

Glad it's working!

Anyway, it did not work with 13.1-RC6, and it works with 13.1-RELEASE. "There is balance in the Force again..." (sorry, probably shouldn't be posting after a couple drams of Buffalo Trace)

 

[Volts]

Maybe I'm confused!

13.1-RC6 what?

 

[Cellobita]

Maybe I'm confused!

13.1-RC6 what?

Sorry - 13.1-RC6 is the underlying OS version for TrueNAS 13-RELEASE; the problematic version for my jail was actually 13.0-RELEASE-p11.

The OS versions that work for the WireGuard jail are 12.3 and 13.1.

P.S.: I have had two or three hard reboots while restarting this jail on TN 13 [system is lightly used, and there are no other jails, VM's or plugins running] - I'll investigate this further.

P.S.2: The WireGuard server at the jail stopped responding after a few hours [the jail itself was still up] - the TrueNAS host was operational the whole time, with no other issues. I'm holding back deployment for now, waiting for -U1.

 

[Cellobita]

Upgraded today, same results; on my system, currently running the just-released TrueNAS Core 13.0-U1, the situation is as follows:

12.3-p5 based jail: WireGuard keeps running, no issues whatsoever [edit: bummer, connection just went down, after a bit over three hours... the issue seems related to the host OS, then ]

13.1-p0 based jail: WireGuard traffic stops after some time - from a few minutes to a couple of hours, the highest uptime I've managed with this combination; the jail itself never goes down, reconnecting the client doesn't fix the issue, and there are no relevant messages or hints from the various logs, either from the host or inside the jail.

The jails have been created following the instructions on the original link to the letter; more than that, I've copied all the relevant config files from one jail to another, they are exactly the same. Very intriguing...

 

[Volts]

WireGuard is still working reliably for me.

When traffic stops, does the wg0 interface still exist?

reconnecting the client doesn't fix the issue

How are you doing that?

When this happens, how do you restore connectivity?

 

[Cellobita]

Shutting down the tunnel and reopening, via the Windows client; connectivity to the TrueNAS host itself (13.0-U1), via SSH, never goes down.

I've been running a ping test over the past 11:30 hours, to a 12.3 based jail on TrueNAS 12.0-U8, and there hasn't been a single lost packet - the WireGuard tunnel is rock-solid:

I'll try testing the 13.1 jail, to check for the existence of the wg0 interface when the connection drops.

BTW, on the TN 13.0-U1 host I've tried both the wg kernel module (via Tunables) and the GO userspace versions, and it makes no difference.

 

[Cellobita]

New test (host 13.0-U1, jail 13.1), connection dropped after 85 minutes; wg0 interface up and running:

Deactivating the tunnel and reactivating doesn't work; restarting the jail works, but only after a short time (1 or 2 minutes)

I'm at a complete loss, here

 

[Volts]

Using NAT - have you tried the PersistentKeepalive option?

 

[Cellobita]

I appreciate your input, but I've been running continuous ping tests, so the WireGuard "handshake" happens every two minutes - NAT forwarding therefore never times out; anyway, were it related to this, I believe that

• it would affect jails on 12.0-U8 the same way, and
• it would not stop me from reactivating the tunnel and having traffic flow through it

I have also discarded router issues - on another TN host, at the same remote site, I have spun up a VM running Ubuntu 22.04 and configured WireGuard on it, and have had zero issues so far. It's just that I think a VM is overkill when a jail - properly working - suffices for my needs.

 

[Volts]

I didn't realize you were running constant pings. I agree with your thought process.

I'm reading the guide you linked and trying to understand the choice of NAT for the jail's primary networking.

Doing so requires NAT to run on the host, so I wonder if TSO should actually be disabled on the host. Or if Disable Hardware Networking should be set on the host's Ethernet interface.

Odd. I stopped a jail with NAT networking, but the host's NAT entries for the jail weren't deleted. Do you see the same - what does `ipfw list` show on the host?

 

[Cellobita]

Food for thought, but I'm outta here for an extended weekend in the country - it's bourbon and barbecue only over the next few days; perhaps at some point I'll get an epiphany...

Back Tuesday!

 

[Whattteva]

I'm running Wireguard with 13.0-STABLE and I don't have any issues. But I'm running it on the actual host using this official guide, not the jail.

 

[Volts]

I can't promise it will fix your intermittent issue, but it would probably be good to restructure the host's ethernet + bridge config. @Patrick M. Hausen has pointed out that TrueNAS, when using jails, does something that FreeBSD advises against. TrueNAS puts an IP address on the Ethernet interface, and then adds that interface to bridge0. FreeBSD says bridge member interfaces must not have an IP address, and that any IP address should be on the bridge.

This can be done by stopping jails, removing the automatically-created bridge0 in the CLI, then creating bridge0 and adding the ethernet to it in the web UI.

FreeNAS drops tagged packets addressed to bHYVE VM (tags: lagg vlan vm bhyve 802.1q networking)

Hello, I am running into a very strange issue in which FreeNAS seems to drop frames destined for a bHyve VM running inside of it. Here is my setup, with IP addresses and MACs generalized. I use a DHCP request here to illustrate the problem. My FreeNAS host is an ASUS P11C-i with two...

www.truenas.com

 

[Cellobita]

I'm running Wireguard with 13.0-STABLE and I don't have any issues. But I'm running it on the actual host using this official guide, not the jail.

I'll try this procedure; sounds easier than a jail. Will report back. Many thanks for pointing this out.