Jail networking not working....

[jagdtigger]

Good evening!

So i wanted to relax a bit and joined my minecraft server that is hosted in a jail. After a while the client thrown my out with a "connection timed out" error. Restarting the jail did not solved it so i rebooted truenas. This did fix it, but after starting the server and joining it did it again. Interesting part is that no matter which jail i use ping just prints a "no route to host" error. Truenas is U8 and the jail is manually created with 12.2-RELEASE-p15

Has anyone ran into this before?

 

[Patrick M. Hausen]

More information, please. Complete hardware and network configuration of your TrueNAS system. Complete configuration of your jail. How do you think anyone could be able to assist you without that information?

 

[jagdtigger]

Havent thought about that, studying accounting wore me down quite a bit it seems....

Case: aerocool rs9
CPU: 1700X
Mobo: MSI MS-7A33
RAM: 2x Corsair Vengeance 32GB
GPU: GeForce GT 710
PSU: be quiet! 500W
NIC: IBM 49Y4242 Intel I340-T4 (mobo NIC diisabled)
HBA: mobo onboard, HP H220 IT mode
Bacplanes: Icybox ib-555, ib2280ssk

Network setup:
igb0, igb1, igb2 -> lagg0 -> bridge0 <- vnet0.9, vnet0.8, vnet0.6, vnet1
igb3 -> bridge200 <- vnet0.10, vnet0
bridge 300

bridge0 is the Main LAN, connects to router via a tplink managed switch, dont use vlan for this network
bridge200 is where the publicly exposed services live, physically separate from Main LAN, connects to router via a dumb switch, also no vlan
bridge300 is currently unused and doesnt connect to any physical LAN, purely internal

Router: Custom built pfsense. (VM's attached to the same network as the jails can be reached and can ping it so i dont think the router is the culprit.)

As for symptoms what puzzles me that all jails loose network connectivity regardless which bridge they are attached to.

Jail config is pretty simplistic, created a jail with the previously mentioned release, installed openjdk8-jre and openjdk17 and downloaded files for the 2 server. Both started with a different script manually and run under the minecraft user. Network wise i disabled NAT and assgned fix IP vie the truenas webui, interface set to vnet0:bridge200.

IDK what went wrong and when, havent played in the last few months.

 

[Patrick M. Hausen]

You have all IP address configuration on the bridge interfaces, not on any of the members? You disabled hardware offloading for all interfaces that are members of bridges?

 

[jagdtigger]

Yes, only have ip's on the bridge interfaces, hw offload wasnt disabled. Disabled it on all of the physical interfaces then rebooted the machine, same end result, connection timed out error in client and cant ping anything from jail.

 

[Patrick M. Hausen]

Then we need the output of iocage get all <jailname> for a jail that does not work.

 

[jagdtigger]

Here you go:

Code:

root@zenifer[~]# iocage get all minecraft
CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:0
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:1
bpf:1
children_max:0
cloned_release:12.2-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.10.1
defaultrouter6:2001:470:211d:10::1
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:minecraft
host_hostuuid:minecraft
host_time:1
hostid:00000000-0000-0000-0000-4ccc6afb4bb2
hostid_strict_check:0
interfaces:vnet0:bridge200
ip4:new
ip4_addr:192.168.10.30/24
ip4_saddrsel:1
ip6:new
ip6_addr:2001:470:211d:10::30/64
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/minecraft/data
jail_zfs_mountpoint:none
last_started:2022-04-11 08:20:52
localhost_ip:none
login_flags:-f root
mac_prefix:a2369f
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:12.2-RELEASE-p15
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:a2369fb35b9f a2369fb35ba0
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:auto
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off

 

[Patrick M. Hausen]

And 192.168.10.30 is the correct IP address matching the network of bridge200? And 192.168.10.1 is the default gateway in that network? And you cannot ping 192.168.10.1 after you used e.g. iocage console minecraft to get into the jail?

if the answer to all of this is yes (which I expect, honestly), set vnet_default_interface to "none" and restart the jail.

And probably put an "up" into the options fields of all physical interfaces that are bridge members and don't have an IP address.

 

[jagdtigger]

OK, i set it that way. But testing will have to wait until tomorrow, im off to sleep then to work.

 

[jagdtigger]

Okay, had to reboot the machine again to get network in the jails. Fired up both game servers and joined to the same as last time, after a few minutes i got the same timeout error and cant ping anything from the jail. All of the jails lost network connectivity again....

 

[Patrick M. Hausen]

Ah ... I did not read from your initial post that it works after a reboot, then fails later. I assumed you had no networking in jails at all. This is weird, never seen anything like this with TrueNAS, although there was a bug in FreeBSD concerning epair interfaces. You could try the 13.0 RC1 once it's published.

 

[jagdtigger]

Well u8.1 didnt help, no 13 rc1 atm....

 

[jagdtigger]

@Patrick M. Hausen
Seems like RC1 fixed it, ill let the client run in the background just to make sure....