Jail fails to start because it can acquire an ip with DHCP

[Usul]

When I try to start my newly created jail I get the following error message:

This translates in middlewared.log to :

code:

[2022/09/24 13:55:33] (INFO) iocage.callback():71 - Test successfully created!
[2022/09/24 13:55:40] (DEBUG) iocage.__start_jail__():248 - Grabbing IPv4 default route
[2022/09/24 13:55:40] (DEBUG) iocage.__start_jail__():250 - Default IPv4 Gateway: 192.168.0.254
[2022/09/24 13:55:40] (DEBUG) iocage.__start_jail__():253 - Grabbing IPv6 default route
[2022/09/24 13:55:40] (DEBUG) iocage.__start_jail__():255 - Default IPv6 Gateway: 2a01:e0a:abd:8a90::1
[2022/09/24 13:55:40] (INFO) iocage.callback():71 - * Starting Test
[2022/09/24 13:55:41] (INFO) iocage.callback():71 - + Started OK
[2022/09/24 13:55:41] (INFO) iocage.callback():71 - + Using devfs_ruleset: 1000 (iocage generated default)
[2022/09/24 13:55:41] (INFO) iocage.callback():71 - + Configuring VNET OK
[2022/09/24 13:55:41] (INFO) iocage.callback():71 - + Using IP options: vnet
[2022/09/24 13:55:47] (INFO) iocage.callback():71 - + Starting services OK
[2022/09/24 13:55:47] (INFO) iocage.callback():71 - + Executing poststart OK
[2022/09/24 13:55:47] (ERROR) middlewared.job.run():367 - Job <bound method accepts.<locals>.wrap.<locals>.nf of <middlewared.plugins.jail_freebsd.JailService object at 0x81b119c40>> failed
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/jail_freebsd.py", line 1297, in start
iocage.start(used_ports=[6000] + list(range(1025)))
File "/usr/local/lib/python3.9/site-packages/iocage_lib/iocage.py", line 1811, in start
ioc_start.IOCStart(
File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_start.py", line 87, in __init__
raise e
File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_start.py", line 84, in __init__
self.__start_jail__()
File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_start.py", line 947, in __start_jail__
iocage_lib.ioc_common.logit({
File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_common.py", line 107, in logit
callback(content, exception)
File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_common.py", line 80, in callback
raise callback_exception(message)
RuntimeError: + Acquiring DHCP address: FAILED, address received: ERROR, check jail logs

Stopped Test due to DHCP failure

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run
await self.future
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 393, in __run_body
rv = await self.middleware.run_in_thread(self.method, *([self] + args))
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1154, in run_in_thread
return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1151, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/jail_freebsd.py", line 1299, in start
raise CallError(str(e))
middlewared.service_exception.CallError: [EFAULT] + Acquiring DHCP address: FAILED, address received: ERROR, check jail logs

I tried to implement the solution described at https://www.truenas.com/community/threads/jails-assigning-separate-nic.82774/post-697219
so I :

• created bridge0, who gets configured with DHCP for both v4 and v6
• tried to set the vnet_default_interface to vnet:bridge0

But the only choices I have for vnet_default_interface are `none`,`igc1` and `auto`. My network is configured like this

root@truenas[/var/log]# ifconfig -a
igc0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4e523bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 6c:bf:b5:02:5b:e8
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igc1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 6c:bf:b5:02:5b:e9
media: Ethernet autoselect
status: no carrier
nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: bridge0 for jails
ether 58:9c:fc:00:03:78
inet6 fe80::5a9c:fcff:fe00:378%bridge0 prefixlen 64 scopeid 0x5
inet6 2a01:e0a:abd:8a90:5a9c:fcff:fe00:378 prefixlen 64 autoconf
inet 192.168.0.14 netmask 0xffffff00 broadcast 192.168.0.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 55
groups: bridge
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

my test jail is defined like this

root@truenas[/var/log]# iocage get all Test
CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:0
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:0
bpf:1
children_max:0
cloned_release:13.1-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:auto
defaultrouter6:auto
depends:none
devfs_ruleset:4
dhcp:1
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:Test
host_hostuuid:Test
host_time:1
hostid:83f73568-3aac-11ed-a471-6cbfb5025be8
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:none
ip4_saddrsel:1
ip6:new
ip6_addr:vnet0|accept_rtadv
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/Test/data
jail_zfs_mountpoint:none
last_started:none
localhost_ip:none
login_flags:-f root
mac_prefix:6ebfb5
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:13.1-RELEASE
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:down
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:6ebfb51b34d2 6ebfb51b34d3
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:auto
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off

Can someone with more experience tell me what I missed? Is there some docs I need to read or a detail in a doc that I missed?

before having a bridge I was trying to directly use the igc1 when defining the jail.

Thank you in advance,
Ludo

 

[Usul]

Ho and the screenshot is from another try, before I started using test as a name for my jail.

 

[Usul]

Also noticed that if I got in advanced config bridge0 is selected.

How can I debug this?

 

[Glorious1]

I really don't know how this stuff works, but I compared your jail config with one of mine that works fine. Below are some of the features that are different in mine.

In my case, I set a desired IP address for the jail (ip4_addr). That's not necessary. I suspect your major problem is that you didn't assign default router address, so the jail doesn't know where to look to get an IP address.

Also it may be a problem that you didn't assign an IPv4 interface as I see in your screenshot (in my case I set vnet0).

My different features:

Code:

allow_raw_sockets:1
allow_tun:1
boot:1
defaultrouter:192.168.0.1
dhcp:0
ip4_addr:vnet0|192.168.0.104/24

 

[Klavaro]

Did you made any progress? I'm stunned that there's multiple threads about this problem, but seems like people just take the easy way out and use static IP or NAT. I prefer assigning IP's from the DHCP server by MAC addresses, which then also gives a hostname to each device so I can access them neatly in the domain. For me the not so neat solution is to assign the IP's on the DHCP server and use static IP on the jails together.

 

[sandhraj]

Did you made any progress? I'm stunned that there's multiple threads about this problem, but seems like people just take the easy way out and use static IP or NAT. I prefer assigning IP's from the DHCP server by MAC addresses, which then also gives a hostname to each device so I can access them neatly in the domain. For me the not so neat solution is to assign the IP's on the DHCP server and use static IP on the jails together.

Has anyone had a fix for this. I am able to configure and access static IP jails but Plex doesn't like that and interferes in playback.

 

[RogueNeurons]

Did you made any progress? I'm stunned that there's multiple threads about this problem, but seems like people just take the easy way out and use static IP or NAT. I prefer assigning IP's from the DHCP server by MAC addresses, which then also gives a hostname to each device so I can access them neatly in the domain. For me the not so neat solution is to assign the IP's on the DHCP server and use static IP on the jails together.

Im still searching for a solution same problem as yours...

 

[jg3]

Add me to the list of people for whom Jails will work if assigned a static IP but get the above message and nothing useful in the logs when using DHCP.
My symptoms have been the same for years on TrueNAS and FreeNAS before that ...
(1) I can create a jail with DHCP and start it up once, look at my DHCP server logs to get the MAC address used to request and correctly get an IP from the DHCP pool.
(2) Create a static DHCP entry for that MAC address, give it an IP outside the pool, restart the jail DHCP correctly assigns the static IP but
(3) On subsequent boots I get the above error message. Restarting the DHCPd on my pfSense server has no impact
(4) Get frustrated and assign a static IP/mask and Gateway. This stinks because I will eventually need to change my GW address. WHAT A PITA.

 

[sretalla]

If anyone is interested in trying to do the appropriate manual bridge setup to "test" it, I suspect it's all related to the automatic bridging that gets done if you just have a NIC and no bridge defined in Network Global Configuration.

Don't try to do any changes on the NIC/bridge config with the jails running (nor VMs)... you may need to set things to manual start and reboot first.

 

[Patrick M. Hausen]

I described how to statically define a bridge interface here:

Jails on Core 13.0-U5.2 do not connect to the internet

iX' implementation violated FreeBSD's documented architecture for the bridge implementation since day one of VNET jails. I don't understand why they do it that way. Using vnet_default_interface: auto performs some "magic" that frequently seems to work (still ending with an illegal configuration)...

www.truenas.com

Replace "lagg0" with your network interface. Move the IP address to the bridge, set vnet_default_interface to "none" instead of "auto".