Debian (Devuan) Linux jail network access problems

[Chris123]

I got a Debian (Devuan) Linux jail up and running on Freenas by following these instructions: https://forums.freebsd.org/threads/setting-up-a-debian-linux-jail-on-freebsd.68434/ (I had to make a few changes in order to make it work on FreeNAS).

So far it seems to work well EXCEPT for one issue: Network access. This issue is not specific to FreeNAS, I get the same problem on a pure FreeBSD-install as well. Other users have reported the same problem, but to my knowledge there hasn't been any solution to this yet. It seems to be an ifconfig / inet / route issue, but everything I've tried have failed.

I'm hoping someone might have some ideas or be able to point me in the right direction on how to configure ifconfig / inet routing.

Trying ifconfig eth0 inet 192.168.1.113 netmask 255.255.255.0 gives the following error:

Added the following to /etc/network/interfaces without any success

Code:

iface eth0 inet static
    address 192.168.1.113/24
    gateway 192.168.1.1

Iocage config.json for 'debjail':

Code:

{
    "CONFIG_VERSION": "14.1",
    "allow_chflags": "0",
    "allow_mlock": "0",
    "allow_mount": "1",
    "allow_mount_devfs": "1",
    "allow_mount_nullfs": "0",
    "allow_mount_procfs": "1",
    "allow_mount_tmpfs": "1",
    "allow_mount_zfs": "0",
    "allow_quotas": "0",
    "allow_raw_sockets": "1",
    "allow_set_hostname": "1",
    "allow_socket_af": "0",
    "allow_sysvipc": "0",
    "allow_tun": "0",
    "available": "readonly",
    "basejail": "no",
    "boot": "off",
    "bpf": "no",
    "children_max": "0",
    "cloned_release": "EMPTY",
    "comment": "none",
    "compression": "lz4",
    "compressratio": "readonly",
    "coredumpsize": "off",
    "count": "1",
    "cpuset": "off",
    "cputime": "off",
    "datasize": "off",
    "dedup": "off",
    "defaultrouter": "none",
    "defaultrouter6": "none",
    "depends": "none",
    "devfs_ruleset": "4",
    "dhcp": "off",
    "enforce_statfs": "2",
    "exec_clean": "1",
    "exec_fib": "0",
    "exec_jail_user": "root",
    "exec_poststart": "/usr/bin/true",
    "exec_poststop": "/usr/bin/true",
    "exec_prestart": "/usr/bin/true",
    "exec_prestop": "/usr/bin/true",
    "exec_start": "/etc/init.d/rc 3",
    "exec_stop": "/etc/init.d/rc 0",
    "exec_system_jail_user": "0",
    "exec_system_user": "root",
    "exec_timeout": "60",
    "host_domainname": "none",
    "host_hostname": "debjail",
    "host_hostuuid": "debjail",
    "host_time": "yes",
    "hostid": "744f9272-1cb5-11e8-919c-6c0b84084db5",
    "hostid_strict_check": "off",
    "interfaces": "vnet0:bridge0",
    "ip4": "new",
    "ip4_addr": "em0|192.168.1.113/24",
    "ip4_saddrsel": "1",
    "ip6": "new",
    "ip6_addr": "none",
    "ip6_saddrsel": "1",
    "jail_zfs": "off",
    "jail_zfs_dataset": "iocage/jails/debjail/data",
    "jail_zfs_mountpoint": "none",
    "last_started": "2019-12-17 06:27:44",
    "login_flags": "-f root",
    "mac_prefix": "02ff60",
    "maxproc": "off",
    "memorylocked": "off",
    "memoryuse": "off",
    "mount_devfs": "1",
    "mount_fdescfs": "0",
    "mount_linprocfs": "1",
    "mount_procfs": "1",
    "mountpoint": "readonly",
    "msgqqueued": "off",
    "msgqsize": "off",
    "nmsgq": "off",
    "notes": "none",
    "nsemop": "off",
    "nshm": "off",
    "nthr": "off",
    "openfiles": "off",
    "origin": "readonly",
    "owner": "root",
    "pcpu": "off",
    "priority": "99",
    "pseudoterminals": "off",
    "quota": "none",
    "release": "EMPTY",
    "reservation": "none",
    "resolver": "/etc/resolv.conf",
    "rlimits": "off",
    "securelevel": "2",
    "shmsize": "off",
    "stacksize": "off",
    "stop_timeout": "30",
    "swapuse": "off",
    "sync_state": "none",
    "sync_target": "none",
    "sync_tgt_zpool": "none",
    "sysvmsg": "new",
    "sysvsem": "new",
    "sysvshm": "new",
    "template": "no",
    "type": "jail",
    "used": "readonly",
    "vmemoryuse": "off",
    "vnet": "off",
    "vnet0_mac": "none",
    "vnet1_mac": "none",
    "vnet2_mac": "none",
    "vnet3_mac": "none",
    "vnet_default_interface": "auto",
    "vnet_interfaces": "none",
    "wallclock": "off"
}

And yes, I know Linux can easily run in a Bhyve VM.

 

[sretalla]

Jails are having issues with static IPs at the moment.. possibly fixed in 11.3.

Can you get a DHCP address? (would need to set bpf=yes)

 

[Chris123]

Jails are having issues with static IPs at the moment.. possibly fixed in 11.3.

Can you get a DHCP address? (would need to set bpf=yes)

Thanks for your reply!

Ok, so I set bpf=yes (and vnet=on, required when setting bpf=yes I guess). I was able to start and log in to the jail, but now ifconfig returned nothing. Could it be that Devuan doesn't recognize the vnet interface? Seems to be some mismatch between what iocage is providing to the jail OS (devuan) and what the jail OS (devuan) is able to interpret....

Here's my output:

 

[sretalla]

How about running ip address or dhclient in the jail?

Also setting dhcp=on (I guess I should have said that too... my bad).

The IP in caps is the interface, not my typing, use lower case.

Also, I think there's something to do to create the vnet interfaces in the jail... it's in a script somewhere for the native jails, but I guess not in devuan. It may be worth searching around for it as I think the concept here is dangerously close to the holy grail of having a jail where we could run a supported version of docker and mount storage directly into it from FreeNAS for use by the docker containers.... I know, some will say that sounds like the product that shall not be named, but whis would effectively be doing it with Linux, so maybe it can be different this time?

 

[Chris123]

How about running IP address or dhclient in the jail?

Also setting dhcp=on (I guess I should have said that too... my bad).

The IP in caps is the interface, not my typing, use lower case.

Also, I think there's something to do to create the vnet interfaces in the jail... it's in a script somewhere for the native jails, but I guess not in devuan. It may be worth searching around for it as I think the concept here is dangerously close to the holy grail of having a jail where we could run a supported version of docker and mount storage directly into it from FreeNAS for use by the docker containers.... I know, some will say that sounds like the product that shall not be named, but whis would effectively be doing it with Linux, so maybe it can be different this time?

When dhcp=on the jail doesn't start. Seems like it is looking for /etc/rc.conf, which will of course result in an error when the jail is not FreeBSD. I guess it want to add ifconfig_epair0b="DHCP". Is there a way to direct iocage to use another file?

I tried to add a rc.conf (with 'debjail' as hostname) just to see what would happen

Since I'm not able to enter the jail when using DHCP, i was only able to try IP address and dhclient in a static-ip-configuration:

Seems like the network interfaces is not configured properly in the jail...?

I agree, having a near full-fledged (non-gui) Linux jail would open some possibilities. All the software I'm planning use ran without any issues (besides network access).

 

[sretalla]

maybe the next thing to try is ifconfig -a (to see if there are any network interfaces at all, even if they may be inactive).

Also, ip address is intended to be the command, don't subsitute the actual address (I would have used ip <address> in that case).

Maybe some of the stuff here will provide a way forward if there are no interfaces at all:

NetworkConfiguration - Debian Wiki

wiki.debian.org

 

[Chris123]

ifconfig -a shows that eth0 and lo0 are "active", so Devuan must somehow see a "network adapter".

Also, there are some statistics of the network adapter in /proc/net/dev

ip address produced the same error as when using ip 192.168.x.x.:

I had a quick look at the link you sent, I see there are some things to try out. It also led me to look into how networking gets initialized on Devuan through init.d and run levels rc0.d, rc1.d, rc2.d,....,rc6.d. The devuan jail uses run level 2 (rc2.d) by default, which supposedly is without networking according to this, but after installing a pure Devuan VM in VIrtualbox it turns out that it also uses run level 2 (with a working network). But I will compare the two in more detail to see if anything obvious is missing.

 

[sretalla]

You could try ifconfig eth0 192.168.x.x netmask 255.255.255.0 up

That might work, but I'm not sure if you will then need to somehow specify a default gateway.

Possibly also useful to do ifconfig lo0 127.0.0.1 up

 

[Chris123]

ifconfig eth0 192.168.1.113 netmask 255.255.255.0 up and ifconfig lo0 127.0.0.1 up gave error. I tried to google the errors but didn't find anything that made sense to me..

Yesterday I tried to install different versions of Devuan (oldstable, testing, and unstable) and also trying on FreeBSD 13 with linux compatibility layer kernel 3.2. Nothing worked. Same issue. (Not sure I understand how the Linux compatibility layer kernel integrates into the linux OS on FreeBSD, 2.6.32 and even 3.2 is a big difference form the kernel version 4.x that comes with Devuan/Debian stable ... can that cause compatibility issues?).

 

[sretalla]

How about ifconfig eth0 up

I agree there's sufficient mysterious magic going on here to have me at a loss regarding how it all works.

Based on the last results, maybe the linux commands are the right ones rather than the FreeBSD ones.

 

[Chris123]

Same problem

 

[vesper]

Sorry for necroposting, but to those who got same problem there is solution:

• copy ifconfig and dhclient utitilites from TrueNAS host /rescue to jail: cp /rescue/ifconfig /mnt/Storage/iocage/jails/debjail/root/sbin/ where /mnt/Storage/ will be path to your pool with iocage
• check how interface named in host OS (em0/re0/eth0 etc) in my case re0
• add IP manualy iocage set ip4_addr="re0|192.168.88.200/24" debjail
• restart jail iocage restart debjail where debjail is my jail name

It should work now.

Code:

* Starting debjail
  + Started OK
  + Using devfs_ruleset: 1004 (cloned from devfs_ruleset 0)
  + Using IP options: ip4.addr=re0|192.168.88.201/24 ip4.saddrsel=1 ip4=new ip6.saddrsel=1 ip6=new
  + Starting services OK
  + Executing poststart OK

Note: vnet, dhcp, bpf should be disabled
iocage set dhcp=0 debjail iocage set bpf=0 debjail iocage set vnet=0 debjail