Linux Jails - Experimental Script

[cap]

I have a few jails. I have noticed that only one of the jails starts automatically when the server is restarted. I am pretty sure that all jails were configured accordingly during setup.
When I look at the configuration, I can't see any difference. What could be the reason for this?
Is it the same for you?

It works. It was my mistake. I had created a directory in the directory where the jails are located, in which a "backup jail" was placed. You are not supposed to create a directory there, otherwise jailmaker will break when booting.

 

[Jip-Hop]

Glad to hear the issue is resolved!

 

[skittlebrau]

The only problem I have left is that I'm struggling to get internet/external connectivity with docker bridge networks (either the default docker0 bridge or docker compose bridge networks) when passing TrueNAS host bridges using ipvlan or macvlan (eg. —network-macvlan=br10 —network-macvlan=br20 —network-macvlan=br50). Using docker host networking with the same containers works fine, but this isn’t sustainable for more than a couple of containers due to port conflicts. If I enter one of the docker containers via bash shell and try to ping or curl anything, it fails. It’s almost like the packets aren’t going anywhere. I can access the web interfaces for any of these containers on separate desktops/notebooks when within the same VLAN and subnet, but not across other subnets.

If I pass a host bridge into the jail with —network-bridge=br10 (or any of the other bridges above) then the docker bridge networks work.

My current workaround is to have one jail for docker containers that need docker host networking (eg. syncthing) with 3x bridges attached via macvlan and and another jail that has just a single bridge (br20) attached (using —network-bridge=br20) where I want docker bridge networking.

I’ve been prodding jailmaker for hours over the last week, so I’m happy to add to the documentation about edge cases people might encounter.

 

[Jip-Hop]

—network-macvlan=br10 —network-macvlan=br20 —network-macvlan=br50

I'm not sure --network-macvlan is supposed to be used with a bridge network interface. It looks like you actually want support --network-bridge= behaviour for multiple veth interfaces/bridges but that feature request is still open. So using the --network-macvlan option has unwelcome side effects?

Anyhow documentation contributions are always welcome :) I only have a single network interface in my NAS and no VLANs so my setup is a lot simpler.

 

[skittlebrau]

I'm not sure --network-macvlan is supposed to be used with a bridge network interface. It looks like you actually want support --network-bridge= behaviour for multiple veth interfaces/bridges but that feature request is still open. So using the --network-macvlan option has unwelcome side effects?

Anyhow documentation contributions are always welcome :) I only have a single network interface in my NAS and no VLANs so my setup is a lot simpler.

In the past (on UnRAID and OMV) I've used macvlan and ipvlan networking on parent interfaces that are bridges and that worked fine, but admittedly that was with Docker on bare metal, so it wasn't nested containers like it is in this case. There probably is a way to work around it, but I think I lack the willpower for it for now.

I think the container setup I've arrived at will work fine for my purposes, so thanks again. I'll put forward some documentation additions when I get some time.

 

[Jip-Hop]

@skittlebrau if the goal is to have syncthing operating on all network interfaces but deny access to the management VLAN can't you just add an iptables rule on TrueNAS SCALE to block access to the management VLAN based on the UID which is running the syncthing process? https://serverfault.com/questions/1138770/iptables-block-ip-range-user I don't think you even need jailmaker in this case, as syncthing can run directly on the host as well.

 

[Jip-Hop]

I did some more reading and came across the RestrictNetworkInterfaces systemd unit setting. Could be very useful to restrict a jail to certain interfaces only.

Required kernel options are available on SCALE:

Code:

egrep '(CONFIG_BPF|CONFIG_BPF_SYSCALL|CONFIG_CGROUP_BPF)=' /boot/config-$(uname -r)
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_CGROUP_BPF=y

However on TrueNAS SCALE (debian bookworm) systemd comes with the BPF_FRAMEWORK config disabled:

Code:

systemctl --version
systemd 252 (252.6-1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

And it looks like the libbpf package is not installed either. Therefore the below doesn't work:

systemd-run -t -p RestrictNetworkInterfaces="lo" ping archlinux.org

This should fail if bpf is functional (but it still pings archlinux.org unfortunately). More here.

But using the IPAddressAllow systemd unit setting does work!

Now I can only access IP 8.8.8.8:
systemd-run -t -p IPAddressAllow=8.8.8.8 -p IPAddressDeny=any ping 8.8.8.8

Now I can only access IP 1.1.1.1 so ping to 8.8.8.8 fails:
systemd-run -t -p IPAddressAllow=1.1.1.1 -p IPAddressDeny=any ping 8.8.8.8

By adding these options to the jail config we can now restrict a jail to specific IP address (ranges):
systemd_run_default_args=-p IPAddressAllow=1.1.1.1 -p IPAddressDeny=any --property=KillMode=mixed --property=Type=notify [...]

But to take the @skittlebrau syncthing example. I think you can use the syncthing.service file as reference and start it directly on the SCALE host using systemd-run (with the corresponding options taken from the .service file). You can then add the -p IPAddressAllow= and -p IPAddressDeny= options to restrict syncthing access to your management VLAN. No need to use a jail. But as indicated above it looks like you can even restrict an entire jail to certain IP ranges, and it would apply to all processes (including synching) running inside it...

 

[Jip-Hop]

And after more reading I think it's quite easy to pass additional bridge interfaces to the jail without using macvlan. You'd have to use the --network-veth-extra= systemd-nspawn option, combined with the -p ExecStartPost= systemd-run option. I hope one of these suggestions solves your last remaining problem @skittlebrau ;)

 

[Yser]

Hi. Noob here. I went through Jailmaker installation successfully, but don't know how to install Portainer or Nextcloud AIO docker image with this.
Can some make a video or step-by-step instruction on how to install Docker image?

 

[skittlebrau]

Hi. Noob here. I went through Jailmaker installation successfully, but don't know how to install Portainer or Nextcloud AIO docker image with this.
Can some make a video or step-by-step instruction on how to install Docker image?

You would just need to follow the normal Docker installation procedure. Assuming you installed Debian Bookworm, follow the official guide: https://docs.docker.com/engine/install/debian/

Set up the repository

Code:

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

Install Docker-CE

Code:

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Install Portainer

Install Portainer CE with Docker on Linux | 2.19 | Portainer Documentation

docs.portainer.io

Code:

# Create the data volume for storing Portainer configuration data 
docker volume create portainer_data
# Pull the image and create the Portainer container
docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

 

[Yser]

Thank you for the reply. So I created jail "jail1". How do we get into jail1 to begin typing commands?

I tried::

root@Server[~]# jlmkr exec jail1 'sudo apt-get update'

but didn't work.

 

[skittlebrau]

Thank you for the reply. So I created jail "jail1". How do we get into jail1 to begin typing commands?

I tried::

root@Server[~]# jlmkr exec jail1 'sudo apt-get update'

but didn't work.

It's easier if you enter the jail shell environment if you've got several commands to run.

Code:

jlmkr shell jail1

@skittlebrau if the goal is to have syncthing operating on all network interfaces but deny access to the management VLAN can't you just add an iptables rule on TrueNAS SCALE to block access to the management VLAN based on the UID which is running the syncthing process? https://serverfault.com/questions/1138770/iptables-block-ip-range-user I don't think you even need jailmaker in this case, as syncthing can run directly on the host as well.

Thanks. I'll look into this option and also veth-extra as you mentioned. I did give it a quick go and was able to see the new interface pop up on the host side, but wasn't sure where to go from there to 'connect' the new veth to my network. I'll see what I come up with since I enjoy a challenge

@skittlebrau if the goal is to have syncthing operating on all network interfaces but deny access to the management VLAN can't you just add an iptables rule on TrueNAS SCALE to block access to the management VLAN based on the UID which is running the syncthing process? https://serverfault.com/questions/1138770/iptables-block-ip-range-user I don't think you even need jailmaker in this case, as syncthing can run directly on the host as well.

That'd work, but I wasn't able to get host networking with docker in a rootless jail working. Docker attempts to alter iptables on the host, but this fails due to the jail being rootless, so the docker service refuses to start.

In the meantime, I'll likely just run syncthing in a jail with br10 (vlan10) passed through, with another jail for docker on br20 (vlan20) for media/IoT services. Realistically it's the desktops/notebooks on that singular vlan that require high speed sync since I work with large files, whereas my other devices are external and go through WAN anyway. I'll run with this arrangement for now until I figure out the configuration needed to get it all into a single jail - the posts you've provided in the thread have given me some things to test out.

Thanks again for your help @Jip-Hop.

 

[Yser]

It's easier if you enter the jail shell environment if you've got several commands to run.

Code:

jlmkr shell jail1

Yours and Hip-hop instructions really helped and got me through to installing Portainer. Thank you very much.

 

[skittlebrau]

Since it's not mentioned in the wiki, are there any particular security implications that users should be aware of?

In the past when I've run docker in an unprivileged lxc on proxmox, the dev team warn users against doing it because it exposes /proc and /sys from the host, so is there anything similar happening with nspawn in order to make the container compatible with docker? I noticed the extra nspawn arguments needed for docker to work inclusive of things like seccomp filtering, but I don't know enough about these flags and what sort of potential risk (however small it is in reality, especially in a homelab scenario) each carries.

It might be good to add this in addition to the general disclaimer so that users can make an informed choice about what route they might want to take regarding how they set up their containers.

 

[Jip-Hop]

I already wrote something about it in the readme on the develop branch. Not too happy with it but I'd like it to be concise. Also I don't know all the details myself. Using the docker_compatible option basically puts the jail in (what other container runtimes call) privileged mode and I intend to rename and/or split this functionality in the future to better reflect what it's doing (and what the consequences may be). I personally don't worry too much about locking down the jail itself too much a.t.m. because I use docker inside it, which I use to isolate those containers with minimal permissions/capabilities and prevent them from running as root. I treat my jail as if it was part of the host OS, and I don't think it introduces security implications compared to if docker was actually part of SCALE itself (with the exception that I have to trust the rootfs image coming from linuxcontainers.org).

 

[K.J]

Can anyone tell me if after installing Dragonfisch 24.0.4

Truenas/truechart has apps like Sandboxes (Jailmaker) running?

I want to be able to install docker-compose, portainer.

What is the biggest problem I can expect?

@Jip-Hop (verwacht jij grote problemen als ik dit ga installeren. Heb gisteren Dragonfisch geïnstalleerd op mijn huidige systeem en dat werkt naar tevredenheid).

 

[Jip-Hop]

Dragonfish hasn't been released so I don't recommend using it with jailmaker yet. I'm working on the next release of jailmaker in the develop branch, which will have lots of improvements as well as a minor change specifically for Dragonfish: use a shell alias to invoke jlmkr instead of a symlink. Using the current version of jailmaker on Dragonfish will likely fail when running the 'install' command of jailmaker but you don't need to install anything, you can just use the python file directly (or manually create a shell alias until the next release of jailmaker is ready).

 

[skittlebrau]

Several weeks later after settling on a chosen way to set up everything, I'm very happy with the performance of everything.

In my opinion, this is currently the best way to run Nextcloud on TrueNAS SCALE since you can use Nextcloud AIO which is the official install method and the easiest for long-term maintenance.

 

[murzik]

I am getting readonly file system error tring to install jlmkr.py
OSError: [Errno 30] Read-only file system: '/mnt/POOL01/jailmaker/jlmkr.py' -> '/usr/local/sbin/jlmkr'
any idea?

 

[anodos]

I am getting readonly file system error tring to install jlmkr.py
OSError: [Errno 30] Read-only file system: '/mnt/POOL01/jailmaker/jlmkr.py' -> '/usr/local/sbin/jlmkr'
any idea?

Is his DragonFish Beta? As I mentioned earlier in this thread, in DragonFish the root filesystem is readonly. You will need to basically enable developer-mode.

/usr/local/libexec/disable-rootfs-protection

@Jip-Hop this is something your script will need to account for.