Linux Jails - Experimental Script

[benda]

Hi,
i try to get Adguard Home running inside a jail.

Used the docker template and only changed the systemd_nspawn_user_args

Code:

systemd_nspawn_user_args=--network-macvlan=eno1
        --resolv-conf=off
        --system-call-filter='add_key keyctl bpf'
        --bind='/mnt/tank/jailmaker/docker-data/adguard:/docker-data/adguard'

/etc/resolv.conf inside the jail

Code:

nameserver 127.0.0.1
search .

Added a static IP like described here.

My compose.yaml

Code:

version: "3"

services:
  adguardhome:
    image: adguard/adguardhome
    container_name: adguardhome
    deploy:
      resources:
        limits:
          cpus: '0.50' # max. 50%
          memory: 300m
        reservations:
          cpus: '0.1'
          memory: 100m
    volumes:
      - '/docker-data/adguard/conf:/opt/adguardhome/conf'
      - '/docker-data/adguard/work:/opt/adguardhome/work'
    ports:
        # DNS
        - 53:53/tcp
        - 53:53/udp
        # DHCP server
        #- 67:67/udp
        #- 68:68/tcp
        #- 68:68/udp
        # HTTPS/DNS-over-HTTPS
        #- 443:443/tcp
        # DNS-over-TLS
        - 853:853/tcp
        # DNS-over-QUIC
        #- 784:784/udp
        # DNSCrypt
        #- 5443:5443/tcp
        #- 5443:5443/udp
        # WebUI
        - 80:80/tcp
        - 443:443/tcp
        #- 443:443/udp
        # WebUI Setup
        - 3000:3000/tcp
    restart: unless-stopped

I can access the web interface via 192.168.0.10, but if i set the DNS on my router to 192.168.0.10 or direct to my Mac under Network Settings, no website will be loaded.

Any idea?

 

[delgas]

I used the jailmaker docker template. https://github.com/Jip-Hop/jailmaker/tree/main/templates/docker

With it i had no problem to install portainer.

Hmm, interesting. Yeah I used the same script when I created my docker jail but had to add the capabilities before I could create containers from within portainer. I was able to create the portainer-ce container from the docker command line without the need for them though. All my testing was using the SCALE BETA, btw.

I'm a bit of a nspawn noob and didn't realize there was a way to pass all capabilities at once, which is preferable so thank you dirtyfreebooter for that tip.

I moved back to CORE for the moment because I didn't have time to re-create all my containers (and also ran into the annoying bridge issue when firing up my VM from CORE in SCALE and trying to map NFS shares >.< for reference ...) When I fire scale back up (from another boot-pool) I will check the ps ax output and see if --capability=all is there by default for a container created using the docker template script.

Either way, I was happy with the performance benefit I saw using jailmaker with zfs binds over a CORE linux VM mapping NFS shares. As expected, the performance was substantially better, so I'm planning on moving toward scale and jailmaker when I find the time, most likely this weekend.

I'm still curious if anyone knows how to limit max memory for a jail. I tried the systemd MaxMemory directive but the jail seemed to NO OP once the ceiling was hit, instead of just not consuming any more. My short term fix is to get medeveil on it and just set a minimum size for my ZFS arc cache by echo'ing to proc with an init command, but it still feels like there might be a better solution from someone in the community. Fingers crossed.

Thank you everyone for your help!

 

[benda]

Hmm, interesting. Yeah I used the same script when I created my docker jail but had to add the capabilities before I could create containers from within portainer. I was able to create the portainer-ce container from the docker command line without the need for them though. All my testing was using the SCALE BETA, btw.

I'm a bit of a nspawn noob and didn't realize there was a way to pass all capabilities at once, which is preferable so thank you dirtyfreebooter for that tip.

I moved back to CORE for the moment because I didn't have time to re-create all my containers (and also ran into the annoying bridge issue when firing up my VM from CORE in SCALE and trying to map NFS shares >.< for reference ...) When I fire scale back up (from another boot-pool) I will check the ps ax output and see if --capability=all is there by default for a container created using the docker template script.

Either way, I was happy with the performance benefit I saw using jailmaker with zfs binds over a CORE linux VM mapping NFS shares. As expected, the performance was substantially better, so I'm planning on moving toward scale and jailmaker when I find the time, most likely this weekend.

I'm still curious if anyone knows how to limit max memory for a jail. I tried the systemd MaxMemory directive but the jail seemed to NO OP once the ceiling was hit, instead of just not consuming any more. My short term fix is to get medeveil on it and just set a minimum size for my ZFS arc cache by echo'ing to proc with an init command, but it still feels like there might be a better solution from someone in the community. Fingers crossed.

Thank you everyone for your help!

Before i found out about Jailmaiker, i also tried Docker inside a VM and NFS shares.
The bridge problem was easy to resolve with this video. How to setup the VM is here a good post.
NFS shares also work, but not easy to manage, because of owner permission.
Then found that sqlite database in combination with NFS shares not work well. So i was looking for a alternative and found Jailmaker.

 

[usergiven]

Can anyone comment on performance of running the docker config/databases on a pool of hard disks vs ssds? I read a few posts mentioning slow performance in Plex. Also, Is it preferred to create the jail and relocate the config/databases into the jail folder or just bind to those files from within the jail?

Thank you so much for your work @Jip-Hop , for years I've used a separate VM with Docker and NFS shares and it has been my great obsession to figure out a way to simplify the infrastructure and run it all on the NAS with bind mounts directly to the pool data. Amazing.

 

[Stux]

Can anyone comment on performance of running the docker config/databases on a pool of hard disks vs ssds? I read a few posts mentioning slow performance in Plex. Also, Is it preferred to create the jail and relocate the config/databases into the jail folder or just bind to those files from within the jail?

Thank you so much for your work @Jip-Hop , for years I've used a separate VM with Docker and NFS shares and it has been my great obsession to figure out a way to simplify the infrastructure and run it all on the NAS with bind mounts directly to the pool data. Amazing.

Agree.

I literally just finished moving all my dockers out of one vm with nfs mounts and into a sandbox yesterday.

Was good to be able to turn off that vm.

And now jellyfin is not limited to the vm’s core count :)

Performance seems better to me. The sandbox shares memory and cpu with the host more effectively, and the host mounts don’t need to round trip via nfs or networking.

Meanwhile, the maintenance of the sandbox “instance” is no worse than a separate vm was.

 

[delgas]

Well, I made the switch to SCALE and sandboxes using jailmaker for docker / portainer instead of a Debian VM in CORE. I have to agree Stux... turning off that VM using NFS shares was very liberating! Years in the making!

I would not have done all this without jailmaker! I'm VERY pleased with performance. I'm no longer dedicating 32GB of RAM to a VM with limited processors!

Over the past week my 'jails' have crushed about 4 terabytes of database syncs.

Of course, my CIFS and Apple Time Machine shares are a total mess w.r.t perms so working through that is next on the list and I expect nightmares for years! Thank you for making this possible, Jip-Hop!

-D

 

[dakabali]

Hi, anyone using paperless-ngx in a jail? I migrated to jailmaker on cobia using the same compose file as before and I receive the following message when trying to add a new document:

Error occurred while consuming document somepdf.pdf: MissingDependencyError: The program 'unpaper' did not report its version. Message was: Warning: using insecure memory! 7.0.0

Any idea about this problem? I use root to run docker both in the old system and in the jail where I set up with the "official"docker configuration file.

 

[usergiven]

Having an issue with setup. To solve for port conflicts with TrueNAS I've followed the documentation to turn off DHCP, create aliases on my network interface for multiple IPs then opted for TrueNAS to only listen for logins on one of them. This would allow me to run a container that listens on port 80 and 443. I believed when reading the docs that this method was simpler than creating a bridge interface - might be wrong. That container fails when starting saying the ports already taken by TrueNAS.

Another issue, which might be related to the first, is when running Home Assistant through Docker Compose, basic things like network discovery aren't working. After some research it might be due to mdns not functioning properly but I have the container setup in Host network mode and there were no issues with it when it was running on Docker on a separate VM. I am using the Debian Bookworm image rather than ubuntu server. Am I overlooking a service that I need to install into the jail?

 

[Stux]

I used a bridge interface and assigned a static ip via my router/dhcp server.

This seemed the simplest to me and the most like running a separate vm.

It means that host networking in docker will behave as it should.

As it turns out, MAC addresses in systemd-nspawn are hashed from the jail name, so they are stable.

 

[dakabali]

Hi, anyone using paperless-ngx in a jail? I migrated to jailmaker on cobia using the same compose file as before and I receive the following message when trying to add a new document:


Any idea about this problem? I use root to run docker both in the old system and in the jail where I set up with the "official"docker configuration file.

Enabling all the capabilities solved it. It DOES NOT seem to be the default behaviour of the docker config script, though.

 

[Janus Ng]

A dumb question.

Is it possible to run a DNS Server container like bind9 inside the jail?

My purpose is to run an internal DNS Server for local LAN only. It does not support external queries.

 

[delgas]

A dummy question.

Is it possible to run a DNS Server container like bind9 inside the jail?

My purpose is to run an internal DNS Server for local LAN only. It does not support external queries.

I run a pihole container inside a jail without any problems. I think scale will not let you bind to low ports, which includes port 53 for DNS, so use macvlan as the networking inside your jail so it gets its own IP address and everything should work as expected imo.

 

[Janus Ng]

I run a pihole container inside a jail without any problems. I think scale will not let you bind to low ports, which includes port 53 for DNS, so use macvlan as the networking inside your jail so it gets its own IP address and everything should work as expected imo.

Thanks for your suggestion. That's just what I need.

 

[udk]

Enabling all the capabilities solved it. It DOES NOT seem to be the default behaviour of the docker config script, though.

CAP_IPC_LOCK was the missing capability. I only added this one and it works. https://man7.org/linux/man-pages/man7/capabilities.7.html

 

[dakabali]

It might be a silly question and proof of my complete lack of knowledge but... would it be possible to run Proxmox in a jail? I know Proxmox needs its own kernel. Is a jail specific kernel possible without breaking something on the host, at all?

 

[Ericloewe]

Is a jail specific kernel possible without breaking something on the host, at all?

Absolutely not. There is one kernel. If you had two kernels, you'd have a VM.

 

[Jip-Hop]

Enabling all the capabilities solved it. It DOES NOT seem to be the default behaviour of the docker config script, though.

The docker config template does not include all capabilities by default. Only the minimal amount to be able to run docker inside the jail. It is true that you may have to specifically add additional capabilities for some features of containers or packages installed.

 

[Jip-Hop]

by default, jailmaker passes

Code:

--capability=all

to each jail. i wonder if that is broken.

Code:

$ ps ax | grep nspawn
   4026 ?        Ss     0:00 /usr/bin/systemd-nspawn --bind-ro=/sys/module --boot \
    --inaccessible=/sys/module/apparmor --quiet --keep-unit --machine=auberon \
    --directory=rootfs --capability=all --property=DeviceAllow=char-drm rw --bind=/dev/dri \
    --network-bridge=br0 --resolv-conf=bind-host ...

Jailmaker doesn't pass --capability=all by default, only when the docker_compatible=1 config is set. But this contains more capabilities than usually required. Which is why I didn't add this to the docker config template.

In the future I plan to deprecate the docker_compatible config setting and instead introduce a
privileged option to disable seccomp, set DevicePolicy=auto and add all capabilities.

 

[Jip-Hop]

I'm still curious if anyone knows how to limit max memory for a jail.

Sorry, I don't know if this is possible. I haven't ran into this myself so didn't look into it.

 

[dakabali]

Jailmaker doesn't pass --capability=all by default, only when the docker_compatible=1 config is set. But this contains more capabilities than usually required. Which is why I didn't add this to the docker config template.

In the future I plan to deprecate the docker_compatible config setting and instead introduce a
privileged option to disable seccomp, set DevicePolicy=auto and add all capabilities.

Thank you for the clarification @Jip-Hop