Linux Jails - Experimental Script

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
"systemd-nspawn may be used to run a command or OS in a light-weight container. In many ways it is similar to chroot, but more powerful since it uses namespaces to fully virtualise the the process tree, IPC, hostname, domain name and, optionally, networking and user databases. It is similar to LXC, but much simpler to configure. Most of the necessary software is already installed on contemporary Debian systems." https://wiki.debian.org/nspawn
Systemd-nspawn may be to SCALE what jails is to CORE and the great thing is: it already works on TrueNAS SCALE!

Some stuff you can do with it:
  • Install packages with apt (inside the jail) without breaking the host
  • Run docker, docker-compose, portainer etc. without the overhead of k3s
  • Bind mount your data pools directly inside the jail, no need for NFS!
  • Possibly replace some of your VMs
  • Use host networking, or macvlan networking for a dedicated IP address via DHCP
  • GPU passthrough
The 'jails' are persistent and survive updates of TrueNAS SCALE if you're using the jailmaker script I'm developing. It's experimental and in early development but has received positive responses from the first testers. I recommend testing it on a non-production system for now.

I'm looking for testers, feedback, collaborators and votes for the request to officially support systemd-nspawn on SCALE. :smile:
 
Last edited:

truecharts

Guru
Joined
Aug 19, 2021
Messages
788
with better performance than the docker-compose TrueCharts App

Looking forward to some evidence for that claim on TrueNAS nightly ;-)
 

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
I've edited the post. I don't have a good setup to benchmark and measure power consumption of the nightlies. But my main server is very energy efficient (consumes around 5W idle). Last time I measured the overhead of k3s was very noticeable... but good to hear that performance (and hopefully idle power consumption) are improving!
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
I've edited the post. I don't have a good setup to benchmark and measure power consumption of the nightlies. But my main server is very energy efficient (consumes around 5W idle). Last time I measured the overhead of k3s was very noticeable... but good to hear that performance (and hopefully idle power consumption) are improving!

Let's agree that the focus is to allow Kubernetes and Apps to continue to work. This provides another Jails-like environment where users also want an resource efficient Linux virtual machine. There is no need to replace the Apps infrastructure that we have.

I'm interested to know whether jails users see all the features they need in this approach.
 

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
Let's agree that the focus is to allow Kubernetes and Apps to continue to work. This provides another Jails-like environment where users also want an resource efficient Linux virtual machine. There is no need to replace the Apps infrastructure that we have.

I'm interested to know whether jails users see all the features they need in this approach.
Haha yes agreed. :smile: The idea is that one could use Apps, 'jails', or both at the same time.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
With jails the root user of the jail does not have to have root access to the system... is it possible to do the same with nspawn?
 

hiro5id

Dabbler
Joined
Aug 21, 2016
Messages
35
Systemd-nspawn may be to SCALE what jails is to CORE and the great thing is: it already works on TrueNAS SCALE!

Some stuff you can do with it:
  • Install packages with apt (inside the jail) without breaking the host
  • Run docker, docker-compose, portainer etc. without the overhead of k3s
  • Bind mount your data pools directly inside the jail, no need for NFS!
  • Possibly replace some of your VMs
  • Use host networking, or macvlan networking for a dedicated IP address via DHCP
  • GPU passthrough
The 'jails' are persistent and survive updates of TrueNAS SCALE if you're using the jailmaker script I'm developing. It's experimental and in early development but has received positive responses from the first testers. I recommend testing it on a non-production system for now.

I'm looking for testers, feedback, collaborators and votes for the request to officially support systemd-nspawn on SCALE. :smile:

!!!’ Run docker, docker-compose, portainer etc. without the overhead of k3s?
YES PLEASE
 

Kushan

Cadet
Joined
Mar 19, 2023
Messages
2
Hey @Jip-Hop , thank you for this script!

I'm currently in the process of building a new server and using TrueNAS for the first time. I'm coming from unRAID where I previously used docker-compose to manage all my docker containers. While experimenting with how TrueNAS handles applications, I'm finding all kinds of unexpected pitfalls and limitations I didn't anticipate (Like not being able to use ports under 9000 or ENV variables being currently broken).

While assessing various workaroudns and options, I came upon this thread and it seems like a pretty clean solution for my needs.

I don't really like using systems against how they were intended to be used, so running docker directly on the host seems like a bad idea. Likewise spinning up a whole VM just to run docker also seems like defeating the purpose. I like the idea of using Systemd-nspawn in the meantime to allow me to essentially use my existing compose file without too much effort. I do eventually plan to move away from compose, but this would let me migrate my server in the short term while I learn the ins and outs of k3s. So thank you for this, it's very useful and has worked well so far with my limited testing.

There's one thing I am unsure of and I'm not even positive how to google the answer, so I am hoping you could help - users and permissions.

Does the "jail" OS inherit the same users and permissions from TrueNAS itself? Or do I need to do something special to avoid containers running as a root user that others can't access and things like that? I'm probably sounding naive, I've just never used Systemd-nspawn before and I'm not entirely sure how it compares to the likes of LXC and traditional VM's.

Thanks again for this!
 

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
Glad it's useful to you. I'm using it solely to run docker myself. I have one jail, installed docker via the official install instructions and it has been running without issues since. No need to create additional users inside the jail. If you bind mount directories from the host into the jail (which you probably want to do), then these files/directories have the same owner and permissions as on the host. You just need to make sure the owner and permissions are appropriate for the docker containers you're using (but that's normal docker stuff). The jail, even though it sits in between the host and the docker containers, doesn't really change anything in terms of users. Hope that makes sense.
 

raidflex

Guru
Joined
Mar 14, 2012
Messages
531
I currently run CORE and have multiple jails split between 3 vlans using bridges in Truenas CORE. Can I accomplish something similar using linux jails? Like installing 3 Linux jails each with docker and assigning each jail to a specific bridge interface.
 

TAG_TEAM

Dabbler
Joined
Mar 18, 2017
Messages
26
I am having difficulties with file volume/mount/bind points and I simply cannot figure it out. I am using JipHop's great invention to run a docker compose jail with portainer. I have done this to run a full install of BAREOS for backing up active web server(s).

I didn't have this problem using the True-Chart docker-compose method, but they did something to break it, so here I am.

I am trying to get the BAREOS jail to write to my root datasets for config, data, pgsql and finally the actual storage. These datsets are laid out under my tank01 dataset.

I have added the --bind settings to those datasets and my path to them in the BAREOS yml file point to those datasets.
Here is where the trouble is. Despite everything I can think to do or change, the docker compose jail wants to refer to its own root as the root of the dataset, so it writes everything inside of the jail (creating the folders and such..) I can't for the life of me figure out how to point its targets to the root /mnt point on my ROOT/ up one level in the tank01 dataset.

Code:
## docker-compose Bareos Director/Storage/Client/webUI and PostgreSQL Database based on Ubuntu
version: '3'
services:
  bareos-dir:
    image: barcus/bareos-director:21-ubuntu-pgsql
    volumes:
      - /mnt/tank01/bareos/config/director:/etc/bareos
      - /mnt/tank01/bareos/data/director:/var/lib/bareos # required for MyCatalog backup
    environment:
      - DB_INIT=true #should be 'true' if bareos db does not exist
      - DB_UPDATE=false
      - DB_HOST=bareos-db
      - DB_PORT=5432
      - DB_NAME=bareos
      - DB_USER=bareos
      - DB_PASSWORD=${DB_PASSWORD} # defined in .env file
      - DB_ADMIN_USER=${DB_ADMIN_USER} # defined in .env file
      - DB_ADMIN_PASSWORD=${DB_ADMIN_PASSWORD} # defined in .env file
      - BAREOS_SD_HOST=bareos-sd
      - BAREOS_SD_PASSWORD=${BAREOS_SD_PASSWORD} # defined in .env file
      - BAREOS_FD_HOST=bareos-fd
      - BAREOS_FD_PASSWORD=${BAREOS_FD_PASSWORD} # defined in .env file
      - BAREOS_WEBUI_PASSWORD=${BAREOS_WEBUI_PASSWORD} # defined in .env file
      - SMTP_HOST=smtpd:8025 # Local smtp container
      #- SENDER_MAIL=your-sender@mail.address #optional
      - ADMIN_MAIL=your@mail.address # Change me!
      # Optional you can gets backup notification via Slack or Telegram
      - WEBHOOK_NOTIFICATION=false # true or false if set to true email notification gets disabled
      - WEBHOOK_TYPE=slack # choose slack or telegram
      - WEBHOOK_URL= # set the slack or telegram URL
      - WEBHOOK_CHAT_ID= # for telegram only set the <chat_id>
    depends_on:
      - bareos-db

  bareos-sd:
    image: barcus/bareos-storage:21-ubuntu
    ports:
      - 9103:9103
    volumes:
      - /mnt/tank01/bareos/config/storage:/etc/bareos
      - /mnt/tank01/bareos/data/storage:/var/lib/bareos/storage
    environment:
      - BAREOS_SD_PASSWORD=${BAREOS_SD_PASSWORD} # defined in .env file

  bareos-fd:
    image: barcus/bareos-client:21-ubuntu
    volumes:
      - /mnt/tank01/bareos/config/client:/etc/bareos
      - /mnt/tank01/bareos/data/director:/var/lib/bareos-director # required for MyCatalog backup
    environment:
      - BAREOS_FD_PASSWORD=${BAREOS_FD_PASSWORD} # defined in .env file
      - FORCE_ROOT=false
      #- PUID=1500 # force bareos user ID
      #- PGID=1500 # force bareos group ID

  bareos-webui:
    image: barcus/bareos-webui:21-ubuntu
    ports:
      - 8080:80
    environment:
      - BAREOS_DIR_HOST=bareos-dir
      - SERVER_STATS=yes
    volumes:
      - /mnt/tank01/bareos/config/webui:/etc/bareos-webui

  bareos-db:
    image: postgres:12
    volumes:
      - /mnt/tank01/bareos/pgsql/data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=${DB_ADMIN_USER} # defined in .env file
      - POSTGRES_PASSWORD=${DB_ADMIN_PASSWORD} # defined in .env file
      - POSTGRES_INITDB_ARGS=--encoding=SQL_ASCII

  smtpd:
    image: devture/exim-relay

  # Optional tools
  # Bareos API
  bareos-api:
    image: barcus/bareos-api:21
    ports:
      - 8000:8000
    environment:
      - BAREOS_DIR_HOST=bareos-dir

  # Bareos metrics exporter for Prometheus
  bareos-exporter:
    image: vierbergenlars/bareos_exporter:v0.6.0
    ports:
      - 9625:9625
    environment:
      - DB_HOST=bareos-db
      - DB_USER=bareos
      - DB_PASSWORD=${DB_PASSWORD}
      - WAIT_FOR_DB=15
    depends_on:
      - bareos-db

#EOF



And for the conf file in the jail I have:

Code:
docker_compatible=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
systemd_nspawn_user_args=--bind='/mnt/tank01/bareos/config/' --bind='/mnt/tank01/bareos/data/' --bind='/mnt/tank01/bareos/pgsql/' --bind='/mnt/tank01/bareos/storage/' --bind='/mnt/tank01/srv/gitlab/config/' --bind='/mnt/tank01/srv/gitlab/data/' --bind='/mnt/tank01/srv/gitlab/logs/'
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collect --setenv=SYSTEMD_NSPAWN_LOCK=0
systemd_nspawn_default_args=--keep-unit --quiet --boot


There could be some syntax errors in that stuff as I have changed it so many times, but the gist of it is there. I have a feeling someone is going to point out something obvious to me that I just hadn't considered. Any help would be greatly appreciated!
 
Joined
Aug 20, 2023
Messages
4
I just want to add my voice to the chorus of those that have used your Jailmaker and are so happy that you shared your knowledge and time to make this script. I never knew about systemd-nspawn and this is exactly what I needed to set up some command line tools directly on my TrueNAS box to get to the bottom of the cause of a performance bottleneck (was it the disk performance, or the network performance?). Thanks to you, I'm able to say categorically it's NOT the disk performance that's the problem in my case.

I highly recommend this to anyone who has been frustrated by the inability to use regular debian packages on their TrueNAS Scale but doesn't want to go the whole VM route, yet is also hesitant about enabling apt-get directly on the host OS. I knew nothing about how to configure and set this up going in, and was able to get it going in a day.

If you're curious about my problem, I posted about it here.

Thanks again Jip-Hop!
 
Last edited:

raidflex

Guru
Joined
Mar 14, 2012
Messages
531
Has there been any changes in the upcoming Cobia release that will affect this script or Systemd-nspawn in general?
 

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
Last time I checked the situation actually improved with the upcoming release because systemd-nspawn is again included by default.
 

Spusuf

Cadet
Joined
Jul 27, 2023
Messages
3
systemd_nspawn_user_args=--bind='/mnt/tank01/bareos/config/' --bind='/mnt/tank01/bareos/data/' --bind='/mnt/tank01/bareos/pgsql/' --bind='/mnt/tank01/bareos/storage/' --bind='/mnt/tank01/srv/gitlab/config/' --bind='/mnt/tank01/srv/gitlab/data/' --bind='/mnt/tank01/srv/gitlab/logs/'
It doesnt appear you posted in the discussion, but just in case you havent figured it out yet, the correct way to bind storage is to do two separated locations beginning with the host (TrueNAS) location followed by the location you want to appear in the jail.
Example: --bind='/mnt/tank/dataset/folder_for_jail:/home/folder_in_jail' In which /mnt/tank/dataset/folder_for_jail is present on the TrueNAS and you want every file in that folder to appear at /home/folder_in_jail in the jail's filesystem.

I'm going to contribute some better documentation to the github soon.
 

JV9

Dabbler
Joined
Aug 19, 2021
Messages
25
Fantastic work Jip-Hop! Exactly what I was looking for.

I'm moving from Core to Scale and really really wanted a lightweight equivalent to Core jails. And I'm not a fan of one-size-fits-all "apps" so I'd much rather roll my own.

I've been testing it in a VM on my laptop to understand how it works (and get some docker experience) before I do it on bare metal. I installed an older scale (22.02.2), then jlmkr & have successfully upgraded to 22.12.4 via ISO and to 22.12.4.2 via the dashboard with no impact on my test jails.

In a post above Jip-Hop noted that "systemd-nspawn is again included by default" - is there a roadmap for scale that notes which direction IX & the devs are leaning with regards to this component?
 
Top