Linux Jails - Experimental Script

[Cellobita]

@Jip-Hop, I can't thank you enough for the work you put into your script; I'm almost done replacing all my Tailscale apps with the equivalent nspawnd jail version. It struck me as preposterous having a constant ~10% CPU usage just to be able to connect remotely to my customers.

 

[Cellobita]

Unrelated question, more of a curiosity, really: why is it that apt runs so slowly inside a jail? I'm trying to install wireguard on a Ubuntu Jammy jail, and the progress is glacial; I have another SSH session connected to the host, and its CPU usage is virtually idle.

 

[Jip-Hop]

I don't know. Apt in the jail works quickly for me.

 

[Cellobita]

Strange indeed; apt install wireguard has been running for something like 45 minutes, and is still at 20% - the apt process barely goes up from 0% CPU, and that for a brief interval, every few minutes. Average load is under 1.0, on a 4-core CPU.

P.S.: monitoring from a second SSH shell, with btop

 

[delgas]

Strange indeed; apt install wireguard has been running for something like 45 minutes, and is still at 20% - the apt process barely goes up from 0% CPU, and that for a brief interval, every few minutes. Average load is under 1.0, on a 4-core CPU.

P.S.: monitoring from a second SSH shell, with btop

I noticed something similar early on in my testing of jlmkr . It turned out to be DNS timeouts for me. Maybe it's something to consider for you?

I have a pihole docker container running in a jail using macvlan networking, and my TrueNAS Scale system can't talk to it. On my DD-WRT router I push pihole as the primary DNS server on my network via DHCP options, and the rest of my clients can talk to it without issue - but not my TrueNAS. This is related to annoying networking configuration issues - I ran TrueNAS on BSD for years and never had to deal with pesky issues like this. Someone earlier in this thread linked a video to set up a bridge to solve this, but it's not a big deal for me to just use my router for DNS on TrueNAS so I haven't looked into it further (using pihole mostly for ad blocking, etc)

Anyway, one of the first things I noticed was super slow apt updates and installs inside jails, and after investigating further, found it to be DNS lookup timeouts. I had statically set the TrueNAS nameserver to what everything else on my network uses - the pihole - and TrueNAS couldn't talk to it, would timeout, and then talk to the secondary nameserver and get a response. I fixed it under network->global configuration - I changed the nameserver to the router (the same IP as the default route for me), and things immediately improved.

May not be your problem, but thought it might be useful to suggest looking in this direction just in case.

-D

 

[Cellobita]

I noticed something similar early on in my testing of jlmkr . It turned out to be DNS timeouts for me. Maybe it's something to consider for you?

I have a pihole docker container running in a jail using macvlan networking, and my TrueNAS Scale system can't talk to it. On my DD-WRT router I push pihole as the primary DNS server on my network via DHCP options, and the rest of my clients can talk to it without issue - but not my TrueNAS. This is related to annoying networking configuration issues - I ran TrueNAS on BSD for years and never had to deal with pesky issues like this. Someone earlier in this thread linked a video to set up a bridge to solve this, but it's not a big deal for me to just use my router for DNS on TrueNAS so I haven't looked into it further (using pihole mostly for ad blocking, etc)

Anyway, one of the first things I noticed was super slow apt updates and installs inside jails, and after investigating further, found it to be DNS lookup timeouts. I had statically set the TrueNAS nameserver to what everything else on my network uses - the pihole - and TrueNAS couldn't talk to it, would timeout, and then talk to the secondary nameserver and get a response. I fixed it under network->global configuration - I changed the nameserver to the router (the same IP as the default route for me), and things immediately improved.

May not be your problem, but thought it might be useful to suggest looking in this direction just in case.

-D

Thank you, it's worth investigating.

Also, I'm not sure of the security implications, but I've been running my systemd-nspawn jails with systemd_nspawn_user_args=--capability=CAP_NET_ADMIN - Tailscale refuses to run otherwise (though it may be simply because I'm not very well versed in this feature, and thus haven't found a safer workaround).

 

[Cellobita]

OK, I'm still testing, but it appears to work OK with SCALE 24.04 RC1 (my initial testing was on 23.10). I'm burned out for today, will resume tomorrow or Wednesday

 

[usergiven]

I know this project is new(ish) to Scale and more testing/feedback is needed but I can say that my initial experience has been extremely positive. As of this past weekend I finalized testing the changes needed to feel comfortable ditching ESXI all together in my home environment. Couple of learning moments along the way. When using TN Scale as a guest under ESXI, creating a bridge failed until I found this post about modifying the network security settings of the host. Obviously TN Scale on bare metal bypasses that. The only other thing I noticed along the way was that Jailmaker complained about using /mnt/tank/jails as an initial spot to run from and wanted to see a jailmaker directory with jlmkr.py in it. I complied however when I reboot Scale I have to manually start the jail each time. I'm assuming this has to do with the location of the jail in a different directory but I'm not sure.

Either way, great work - was able to say goodbye to VMware and Core as well as embrace Scale on bare metal and couldn't be happier. It's been a dream. Thank you!

 

[Stux]

I know this project is new(ish) to Scale and more testing/feedback is needed but I can say that my initial experience has been extremely positive. As of this past weekend I finalized testing the changes needed to feel comfortable ditching ESXI all together in my home environment. Couple of learning moments along the way. When using TN Scale as a guest under ESXI, creating a bridge failed until I found this post about modifying the network security settings of the host. Obviously TN Scale on bare metal bypasses that. The only other thing I noticed along the way was that Jailmaker complained about using /mnt/tank/jails as an initial spot to run from and wanted to see a jailmaker directory with jlmkr.py in it. I complied however when I reboot Scale I have to manually start the jail each time. I'm assuming this has to do with the location of the jail in a different directory but I'm not sure.

Either way, great work - was able to say goodbye to VMware and Core as well as embrace Scale on bare metal and couldn't be happier. It's been a dream. Thank you!

Did you add a startup task to run jlmkr.py startup

 

[Stux]

Did you add a startup task to run jlmkr.py startup

And I think you need to set startup=1 in the config for the jail too.

 

[usergiven]

Startup on post-init as well as the jail configuration were setup as outlined in the documentation but the jail never starts. I'll keep testing it to see if I can figure it out.

 

[Stux]

@Jip-Hop
Can I suggest creating a new thread on the new forums so we can continue discussing jailmaker in a single thread, and leave a forwarding link here?

 

[Jip-Hop]

@Stux Which new forums do you mean?

@usergiven The issue is that the installation instructions in the TrueNAS docs are not ideal. See https://github.com/Jip-Hop/jailmaker/issues/107

If anyone could update those community docs that would be a great help!

 

[skittlebrau]

Thank you, it's worth investigating.

Also, I'm not sure of the security implications, but I've been running my systemd-nspawn jails with systemd_nspawn_user_args=--capability=CAP_NET_ADMIN - Tailscale refuses to run otherwise (though it may be simply because I'm not very well versed in this feature, and thus haven't found a safer workaround).

Tailscale requires CAP_NET_ADMIN capability to be enabled. I found I also had to enable CAP_NET_RAW in order for Tailscale to work.

I stopped using the official TrueNAS app for Tailscale and switched to running it in a nspawn container. It was the last remaining TN app I was running, and the resource usage drop after unsetting the pool in kubernetes was very noticeable.

 

[Stux]

@Stux Which new forums do you mean?

TrueNAS Community Forums

TrueNAS Community Help and Discussion

forums.truenas.com

 

[Etorix]

Nice example of why announcing a forum shutdown with five day notice on April Fools Day is not such a great idea…

 

[Cellobita]

Tailscale requires CAP_NET_ADMIN capability to be enabled. I found I also had to enable CAP_NET_RAW in order for Tailscale to work.

I stopped using the official TrueNAS app for Tailscale and switched to running it in a nspawn container. It was the last remaining TN app I was running, and the resource usage drop after unsetting the pool in kubernetes was very noticeable.

FWIW, it's working perfectly for me without CAP_NET_RAW - I don't have any running as exit nodes, or advertising subnets, perhaps the argument is needed in such cases.

And yes, the Kubernetes overhead was very noticeable, glad to have found an easy workaround in Jailmaker; thanks again, @Jip-Hop

 

[Cellobita]

I have created a placeholder for continuing this thread on the new forum:

Sandboxes (Linux Jails) - Jailmaker Experimental Script

Just a placeholder for continuing the original thread on the old forum, discussing the very useful jailmaker script by Jip-Hop Old thread here Download the script from Jip-Hop’s github page

forums.truenas.com