LXC integration in Scale at KVM levels of comfort?

K

Kailee71

Contributor
Joined
Jul 8, 2018
Messages
110
Hi all,

As a long-time user of FreeNAS, then Core, the advent of Scale is like a dream come true for me. Currently I am using Core as a VM on ESXi (don't worry, proper LSI hardware with passthough, host is a DL560 G8 with 2x 2690 v2, 256Gb 1866 DDR3 whilst testing), and have additional VMs on the same ESXi to provide compute capability (CFD and rendering). I have done some quick & dirty benchmarks and Scale is looking very promising;

mesh (s)mesh (relative)sim (s)sim (relative)
Focal on Bare metal235190.21
VM on ESXi2450.9695.40.95
Proxmox KVM3130.751740.52
Proxmox LXC2490.9495.20.95
Scale KVM3620.65207.50.43
Scale Docker2460.96103.00.88
Scale Native2301.0296.80.93

Higher numbers in the (relative) columns mean better performance. Proxmox LXC performance is virtualy identical to natively installing solvers on Scale (which I would like to avoid for obvious reasons). So my question is - are there any plans to integrate LXC containers with the same kind of comfort as KVM VMs in Scale? This would enable me to reduce complexity of this setup both in hardware and software (ditch ESXi, simplify booting, no need for passthough, slog devices for NFS VMs etc etc).

Any pointers more than welcome!

Kai.
 
K

Kailee71

Contributor
Joined
Jul 8, 2018
Messages
110
Yes what I would really appreciate is the "full os experience" provided by LXC in contrast to the "one app" experience provided by Docker & Co. It's both way easier to design and deploy, and also makes use much more flexible. At the end of the day what I need is seperately usable complete Linux instances with minimal impact on achievable performance. Think renderfarm/compute nodes. Just like proxmox provides right now, but I just need Truenas. I'm not sure what I should do in Jira. The value of a "me too" post there probably tends to 0.
 
Arwen

Arwen

MVP
Joined
May 17, 2014
Messages
3,611
@Kailee71 There is a "Vote for this issue" place on the Jira feature request. So, add your vote! No need for a comment of "Me too".
 
K

Kailee71

Contributor
Joined
Jul 8, 2018
Messages
110
Ooh I missed that - vote added. Thanks for the heads up Arwen!
 
J

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
Perhaps you'd like to take a look at my jailmaker.sh script. It uses systemd-nspawn instead of LXC to create a "seperately usable complete Linux instances with minimal impact on achievable performance" and doesn't alter the host OS at all (systemd-nspawn is istalled by default). My script is still experimental so I recommend you to try it out in a fresh TrueNAS SCALE VM.
 
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
Jip-Hop said:
Perhaps you'd like to take a look at my jailmaker.sh script. It uses systemd-nspawn instead of LXC to create a "seperately usable complete Linux instances with minimal impact on achievable performance" and doesn't alter the host OS at all (systemd-nspawn is istalled by default). My script is still experimental so I recommend you to try it out in a fresh TrueNAS SCALE VM.
Click to expand...
Interesting idea... Could you compare doing that with running the docker-compose App from TrueCharts? What are the benefits of each?
 
J

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
The big benefit of TrueCharts docker-compose app is that works within the Apps ecosystem iXsystems is offering. So we can be quite sure it won't be going away any time soon. The downside compared to running docker natively or with systemd-nspawn is that it has worse performance.

But besides performance, systemd-nspawn also has flexibility going for it. I can install and run docker-compose inside systemd-nspawn (then it will fulfill a similar role as the docker-compose app), but I don't have to use Docker inside it. The other day I was playing around with LXC inside my systemd-nspawn 'jail' and was able to boot a Debian container. Didn't have time to play with it more.

Have you ever wanted to install something on TrueNAS SCALE (with apt)? It's Debian under the hood, so technically you can, but you shouldn't. For me systemd-nspawn fixes this. I can now have as many environments (jails) as I like, with direct access to all my files in my pools with bind mounts, and install software and follow many tutorials written for plain Debian, without messing up the TrueNAS host OS. :)

So in that sense it would be better to compare it to VMs, which SCALE already has. VMs have better isolation, which can be good for security purposes. But it's more resource intensive and sharing files has to be done over the network: NFS for best performance (which is definitely not as good as bind mounts). And not all software will work properly with NFS mounted storage... So there systemd-nspawn (or something similar like LXC) would be a huge win in my opinion. :)

By the way, I think using systemd-nspawn can be complementary to Apps and doesn't interfere with it (but haven't verified this compatibility).
 
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
Jip-Hop said:
The big benefit of TrueCharts docker-compose app is that works within the Apps ecosystem iXsystems is offering. So we can be quite sure it won't be going away any time soon. The downside compared to running docker natively or with systemd-nspawn is that it has worse performance.

But besides performance, systemd-nspawn also has flexibility going for it. I can install and run docker-compose inside systemd-nspawn (then it will fulfill a similar role as the docker-compose app), but I don't have to use Docker inside it. The other day I was playing around with LXC inside my systemd-nspawn 'jail' and was able to boot a Debian container. Didn't have time to play with it more.

Have you ever wanted to install something on TrueNAS SCALE (with apt)? It's Debian under the hood, so technically you can, but you shouldn't. For me systemd-nspawn fixes this. I can now have as many environments (jails) as I like, with direct access to all my files in my pools with bind mounts, and install software and follow many tutorials written for plain Debian, without messing up the TrueNAS host OS. :)

So in that sense it would be better to compare it to VMs, which SCALE already has. VMs have better isolation, which can be good for security purposes. But it's more resource intensive and sharing files has to be done over the network: NFS for best performance (which is definitely not as good as bind mounts). And not all software will work properly with NFS mounted storage... So there systemd-nspawn (or something similar like LXC) would be a huge win in my opinion. :)

By the way, I think using systemd-nspawn can be complementary to Apps and doesn't interfere with it (but haven't verified this compatibility).
Click to expand...

If everything is working well, please think about what would be needed for it to withstand reboots and software updates. By all means make it a suggestion and provide a link. Other users can the upvote as needed.
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
iX did promise Linux containers from the beginning as something that would be available... not high on the priority list though.

If this has delivered an easy way to do it (albeit without GUI support), great!

If some folks can do a bit of testing and suggest its inclusion as a feature request, maybe we can get some votes behind it and make it happen.

The cases I can think of is where complex passthrough to a VM doesn't really work with the options currently available in the GUI. Assuming a container can overcome that, it's at very least a temporary workaround to have something able to run.
 
J

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
Sounds good to me!

sretalla said:
maybe we can get some votes behind it and make it happen.
Click to expand...

By "make it happen" are you referring to GUI support of systemd-nspawn (Linux containers)? Because, as of right now, it's already happening with the included tools (albeit without GUI support).

morganL said:
think about what would be needed for it to withstand reboots and software updates.
Click to expand...

These issues have been solved in my script. Everything is stored on a data pool, so nothing is lost on upgrades, and the container can be started on reboot automatically by using a Post-Init command. My script currently includes an 'up' command which starts all the containers made with it (although I called them jails).
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
i like the name linux jails....can each jail can get its own Ip address?
 
mistermanko

mistermanko

Guru
Joined
Jan 27, 2020
Messages
577
Yes, please make it happen! Voted!
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
You can already try it out at the CLI with jailmaker linked in #6.
 
J

Jip-Hop

Contributor
Joined
Apr 13, 2021
Messages
118
morganL said:
i like the name linux jails....can each jail can get its own Ip address?
Click to expand...
They sure can! The jailmaker.sh script has been updated to allow creating jails with their own IP address by using the macvlan option. A macvlan interface is a virtual interface that adds a second MAC address to an existing physical Ethernet link. The IP address is obtained automatically via DHCP. The wizard will ask you to specify an ethernet network interface to create a macvlan interface from, which you can leave blank if you wish to use host networking instead.

It's also possible to set fixed IP addresses without using DHCP, or use a dedicated network interface. Check out the systemd-nspawn Networking Options for more info.
 
M

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
Jip-Hop said:
They sure can! The jailmaker.sh script has been updated to allow creating jails with their own IP address by using the macvlan option. A macvlan interface is a virtual interface that adds a second MAC address to an existing physical Ethernet link. The IP address is obtained automatically via DHCP. The wizard will ask you to specify an ethernet network interface to create a macvlan interface from, which you can leave blank if you wish to use host networking instead.

It's also possible to set fixed IP addresses without using DHCP, or use a dedicated network interface. Check out the systemd-nspawn Networking Options for more info.
Click to expand...
Do you think its ready for others to try the script?
When so, perhaps start a thread around "Linux Jails - Experimental" - get some feedback.
 
You must log in or register to reply here.
Top