Where is FreeNAS going with Docker in 2018?

[jclendineng]

FWIW corral was amazing for me. I know it was buggy but my goodness docker worked so well at least for me. I hated it when docker got taken away, at least the ease at which i could spin up a docker instance. Ive gotten used to bhyve, and like it just as much now but docker would be nice...though id have to get more cores :D

 

[diskdiddler]

Well of course it's possible, people have been doing this since iohyve appeared in FN9 and searching the forum will show you the different approaches. Good luck with the rest ...

If the FreeNAS (FreeBSD) Base OS and bhyve system is powerful enough to be reliable and run a linux build, consistently / reliably / fast enough, which can do docker 'right' (within reason, for a VM of course) then it's eventually possible for the developers to shoe-horn this into the OS UI as a feature.

They've added it as it is, it's just not a good linux build.
Perhaps FreeNAS 11.2 will come with a better quality bhyve and 11.3 might get a new linux VM

What do I know, it's possible, we'll see, with enough badgering.

 

[NetSoerfer]

I'm currently fiddling with a new docker-only setup (no rancher, kubernetes, etc.) on a minimal debian in a bhyve VM, using NFS shares for persisting data. I'm writing up a step-by-step as I go, which I will post here once I'm done.

First version's up: https://forums.freenas.org/index.ph...ebian-minimal-in-a-uefi-bhyve-vm-with-nfs.97/

 

[fletom]

as someone who really likes FreeNAS and wants to see it succeed, reading this thread is rather saddening.

I believe that without real Docker support FreeNAS will slowly die. Docker is too powerful, popular, and convenient to be an optional feature on a server platform.

with equal conviction I also believe that any implementation of Docker that does not provide seamless filesystem integration (i.e. the ability to mount locations on FreeNAS volumes inside containers the way every other docker host works), is a non-starter. my understanding of a regular VM-based solution is that it makes this impossible or prohibitively difficult, which makes it a waste of time. it seems like a solution designed by a person or a group of people at iXsystems who are tasked with implementing nominal "Docker support" for marketing purposes, but who never actually plan on using it. on a data storage server, people simply want tools that can easily access their data.

I hope I don't insult anyone who has been working on the RancherOS stuff, but I genuinely think the best path forward would be to scrap that and wait/hope until someone gets "Docker Done Right", as in the original blog post, up and running. that may include lobbying the Docker project for more interoperability, or furthering work on the docker-freebsd port, or using the same kind of seamless virtualization technique that Docker for macOS and Windows uses.

just my two cents.

 

[diskdiddler]

Based on the replies in this thread and the activity I've seen on the bugs board, I would, in the very least, guess that docker isn't as much a priority as iocage / jails being fixed up.

Perhaps 11.3 will be the docker release.

One thing, for 100% certain in their defense, is that the FreeBSD devs up the chain, have not got docker working locally.
Several people have tried, multiple times, there's no official, clean working, up to date, docker support for FreeBSD natively, none. Not that I can find. Nor can I even find a (recent) experimental branch which is working well.

If it's ever going to work seamlessly in FreeNAS, it *has* to be done in a VM, at this point in time and seemingly for the future.

 

[Ericloewe]

It's important to keep in mind that getting Docker to work is of very limited use - all you'd get is a jails manager.
None of the images (or whatever the terminology) out there would work, because they're Linux applications. Getting those to work is a whole different can of worms.

 

[jclendineng]

Agreed, FreeNAS is not going to try something that is a failure upstream. If you want docker, ditch freenas and use proxmox or the like. IMO more than a jail manager since you can run isolated services in separate namespaces for multiple uses, but like you say it is linux only. The draw of BSD is the security and jails. Linux has LXC, so its really up to what you need. BHYVE is still very beta but its great and going to get greater. Almost everything can be run using jails, IMO a great, baremetalish solution that just works when done right. Anything you need linux for (not a lot) can be done in a VM.

 

[diskdiddler]

It's important to keep in mind that getting Docker to work is of very limited use - all you'd get is a jails manager.
None of the images (or whatever the terminology) out there would work, because they're Linux applications. Getting those to work is a whole different can of worms.

If you got it working under a VM though, it does give you access to a very wide range of docker containers though, yes? I mean some of them are quite useful.
Example, Pi-Hole (we don't have a plugin for that) - I believe that's been 'dockered' also, "Hass.IO" the home automation software. Both of those wouldn't even need any weird NFS / Samba or other tricky systems to talk back to the data.

So from where I sit, it seems that there's still uses for docker support, eventually. (I refuse, until I have a system which can afford to run a 2 or 4GB VM in the background, just for 'fun' indefinitely)

 

[Patrick M. Hausen]

with equal conviction I also believe that any implementation of Docker that does not provide seamless filesystem integration (i.e. the ability to mount locations on FreeNAS volumes inside containers the way every other docker host works), is a non-starter.

Agree on the filesystem integration. But keep in mind that Docker is not the only container technology out there and does have serious problems precisely because of the tight tie-in with the underlying Linux platform.
Jails provide all the functionality needed. In fact we build our entire hosting data centre on jails.
Search for Bryan Cantrill on YT if you like a more in-depth discussion of where Docker falls short.

Kind regards
Patrick

 

[adrianwi]

"Hass.IO" the home automation software.

I've got this running in a jail with Configurator (so you can edit .yaml files through the WUI) and HA Dashboard (for creating simple menus for tablets, although used App Daemon which is useful for other things too) using a very simple iocage script. Here's a link to the HA forum > https://community.home-assistant.io...s-home-assistant-appdaemon-configurator/54834

 

[diskdiddler]

I've got this running in a jail with Configurator (so you can edit .yaml files through the WUI) and HA Dashboard (for creating simple menus for tablets, although used App Daemon which is useful for other things too) using a very simple iocage script. Here's a link to the HA forum > https://community.home-assistant.io...s-home-assistant-appdaemon-configurator/54834

Hah interesting timing.
I was just working on learning how to make a plugin for this (I'm well stuck)

I found this guys post
https://github.com/home-assistant/home-assistant/issues/11231#issuecomment-362826076

and I went down a rabbit hole of mess, pkg install python, python27, python34, sqlite, etc etc in an attempt to solve the problem. (I ended up getting errors about ""No module named '_sqlite3'"")

So I got half way there, I was hoping to write a plugin for the sake of learning. In the very least, that guys post should be mirrord over here.

 

[adrianwi]

Still haven't really got my head around FreeNAS 11.2 plugins, but think the iocage scripts are brilliant! Are the plugins not just a clickable button way of running a script? I was running HA in a Docker container on an Ubuntu VM in FreeNAS, but when I found this create a jail in a few minutes that provides much more than I'd managed in Docker! Only took a little whilst to copy across the .yaml configuration so the jail does everything the container was. Anyway, probably a discussion for another thread...

 

[Ericloewe]

If you got it working under a VM though, it does give you access to a very wide range of docker containers though, yes?

In a Linux VM or something that convincingly pretends to be Linux, yes.

 

[diskdiddler]

In a Linux VM or something that convincingly pretends to be Linux, yes.

Well have faith, once I have an 8 core, 32GB system, 2 cores and 6GB isn't gonna bother me in the slightest, just need the Rancher OS kept up to date and the tools to manage it.

 

[diskdiddler]

Dumb question:

When you add a VM to freenas and can choose regular or docker, is there any actual difference?

Is it just a template for a specific low power VM for the Docker VM, or does it have some different ties to the system to making a normal VM?

I'm going to have a freenas machine capable of throwing 2 CPUs and 6gb of ram "away" on this to play with indefinitely.

 

[diskdiddler]

Dumb question:

When you add a VM to freenas and can choose regular or docker, is there any actual difference?

Is it just a template for a specific low power VM for the Docker VM, or does it have some different ties to the system to making a normal VM?

I'm going to have a freenas machine capable of throwing 2 CPUs and 6gb of ram "away" on this to play with indefinitely.

It's been approximately a month, seriously does anyone know the difference here?

I'm having some issues with Rancher, about to re-install it from scratch and I know, for a fact, FreeNAS will pull down 1.4.1 - 1.4.2 is out.
Couldn't I just manually create a VM, use the RancherOS iso for 1.4.2 and be up to date?

I genuinely don't know what the difference is with a "VM" or "DockerVM" - besides it automatically pulling down Rancher?

 

[warllo]

It's been approximately a month, seriously does anyone know the difference here?

I'm having some issues with Rancher, about to re-install it from scratch and I know, for a fact, FreeNAS will pull down 1.4.1 - 1.4.2 is out.
Couldn't I just manually create a VM, use the RancherOS iso for 1.4.2 and be up to date?

I genuinely don't know what the difference is with a "VM" or "DockerVM" - besides it automatically pulling down Rancher?

@diskdiddler

If you want to run docker containers on FreeNAS the FreeNAS way you use the Rancher VM.

If you want to run a VM download whatever supported os you want any use it.

Rancher is simply a specialized distro for docker.

 

[diskdiddler]

@diskdiddler

If you want to run docker containers on FreeNAS the FreeNAS way you use the Rancher VM.

If you want to run a VM download whatever supported os you want any use it.

Rancher is simply a specialized distro for docker.

Sorry!

No, I realise that, 10,000% - I'm not quite that dumb :)

My question is, what difference within Byhve and the FreeNAS OS is there, are some other processes running? Less processes running? I know the Rancher 'mode' for a VM will specifically make a /specific type/ of VM - but is there ANY shortcoming, in manually creating my own DockerVM instead?

 

[KrisBee]

It's been approximately a month, seriously does anyone know the difference here?

I'm having some issues with Rancher, about to re-install it from scratch and I know, for a fact, FreeNAS will pull down 1.4.1 - 1.4.2 is out.
Couldn't I just manually create a VM, use the RancherOS iso for 1.4.2 and be up to date?

I genuinely don't know what the difference is with a "VM" or "DockerVM" - besides it automatically pulling down Rancher?

I'm sure I've said repeatedly in this thread that the FreeNAS devs do not appear to kept watch on what is the latest rancheros iso (currently 1.4.2). It's only ever been updated after people raise the issue.

If it were possible to create a standard VM directly from a rancheros iso, why would the FreeNAS devs needed to have created a separate "docker VM" function? So the answer to your question about using a rancheros iso is no, it is not possible. See my post near the beginning of this thread at #11. A rancheros iso does not support UEFI, it is "syslinux" based and does not contain a grub config via which the bhyve grub-loader could load its linux kernel.

The FreeNAS implementation treats a "docker VM" like a linux embedded system combining a "kernel" , "initrd" and "root filesystem". See what you end with in the hidden dataset <your pool name>/.bhyve_containers

Just keep watch on the the middlewared log ( e.g. tail -f /var/log.middlewared.log) as you create a "docker VM" and a "normal VM" and you will get some idea of what is going on in the background.

Personally, for small scale docker deployment, I don't see any obvious drawback to creating a standard linux VM in which you can install docker, docker compose and for example portainer for docker management. The base OS may not be so slim as rancheros, but IMHO would be easier to understand & manage and importantly can be updated by the user. Rancher is now recommending 1280 MB of rancheros and once you add the RancherUI, memory requirements grow nearer to 4GB. So using rancher is hardly light on resources.

Another question is how to configure access to your pool data. In Rancher the intention seems to be to use NFS only, although CIFS is possible too. You have to work out how to do this via yaml config files, creating network shares on FreeNAS which are then mounted inside rancheros. The principal is the same within a standard linux VM running docker, but you can just use the standard linux syntax to mount the network shares. As always, care must be taken to provide the correct user/group access to data and whether its rw or ro only.

If you know your way around cloud-init ( I don't) then perhaps rancheros offers other advantages. But latest ubuntu supports cloud-int and was being recommended earlier in the year by Rancher for use with their RancherUI version2 , which is designed for kubernetes use. I don't see that as the realm of the typical home user.

 

[brando56894]

Personally, for small scale docker deployment, I don't see any obvious drawback to creating a standard linux VM in which you can install docker, docker compose and for example portainer for docker management. The base OS may not be so slim as rancheros, but IMHO would be easier to understand & manage and importantly can be updated by the user. Rancher is now recommending 1280 MB of rancheros and once you add the RancherUI, memory requirements grow nearer to 4GB. So using rancher is hardly light on resources.

Totally agree, I was just about to post this. I've been checking in on the development of FreeNAS over the past year, and since I virtualize it to test it, I have never been able to get RancherOS to work 100% because apparently the serial console doesn't work when virtualized, I can't send any input.

I'm currently using Linux and Portainer on my server and Portainer makes things so much easier. I have written a few docker-compose scripts as well, so I can migrate my entire setup from one OS to another in a few minutes.

I'll probably be switching back to FreeNAS soon since 11.3 contains most of the features I'm waiting for and especially since I saw recently that FreeBSD is rebasing their ZFS code on ZFSonLinux, which was going to be an issue for me before because I couldn't migrate my pool without destroying it.