Where is FreeNAS going with Docker in 2018?

[tofagerl]

Yeah, not many people understand that Docker is really just a piece of software that gathers tons of different kernel functions, hacks and workarounds and makes running containers easy. It was always possible, but Docker made it easier and that got people interested.
Those kernel functions, hacks and workarounds are obviously not all in FreeBSD.

Also, virtualizing the Docker host system is REALLY not that big of a deal. It adds tons of complexity, but most of that is taken care of by the virtualizing layer anyway. The main problem is disk IO, which is mainly solved through using networked filesystems. A good NFS system that works well and doesn't require more configuration than Docker Compose, and we're 99% there imo.

 

[CompuGlobalHyperMegaNet]

The problem isn't Docker itself. That's fairly easy.

The problem is that everything built around Docker is also built around Linux.

Then the obvious solution would be to switch FreeNAS's base OS from FreeBSD to Linux...


/s

 

[rvassar]

Then the obvious solution would be to switch FreeNAS's base OS from FreeBSD to Linux...
/s

Which immediately hits the CDDL / GPL linking problem... And it's not a small problem.

More like: "ZFS or Docker... Choose one..."

-Rob

 

[danb35]

Which immediately hits the CDDL / GPL linking problem... And it's not a small problem.

...which is why no Linux distros ship with ZFS. Oh wait...

 

[rvassar]

...which is why no Linux distros ship with ZFS. Oh wait...

Ahhh.. But has it been tested by Oracle's lawyers and survived? That's the test, and the only one that counts. Being ignored means nothing, that's just a tactic... Oracle is the king of "rent seeking". Ask anyone that's ever done their own OEDB install and lived thru the license audit.

 

[danb35]

But has it been tested by Oracle's lawyers and survived?

Can't answer that one directly--but Proxmox isn't the only Linux OS that ships with ZFS support; Ubuntu does as well (though Proxmox adds the feature of a point-and-click ZFS-on-root installation). Between the two, it seems safe to assume their lawyers have reviewed the respective licenses and given the thumbs-up.

 

[rvassar]

Can't answer that one directly--but

iX has done a portion of this for us. See:

https://www.ixsystems.com/blog/zfs-vs-openzfs/

I'm going to guess the Linux distro's are building the source as some part of the install...

 

[diskdiddler]

I got into a discussion with a guy on Reddit who wanted to take a try at getting Docker working on freebsd. I told him it's been tried 3 times at least already, by random people, if he does try, he should collaborate with others so it doesn't go to waste.


I doubt we'll see native support for 2 or 3 years, if at all.

 

[diskdiddler]

Yeah, not many people understand that Docker is really just a piece of software that gathers tons of different kernel functions, hacks and workarounds and makes running containers easy. It was always possible, but Docker made it easier and that got people interested.
Those kernel functions, hacks and workarounds are obviously not all in FreeBSD.

Also, virtualizing the Docker host system is REALLY not that big of a deal. It adds tons of complexity, but most of that is taken care of by the virtualizing layer anyway. The main problem is disk IO, which is mainly solved through using networked filesystems. A good NFS system that works well and doesn't require more configuration than Docker Compose, and we're 99% there imo.

Pardon my ignorance, but why are people complaining about it then? Is the performance overhead high? Or the choice of Linux VM inappropriate?

I have heard bhyve could be improved, is it due to that? (Reliability, performance?)


Personally, it's not viable until my next FreeNAS system with 32gb and several more cores (I'm hoping at least 6)

 

[Ericloewe]

Bhyve can certainly be improved (and is being improved), but, as I understand it, the complaints mostly focus on the fiddling that is needed to get things to run. Some of it due to bhyve, some of it due to the guest OS.

Yeah, not many people understand that Docker is really just a piece of software that gathers tons of different kernel functions, hacks and workarounds and makes running containers easy. It was always possible, but Docker made it easier and that got people interested.
Those kernel functions, hacks and workarounds are obviously not all in FreeBSD.

I'll note that FreeBSD doesn't require hacks and workarounds, since it actually supports containers (going on almost two decades, in fact) - real secure containers, not the paper prison Docker was limited to on Linux (and perhaps still is). Docker, rather stupidly, was initially written as a rather monolithic thing for Linux, tying the (reputably) good front-end with Linux's under-the-hood fake containers.
Apparently there was some work to modularize it, so that you could have the front-end and a layer that interfaces with jails - basically, you'd end up with a jail manager like iocage. I don't know what the status of that is, but it solves a different problem than the one most people who want Docker want to see solved.

 

[diskdiddler]

Bhyve can certainly be improved (and is being improved), but, as I understand it, the complaints mostly focus on the fiddling that is needed to get things to run. Some of it due to bhyve, some of it due to the guest OS.

Will this fiddling be fixed long term? (is it the plan?) I'd assume yes, that the docker support will end up seamless?
I know it was clearly delayed / re-done entirely after 10.

I will say, I didn't know how ioCage would end up, I was very skeptical if it would actually be done properly or not.
Having now tested nightlies for a couple of months, it seems to me that within a few months Warden will be gone, iocage in it's place and just as easy as it ever was in the past, thank goodness.

Hopefully some 'normies' start contributing to making the iocage freenas plugins, having read some of the code in the scripts, it does (appear) to be easy to make them. We're still stuck at 22 for the time being though.

 

[Ericloewe]

Will this fiddling be fixed long term?

I assume so. Again, the ugly seams of Linux mix with the ugly seams of bhyve, so it's not very simple to unilaterally fix.

 

[dmshimself]

I'm a home/lab user who's happy enough with FreeNAS and rancheros as it stands on a small HP microserver. I'm running mythtv, DHCPD, home brewing software, pi-hole and icecast and they are solid.

But I nearly gave up when I followed the FreeNAS manual. I'm probably getting the terminology wrong here. The base OS image for rancher that FreeNAS installs seems pretty slim and works well. But when you try and follow the FreeNAS manual putting the full rancher management layer on top, it became quite hard (for me) to follow along and I found it easy to loose sight of achieving some wins and simple initial steps. As soon as I used portainer and the regular docker CLI from within rancher, things became much clearer and quicker to deploy.

If the manual provided some first steps for those who were starting off, I suspect a lot more people would get going with docker in FreeNAS, get some benefits and perhaps then come back to the larger Rancher world if they ever needed too.

I have probably missed some excellent tutorials somewhere, but I thought I'd chip in with my 2 NZ cents.

 

[KrisBee]

@dmshimself You're experience reflects my original post where I suggested rancheros +RancherUI was overkill for small docker container deployments. I mentioned portainer too, which continues to improve. The rancheros image used in FreeNAS is now quite out of date ( see: https://github.com/rancher/os/releases ) nor can you easily add docker-compose to a rancheros base. There are numerous posts/tutorials on the forum about using docker containers.

 

[diskdiddler]

Would it be difficult to switch to a new RancherOS image or Portainer Image?

 

[J. Greg Williams]

Definitely feeling frustrated with FreeNAS and the lack of coherent direction. I would very much like an official response to this topics question. Where is FreeNAS going with docker?

<frustration>
I ran Corral up until December of last year and except for some weird UI stuff, was very happy with it's ability to manage docker containers backed by my ZFS pool. This was definitely a step up from the plugins and custom jails I'd been trying to maintain in 9.3.

I am now struggling to get anything working properly, between Bhyve issues preventing me from being able to boot Ubuntu VM's or the fixed version of RancherOS as a Docker VM. There is a definite lack of documentation of how this is supposed to work when not obfuscated by the UI, so I am unable to update the Docker VM myself to get RancherOS 2.0.

I have finally reverted to just using plugins in order to restore some functionality.

Looking back through this thread the links to diffs are all broken and it seems a bunch of bugs have been moved to backlog which seems like a punt and makes me concerned about what is going on.

I think containers make a lot of sense for simple compose-able applications. I think it makes sense to have RancherOS as a base to manage these containers since it handles a lot of the details for you and has a team actively improving it. It's container based itself so it should be easy to pull the latest ISO automatically and just save the RANCHER_STATE to a zvol.

Ultimately the FreeNAS team needs to decide and then convey what is the path forward regarding the middleware layer.

 

[diskdiddler]

I am under the impression it is intended to improve, but I do not know just how well.

From ALL the googling I have, I can almost 100% assure you, there won't be native 100% docker support that's really great and reliable under FreeBSD any time soon, period. It will HAVE to be under a VM.

However, if it worked well enough on FreeNAS 10, also utilising a Bhyve VM, then I'm a bit perplexed, why it can't work just as well, under FN11, using Bhyve again with the right choice of linux VM as the docker host?

J.Greg: I do know that FN 11.1 U5 is a long way behind 11.2, there's some big changes, so perhaps docker support is better on the nightlies (if you're willing to test)

Any more official response from staff?

 

[diskdiddler]

Ok so in the interest of doing some testing.

I have the latest nightly release and I tried the new UI.
So I ran through attempted to set it up and start it.
As soon as I start the VM, it tries to pull down the docker image, it gets to 80% then

"dictionary changed size during iteration" and it stops.

???


The same thing, under the old UI?
It (seems to) successfully download the required file and build the VM
As soon as I try to start the VM it just instantly stops as if there was an error, no message is displayed.


Latest nightly release, as of tonight.

 

[diskdiddler]

Is there a guide to get docker working, simply so I can do some testing? I was surprised to see how broken this appears to be at the moment?

 

[KrisBee]

@diskdiddler

Scroll to the bottom of the side bar to access the latest FreeNAS guide. I’ve seen that unhelpful error message, but I did manage to create a “Docker VM” using the new UI in FreeNAS-11.2-MASTER-201805250605

I am under the impression it is intended to improve, but I do not know just how well.

I’m not sure what makes you says this. As far as I can tell, behind the New UI nothing has fundamentally changed for “Virtual Machines”. Creating a “Docker VM” via the new UI still downloads an out of date 1.1.3 rancheros base which the user cannot upgrade. Hard coding the rancheros version seems to break one of the most basic rules of software development, as I understand it.
(See: /usr/local/lib/python3.6/site-packages/middlewared/plugins/vm.py )

@J. Greg Williams

Unless the implemenation of creating a"Docker VM" changes, the rancheros base will never be user upgradeable. The latest Ubuntu 18.04 LTS does not work in bhyve at the moment (fix availabe but not applied until ?), so you would need to use Ubuntu 16.04 LTS if you want to install RancherUI 2. If you wanted to use rancheros so it is user upgradeable, you'd have to use iohyve at the CLI with a custom config.