Docker Options & FreeNAS

IOSonic

Explorer
Joined
Apr 26, 2020
Messages
54
Hello,

I am evaluating several FOSS NAS solutions for home surveillance, file storage, and containerization (specifically, Docker). Although my desire for native docker support is a priority for my build, FreeNAS offers so many other great features that I do still want to fully understand my options for getting Docker running in FreeNAS before making a decision. I am looking for a low-maintenance solution that will not require too many workarounds, survive updates and will likely be supported by iXsystems for some time to come.

I understand iXsystems removed support for the popular containerization OS RancherOS in the most current build of 11.3. Does anyone know why? My understanding is that, in attempting to run this VM in 11.3, you will have to make some modification to the boot loader, which is not supported by FreeNAS. I don't want to go this route, because of sustainability issues -- it may be difficult to get help if I run into problems running a "hacky" VM implementation that was never supposed to work on FreeNAS, and I speculate that these modifications would likely have to be repeated or break something during later OS updates.

This brings me to my actual questions: first, is my logic for not wanting to modify and run RancherOS sound -- is there any information I am missing that may help me make a more informed decision? Secondly, can anyone think of any foreseeable issues with opting for a different OS like CoreOS?

RancherOS would be ideal -- while I have plenty of RAM (32GB), I have a low-powered Celeron 9400 that is probably not suited to virtualization tasks, so lightweight would be possible. However, I think I might be willing to pop-in a beefier CPU if I get a stable Docker solution that doesn't require a lot of ongoing maintenance.

I'm interested to hear your insight! Thanks in advance!
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,555
foreseeable issues with opting for a different OS like CoreOS?
FreeNAS isn’t an operating system, it’s an appliance built on top of FreeBSD. You get a very nice web UI, effortless system backups and restore and a big box of tools. Opting for a vanilla OS instead is opting to do all the heavy lifting yourself in running a complex server. If docker is a must, and there is no other host for it, then bhyve can manage docker hosts, nothing has been deliberately removed. Just that iX stopped maintaining one themselves. I run my containers in an Ubuntu VM, looking at moving to something else now in these strange times, but haven’t yet. But I prefer to run as much as possible in jails, just to skip the hassle of storage over NFS, I prefer nullfs any day of the week
 

IOSonic

Explorer
Joined
Apr 26, 2020
Messages
54
Hi Garm. Thanks for replying.

FreeNAS isn’t an operating system, it’s an appliance built on top of FreeBSD. You get a very nice web UI, effortless system backups and restore and a big box of tools.

Yes, I am very much aware on both counts :)!

Allow me to clarify: my question was about the OS I should run within FreeNAS' bhyve virtualization engine to host docker containers. I see that you are using Ubuntu. Many seem very intent on running RancherOS, in spite of the obstacles to making it run. I wonder why that is, when they could just as easily use CoreOS, Ubuntu, etc. My best guess is the small footprint of RancherOS.

Even if one decides to modify the RancherOS bootloader to run it, is it sustainable? Every time one upgrades RancherOS, will they need to modify the bootloader or some part of FreeNAS to make it all work?

If I stick with FreeNAS, there are certainly many jails I wish to install, but there are also some things I certainly prefer or must run in Docker. The only question is the best way to do it. You see, I spend my entire day doing IT for a living, so at home, I would prefer things to work as easily as possible.

... just to skip the hassle of storage over NFS, I prefer nullfs any day of the week

It sounds like storage is presented from Docker to FreeNAS over NFS? Is this the only option? By experience, I have learned to avoid it whenever possible, just like you hahaha.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Is this the only option?

From a FreeNAS to the VM running Docker, actually. NFS seems the sanest option. You could also do SMB, but that seems less well suited.
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
With respect to RancherOS being deprecated in 11.3, I believe it's mostly because RancherOS 1.5.5 from Dec 30, 2019 is the last public build, and RancherOS itself is being EoL'ed upstream. See https://rancher.com/support-maintenance-terms/:

RancherOS

VERSIONGAEOMEOL
1.5.xDecember 29, 2018December 29, 2020June 29, 2021
1.4.xMay 30, 2018May 30, 2019November 30, 2019
1.3.xMarch 27, 2018March 27, 2019September 27, 2019

Instead, RancherLabs' new hotness for a minimalist Docker environment is k3os. However, this would require installation with Docker as the container back-end, instead of containerd. Unfortunately, k3os hasn't achieved 1.x status yet, and I wouldn't trust it for production for a while.

As for myself, I was able to get RancherOS 1.5.5 working in 11.3, but it was a month-long struggle to work around the limitations of the RancherOS installer and syslinux boot loader, which requires using a FreeNAS VM configured for grub boot. See my post in the Jails and bhyve forum, titled grub boot HOW-TO, to get an idea of the gyrations I had to go through.

On a positive note, RancherOS 1.5.x supports VirtIO disks, so I created a 2 GB RAW file hosting the install image, and made that the boot disk for the RancherOS VM, connected via VirtIO as /dev/vda. I also created a 25 GB zvol as the 2nd disk for the VM, again connected via VirtIO as /dev/vdb. The zvol is partitioned into /dev/vdb1, which is a 4 GB swap partition labeled as RANCHER_SWAP, and a 20 GB Docker root partition, /dev/vdb2. Lastly, I added a VirtIO NIC.

Since RancherOS uses syslinux, you'll have to manually create a grub.cfg that emulates the syslinux boot string. Mine is as follows:

Code:
set timeout=0
set default=rancheros

menuentry "RancherOS" --id rancheros {
  set root='hd0,msdos1'
  linux /boot/vmlinuz-4.14.138-rancher printk.devkmsg=on panic=10 rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait rancher.resize_device=/dev/vda
  initrd /boot/initrd-v1.5.5
}


Within the VM, I set the mounts via a YAML file in /var/lib/rancher/conf/cloud-config.d:

Code:
mounts:
- - LABEL=RANCHER_SWAP
  - ""
  - swap
  - ""
- - /dev/vdb2
  - /var/lib/docker/mnt
  - ext4
  - ""
rancher:
  docker:
    graph: /var/lib/docker/mnt


Note, for Docker, I don't perform NFS mounts using the YAML file, but I create local driver Docker volumes using NFS mounts instead:

Code:
docker volume create --driver=local --opt type=nfs --opt o=addr=<FreeNAS NFS IP>,rw,local_lock=all --opt device=:<FreeNAS NFS share> <Docker volume name>


This primarily only works for NFS folders that are mostly read-only, as the locking semantics for NFS aren't great for write-heavy applications. For volumes requiring reliable locking, Docker volumes using the local driver are sufficient, as this creates an ext4-backed folder within the Docker root:

Code:
docker volume create <Docker volume name>
 
Last edited:

LeoSum

Dabbler
Joined
Dec 13, 2015
Messages
36
I would like to throw in the quite new Fedora CoreOS (FCOS) as an idea here. If I understand correctly it is the successor of Container Linux as well as Project Atomic and is somehow developed in parallel to the other CoreOS (without Fedora). Anyway, FCOS is UEFI bootable and quite easy to install:
  • grab the image from here and boot from it as CD in a VM: https://getfedora.org/coreos/download?tab=metal_virtualized&stream=stable
  • create your fcc file to generate an ignition file from it (I found this quite tricky as there doesn't seem to be a lot of documentation on it, but you can use mine as a starter)
  • use VNC to connect to your VM, then pull the ign from your webserver and use it to install to your virtio disk
    Code:
    curl -LO http://192.168.178.21:8000/fcos.ign
    sudo coreos-installer install /dev/vda --ignition fcos.ign
  • then reboot and after a while (all docker images need to be pulled first) portainer should appear on your CoreOS VMs DHCPed IP on Port 9000, e.g.
    Code:
    http://192.168.178.22:9000
So here's my fcc file. Maybe this saves somebody the headache that I had to figure out the format:
It installs as systemd units:
  • docker-compose (which you don't really need if you want to use portainer anyway)
  • an nfs-client to mount your nfs-share
  • portainer to avoid the commandline of CoreOS in the future as much as possible and
  • watchtower to automatically upgrade your containers if an update to their image appears
The idea is to use that nfs-share for all containers. I create subfolders which I pass on into the containers as a volume, as can be seen for the portainer container. All further containers are created via portainer.

Code:
variant: fcos
version: 1.0.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ecdsa-sha2-nistp521 AAAA... #your ssh key goes here
systemd:
  units:
  - name: install-docker-compose.service
    enabled: true
    contents: |-
      [Unit]
      Description=Install docker-compose
      Wants=network-online.target
      After=network-online.target network.target nss-lookup.target
      ConditionPathExists=!/opt/bin/docker-compose

      [Service]
      Type=oneshot
      RemainAfterExit=yes
      ExecStart=/usr/bin/mkdir -p /opt/bin/
      ExecStart=/usr/bin/curl -sL "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-Linux-x86_64" -o /opt/bin/docker-compose
      # In theory the following would fetch the latest binary, but >1.24 has missing python dependency, I'll keep it here for the future, who knows:
      # wrapped inside a bash command as it won't work as ExecStart command directly:
      # ExecStart=/bin/bash -c '/usr/bin/curl -sL `/usr/bin/curl -s "https://api.github.com/repos/docker/compose/releases/latest" | /usr/bin/jq -r .assets[].browser_download_url | /usr/bin/grep Linux | /usr/bin/grep -m 1 x86_64` -o /opt/bin/docker-compose'
      ExecStart=/usr/bin/chmod +x /opt/bin/docker-compose

      [Install]
      WantedBy=multi-user.target
  - name: docker.nfs.service
    enabled: true
    contents: |-
      [Unit]
      Description=NFS Client Container for Shared Volume
      After=docker.service
      Requires=docker.service network.target network-online.target

      [Service]
      TimeoutStartSec=0
      Restart=always
      ExecStartPre=-/usr/bin/docker stop %n
      ExecStartPre=-/usr/bin/docker rm %n
      ExecStartPre=/usr/bin/docker pull d3fk/nfs-client
      # Privileged mode is required due to SELINUX (https://github.com/portainer/portainer/issues/849)
      ExecStart=/usr/bin/docker run --rm --privileged=true --name %n --net=host -v /mnt/shared_nfs:/mnt/nfs-1:shared -e SERVER=192.168.1.2 -e SHARE=/mnt/ssd_data/CoreOSdata d3fk/nfs-client
      ExecStop=/usr/bin/docker stop -t 15 %n

      [Install]
      WantedBy=multi-user.target
  - name: docker.portainer.service
    enabled: true
    contents: |-
      [Unit]
      Description=Portainer Admin Container
      After=docker.service docker.nfs.service
      Requires=docker.service docker.nfs.service

      [Service]
      Type=oneshot
      RemainAfterExit=yes
      TimeoutStartSec=0
      ExecStartPre=-/usr/bin/docker stop %n
      ExecStartPre=-/usr/bin/docker rm %n
      ExecStartPre=/usr/bin/docker pull portainer/portainer
      ExecStart=-/usr/bin/mkdir -p /mnt/shared_nfs/portainer_data
      # Privileged mode is required for binding to local socket to work due to SELINUX (https://github.com/portainer/portainer/issues/849)
      ExecStart=/usr/bin/docker run --privileged=true -d -p 9000:9000 --name %n --restart always -v /var/run/docker.sock:/var/run/docker.sock -v /mnt/shared_nfs/portainer_data:/data portainer/portainer --no-analytics --host=unix:///var/run/docker.sock
      ExecStop=/usr/bin/docker stop -t 15 %n

      [Install]
      WantedBy=multi-user.target
  - name: docker.watchtower.service
    enabled: true
    contents: |-
      [Unit]
      Description=Watchtower Container
      After=docker.service
      Requires=docker.service

      [Service]
      Type=oneshot
      RemainAfterExit=yes
      TimeoutStartSec=0
      ExecStartPre=-/usr/bin/docker stop %n
      ExecStartPre=-/usr/bin/docker rm %n
      ExecStartPre=/usr/bin/docker pull containrrr/watchtower
      ExecStart=/usr/bin/docker run --privileged -d --name %n -v /var/run/docker.sock:/var/run/docker.sock:z containrrr/watchtower
      ExecStop=/usr/bin/docker stop -t 15 %n

      [Install]
      WantedBy=multi-user.target
 
Last edited:

Leninux

Cadet
Joined
Aug 24, 2017
Messages
2
Thanks LeoSum for this instructions! Very helpful!

I've used your fcc file, but I've made two changes:
  • Portainer failed to start with this error: /usr/bin/docker: Conflicting options: --restart and --rm.. Removing the --rm flag everything works fine. I've checked in the portainer install docs, and this flag is not used. Why do you want to remove the container? I'm missing something?
  • I've removed docker-compose service and everything stills works fine.
Thanks again!
 

LeoSum

Dabbler
Joined
Dec 13, 2015
Messages
36
You are right, docker-compose isn't needed if one uses portainer and can be removed. It was just my first try before I discovered portainer. And I figured I'd leave it in there since it was such a struggle and some people might want it.

And of course you are also right about the rm flag, it's not
in my actual setup. I edited my previous post. Thanks for catching it.
 
Last edited:

Formica

Cadet
Joined
Dec 16, 2020
Messages
1
Hello,

@LeoSum @Leninux I am curious if you have any feedback or follow up on your experience with using Fedora CoreOS as a Docker host VM on TrueNAS as described here over the ensuing couple of months. I am mostly a Windows guy, but I have modest Linux experience, primarily in the Debian space. So I am a little leery of dipping my toes in the Fedora waters after having done most of my work on the Debian/Ubuntu side of things.

That said, what is described here pretty much fits my use case. I was going to build a home server with lots of disk, running Ubuntu 20.04 and Docker with KVM or Xen, or maybe VMware ESXi. But I recently considered going with TrueNAS Core and OpenZFS instead, and using bhyve to host a Linux VM to do my Docker stuff that way instead. I don't intend to virtualize Windows (probably), so I feel like bhyve should do just fine for my purposes.

Am I on the right track? Given the simplicity of what I am going for, would I probably do just fine hosting Ubuntu or Debian instead? Performance isn't really a huge concern; I will probably have a Ryzen 3000 series processor and at least 32GB of DDR4 with an M.2 SSD system disk. Thank you for your time and consideration!
 

LeoSum

Dabbler
Joined
Dec 13, 2015
Messages
36
Hello,

@LeoSum @Leninux I am curious if you have any feedback or follow up on your experience with using Fedora CoreOS as a Docker host VM on TrueNAS as described here over the ensuing couple of months.
I have been using it since I posted here. It is still working very well with no real issues. The only hassle is to get data into and out of FCOS VM and the containers. I would like to have persistent data (configurations, etc.) stored on a ZFS dataset. For some containers it works to hand it in via nfs, but most times it doesn't due to permission issues. So the data is stored inside the VM which I don't think is ideal. But that is really my only issue.

I am mostly a Windows guy, but I have modest Linux experience, primarily in the Debian space. So I am a little leery of dipping my toes in the Fedora waters after having done most of my work on the Debian/Ubuntu side of things.
After setting it up via the ignition file you don't really have to do much on the FCOS command line if you use portainer.

That said, what is described here pretty much fits my use case. I was going to build a home server with lots of disk, running Ubuntu 20.04 and Docker with KVM or Xen, or maybe VMware ESXi. But I recently considered going with TrueNAS Core and OpenZFS instead, and using bhyve to host a Linux VM to do my Docker stuff that way instead. I don't intend to virtualize Windows (probably), so I feel like bhyve should do just fine for my purposes.
I really only use docker for quickly trying out stuff or running services that don't run natively on FreeNAS TrueNAS. Everything else is better run in a jail, as this lets you directly access ZFS storage and also produces less overhead than Docker. It is a little harder to manually set up though (FreeBSD is a little different from Debian/Ubuntu), but also gives you more control. There are also ready-to-use plugins (basically preconfigured jails) for many services.

Am I on the right track? Given the simplicity of what I am going for, would I probably do just fine hosting Ubuntu or Debian instead? Performance isn't really a huge concern; I will probably have a Ryzen 3000 series processor and at least 32GB of DDR4 with an M.2 SSD system disk. Thank you for your time and consideration!
Hardware sounds fine I guess, but depends on how many services you want to host. If you really plan to dockerize everything and don't want to get into FreeBSD, maybe go bare metal FCOS. An Ubuntu Server should also be fine. However for me the main advantages, that I would always choose TrueNAS again for, are jails and the ZFS filesystem.
 

Leninux

Cadet
Joined
Aug 24, 2017
Messages
2
I am curious if you have any feedback or follow up on your experience with using Fedora CoreOS as a Docker host VM on TrueNAS as described here over the ensuing couple of months.

Like LeoSum, I've been using the FCOS VM without issues since then, but I haven't upgrade yet my setup to TrueNAS neither Portainer v2. The only issue I've had is the lack of memory ram in my setup (16Gb), the machine crashes more or less twice a month, but after a reboot everything works fine again. I've more than 15 containers running (Traeffik, HAproxy, Nginx, PHP-FPM, MariaDB, Heimdall, NextCloud, Emby, Grafana, Influxdb...) and another Ubuntu Server VM with Sandstorm. For me, the main advantage of using Docker is the lifecycle of the app, the jails usually have outdated versions of the app and the libraries.

Am I on the right track? Given the simplicity of what I am going for, would I probably do just fine hosting Ubuntu or Debian instead? Performance isn't really a huge concern; I will probably have a Ryzen 3000 series processor and at least 32GB of DDR4 with an M.2 SSD system disk. Thank you for your time and consideration!

Remember to check your hardware with the official recommendations (ECC RAM, VT-d/AMD-Vi and AES-NI CPU...).

I have been using it since I posted here. It is still working very well with no real issues. The only hassle is to get data into and out of FCOS VM and the containers. I would like to have persistent data (configurations, etc.) stored on a ZFS dataset. For some containers it works to hand it in via nfs, but most times it doesn't due to permission issues. So the data is stored inside the VM which I don't think is ideal. But that is really my only issue.

I've all the persistent data stored on a ZFS dataset, but wasn't easy; most docker images works fine if in Portainer configuration the user/group ids are the same used in the host.
 

arnecl

Cadet
Joined
Feb 5, 2021
Messages
1
Thanks @LeoSum for your hints!

Unfortunately, I'm experiencing some issues with FCOS >32 (see Fedora CoresOS IssueTracker #735) which gets stuck in a boot-loop after install
Failed to start Ignition OSTree: Generate filesystem UUID (boot) on TrueNAS 12.0U1.1.
Can anyone reproduce the failure described there?

Workaround for now is staying on FCOS 31.20200517.3.0 (fedora-coreos-31.20200517.3.0-live.x86_64.iso) which works fine for me with VirtIO as well as AHCI Disks.
 

LeoSum

Dabbler
Joined
Dec 13, 2015
Messages
36
Thanks @LeoSum Can anyone reproduce the failure described there?

Sorry, but I currently don't have the time to mess around with my setup much. I plan to upgrade TrueNAS and Portainer in a few months, so maybe then I can also try to move to newer FCOS versions. Hopefully your issue will be resolved by then though.

I've all the persistent data stored on a ZFS dataset, but wasn't easy; most docker images works fine if in Portainer configuration the user/group ids are the same used in the host.

@Leninux could you elaborate on that user/group id configuration a bit? I really couldn't figure out how to do this.
 

MrFollies

Cadet
Joined
Nov 3, 2016
Messages
4
I've all the persistent data stored on a ZFS dataset, but wasn't easy; most docker images works fine if in Portainer configuration the user/group ids are the same used in the host.

Via NFS?

I'm quite experienced with running docker for lots of little things (testing websites and small web apps) etc...
But I've always rolled my own with ubuntu or amazon linux. Mostly using docker-compose files.

I'm keen to give FCOS or core.os a go to see how it works. I believe the idea is that you have the ignition file that will configure the FCOS instance and then boot up the wanted docker containers....

Is it possible to put all of this into a local git repo so you can achieve that? Including the config for things like portainer etc...
 
Top