Resource icon

FreeNAS 11, RancherOS (Docker), and Portainer

So when FreeNAS 11 came, I updated from 9.10 to get rolling on the new options. Unfortunately this killed my VirtualBox jail that I was using to run some services I couldn't get working in a jail. So I started looking on how to get Docker running under FreeNAS 11 so I could replace those services and get some future proofing for the 11.1 and onward releases that should support Docker out of the box. From the posts on the forums they look to be heading for RancherOS as the underlying Docker operating system.

I originally tried to get this working under the default VM tab in the new interface, but it was not cooperating. I then moved over to using iohyve and bhyve from the command line to get this working. Here are the basic steps I used to get this kicking.

EDIT: There is also a Reddit thread I created about this with some discussion going on if you are interested.
https://www.reddit.com/r/freenas/comments/6bpmjv/howto_freenas_11_rancheros_docker_and_portainer/

Prep

Setup the iohyve pool and options. Select where the pool you want, and the main network card you want attached.
iohyve setup pool=<storage pool> kmod=1 net=<NIC>

Fetch the latest RancherOS iso for the install.
iohyve fetch https://releases.rancher.com/os/latest/rancheros.iso

Should see the ISO you downloaded above listed.
iohyve isolist

Create the VM, and give it a size of disk you need. I used 40G since I wouldn't need tons of space.
iohyve create RancherOS <SIZE>G

Set the VM startup options. I'm keeping this minimal as I don't want to take too many resources. Adjust the ram and cpu as you want.
iohyve set RancherOS loader=grub-bhyve ram=2G cpu=1 con=nmdm0 os=debian

Should see the VM listed now.
iohyve list

RancherOS Install

Kick off the VM and tell it to load the ISO for installation.
iohyve install RancherOS rancheros.iso

Open a new SSH window and run the below command to get a console connection to the new VM. You have to hit enter again to get a prompt.
sudo iohyve console RancherOS

Look, a bootloader! Enter the commands below to get it to actually boot into the installer. Use the TAB key where I say so it will auto-fill in case the version of the installer you downloaded is newer than mine. (Make note of these commands somewhere! You will need them later.)

Code:
grub> set root=(cd0,msdos1)
grub> linux /boot/vm<TAB> ro rancher.password=rancher
grub> initrd /boot/initrd<TAB>
grub> boot


It's alive! Login with the username and password rancher/rancher. Run the below command to edit cloud-config.yml so we can put your SSH key in. Oh yeah, generate a ssh keypair to use for access. On windows I used puTTY to generate my keypair.
[rancher@rancher ~]$ vi cloud-config.yml

This is what the file should look like, it's REALLY picky about spacing. No tabs. Enter it just like this.

Code:
# cloud-config
ssh_authorized_keys:
- ssh-rsa <your pubkey here>


Now we can kick off the installer to write everything to the VM. It likely won't reboot at the end, and thats ok.
sudo ros install -c cloud-config.yml -d /dev/sda

Make sure the VM actually stopped, need to make a couple changes.
iohyve list

This is to fix the booting issues.
iohyve set RancherOS os=custom
sudo iohyve set RancherOS boot=1


Go to where you setup the pool, and you should see an iohyve folder and inside there a folder for this VM. (ie /mnt/iohyve/RancherOS) You should see a device.map file, edit that and remove everything but the line for hd0.

Create a file called grub.cfg and enter the stuff below, chaning it to match what you used before when you booted the VM. If this stuff ever gets updated, you will have to do the same.

Code:
set root=(hd0,1)
linux /boot/vmlinuz-* printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait console=tty0
initrd /boot/initrd-*
boot


Set some tuneable entries for FreeNAS to start this VM on a reboot. All of these are rc.local types. Use your network interface obviously.
Code:
ng_ther_load = YES
iohyve_flags = kmod=1 net=<NET>
iohyve_enable = YES


Reboot everything and see if it comes up. You can use the console command up above to get back in and the prompt should show you what the outside IP is of the VM. I then used my router to setup a static DHCP lease for the VM so I could get moving forward. Then I setup SSH with the private key so I could SSH into the VM.

There are some good docs for RancherOS. Check them out, I used them to get all this working. Read them to get a good overview.
https://docs.rancher.com/os/

Now, RancherOS is meant to pair with Rancher, their own GUI for docker. However when I tried it, it was rather overkill for what I was wanting. So I had heard of Portainer and decided to give that a try.

Portainer and NFS Storage

So, Portainer, NFS, and persistent storage for the containers. This is all specific to my configuration, but it should work anywhere. (I think.)

Go create a user for the docker containers on FreeNAS. Use:
Code:
UserID 1101
Username: Docker
Check the box to make a group
Home: /nonexistant
Shell: nologin
Full Name: Docker Containers
Check Disable Password Login


Now, setup a storage location for the configurations of the containers. I created a dataset called docker under my main pool. So: /MainStorange/docker was mine.

After you make it, edit the permissions for it. I used Apply Owner for user and group and set them both to docker, permission type was unix, and I set it recursively.

Now we need to share it. I went to the sharing tab and then selected UNIX (NFS) up top. Create a new share and click advanced.

Code:
Path: The path you made above, so mine was /mnt/MainStorage/docker
Authorized IP addresses or hosts: I entered the IP of the VM here, since no one else will need this.
Mapall User and Mapall Group: I set these both to docker and left it.
I know this might be a security issue, but damn me if I could get the permissions playing nicely otherwise.

Now to setup NFS and portainer back in the VM.

So here is the file I used to configure my NFS share for the config files. SPACES MATTER! NO TABS! You need to be root to create this file, use whatever text editor you want. File name/location is on the first line. Edit the SERVER and SHARE lines at the bottom to match your FreeNAS and the share you made above.

Code:
#/var/lib/rancher/conf/cloud-config.d/nfs-config.yml
write_files:
  - path: /etc/rc.local
  permissions: "0755"
  content: |
  #!/bin/bash
  [ ! -e /usr/bin/docker ] && ln -s /usr/bin/docker.dist /usr/bin/docker
rancher:
  services:
	nfs-config:
	  image: d3fk/nfs-client
	  labels:
		io.rancher.os.after: console, preload-user-images
		io.rancher.os.scope: system
	  net: host
	  privileged: true
	  restart: always
	  volumes:
		- /usr/bin/iptables:/sbin/iptables:ro
		- /mnt/config:/mnt/config:shared
	  environment:
		SERVER: 192.168.1.20
		SHARE: /mnt/MainStorage/docker
		MOUNTPOINT: /mnt/config

Now we need another file for portainer. Change the TZ below if needed, this is Eastern.

Code:
#/var/lib/rancher/conf/cloud-config.d/portainer.yml
rancher:
  services:
	portainer:
	  image: portainer/portainer
	  container_name: "portainer"
	  environment:
		PGID: '1101'
		PUID: '1101'
		TZ: America/New_York
	  labels:
		io.rancher.os.after: console, preload-user-images
	  privileged: true
	  restart: always
	  ports:
		- "9000:9000"
	  volumes:
		- /var/run/docker.sock:/var/run/docker.sock
		- /mnt/config/portainer:/data

Type reboot and it should restart. After it comes back, you should be able to hit the IP:9000 and see portainer.

Set a password, and it will ask if you want to use a remote docker system or the local one. Select local.

I used some of the following posts/threads to figure a lot of this out.

https://gist.github.com/s1lvester/c082027208bbfb6c80c8d6f5bbf6f047
https://www.cod3r.com/2017/01/docker-in-freenas-9-10/
https://forums.freenas.org/index.php?threads/rancheros.53769/

So, its running, now what?

I needed to get my services moved over from my VM. Mainly, I was running stuff that needed access from the outside world to the inside, or stuff than didn't play nice in jails

Currently for me that was nginx, Ombi, Jackett, and some personal pages. I'll give examples on how I granted access to my other shares and Jackett.

For the other shares I fudged things a bit. The NFS containers all talk about the ability to do multiple mounts, or perhaps even do some kind of setupw here I just run the NFS container to give the system access to
the NFS protocol, and mount things elsewhere. This didn't work for me. I'm sure it CAN work, but I wanted this at least stumbling along. So I just created another NFS service container for my other main dataset. The file I did it with is this:

Code:
# /var/lib/rancher/conf/cloud-config.d/nfs-vault.yml
write_files:
  - path: /etc/rc.local
  permissions: "0755"
  content: |
  #!/bin/bash
  [ ! -e /usr/bin/docker ] && ln -s /usr/bin/docker.dist /usr/bin/docker
rancher:
  services:
	nfs-vault:
	  image: d3fk/nfs-client
	  labels:
		io.rancher.os.after: console, preload-user-images
		io.rancher.os.scope: system
	net: host
	privileged: true
	restart: always
	volumes:
	   - /usr/bin/iptables:/sbin/iptables:ro
	   - /mnt/vault:/mnt/vault:shared
	environment:
	  SERVER: 192.168.1.20
	  SHARE: /mnt/MainStorage/Vault
	  MOUNTPOINT: /mnt/vault


There are a couple points above. Obviously this is for my setup, you need to change the SERVER, SHARE, and MOUNTPOINT. But if you notice in a difference from the main post, the entry under services is nfs-vault, not nfs-config. This
is so it would launch a new NFS container and provide access to my particular main dataset. If I needed to do more, I would just clone this file and name it something else, nfs-moreshares, or whatever.

I know this is kludgy, and it likely uses more resources than another method, but it does work. This gets me working until 11.1 and a more elegant solution.

For Jackett, I originally set things up from the command line by hand. I just got it up and running and was happy. But containers are ephemeral and I was concerned about it disappearing at some point, and a service would do better.

This is the command line version I used.

Code:
docker create \
--name=jackett \
-v /mnt/config/jackett:/config \
-v /mnt/config/jackett/downloads:/downloads \
-e PGID=1101 -e PUID=1101 \
-e TZ=America/New_York \
-p 9117:9117 \
linuxserver/jackett


This is what I replaced it with. I used the same location as the services above, /var/lib/rancher/conf/cloud-config.d and created jackett.yml in there. RancherOS should auto-load it at boot.

Code:
rancher:
  services:
	jackett:
	  image: linuxserver/jackett
	  container_name: "jackett"
	  environment:
		PGID: '1101'
		PUID: '1101'
		TZ: America/New_York
	  restart: always
	  labels:
		io.rancher.os.after: portainer
	  ports:
		- "9117:9117"
	  volumes:
		- /mnt/config/jackett:/config
		- /mnt/config/jackett/downloads:/downloads

Couple things on this one, you can see where I map the persistent storage for the container down under volumes. I make sure to put the name of the app on the host side so things stay clean. Other services would work the same way, for example Ombi has /mnt/config/ombi as the first part. I also have this set to not start until after portainer comes up.
  • Like
Reactions: danjng
Author
SaskiFX
Views
3,280
First release
Last update
Rating
0.00 star(s) 0 ratings
Top