Resource icon

[HOWTO] FreeNAS 11, RancherOS (Docker), and Portainer

SaskiFX

Dabbler
Joined
Mar 18, 2015
Messages
27
So when FreeNAS 11 came, I updated from 9.10 to get rolling on the new options. Unfortunately this killed my VirtualBox jail that I was using to run some services I couldn't get working in a jail. So I started looking on how to get Docker running under FreeNAS 11 so I could replace those services and get some future proofing for the 11.1 and onward releases that should support Docker out of the box. From the posts on the forums they look to be heading for RancherOS as the underlying Docker operating system.

I originally tried to get this working under the default VM tab in the new interface, but it was not cooperating. I then moved over to using iohyve and bhyve from the command line to get this working. Here are the basic steps I used to get this kicking.

Prep

Setup the iohyve pool and options. Select where the pool you want, and the main network card you want attached.
iohyve setup pool=<storage pool> kmod=1 net=<NIC>

Fetch the latest RancherOS iso for the install.
iohyve fetch https://releases.rancher.com/os/latest/rancheros.iso

Should see the ISO you downloaded above listed.
iohyve isolist

Create the VM, and give it a size of disk you need. I used 40G since I wouldn't need tons of space.
iohyve create RancherOS <SIZE>G

Set the VM startup options. I'm keeping this minimal as I don't want to take too many resources. Adjust the ram and cpu as you want.
iohyve set RancherOS loader=grub-bhyve ram=2G cpu=1 con=nmdm0 os=debian

Should see the VM listed now.
iohyve list

RancherOS Install

Kick off the VM and tell it to load the ISO for installation.
iohyve install rancheros rancheros.iso

Open a new SSH window and run the below command to get a console connection to the new VM. You have to hit enter again to get a prompt.
sudo iohyve console rancheros

Look, a bootloader! Enter the commands below to get it to actually boot into the installer. Use the TAB key where I say so it will auto-fill in case the version of the installer you downloaded is newer than mine. (Make note of these commands somewhere! You will need them later.)

Code:
grub> set root=(cd0,msdos1)
grub> linux /boot/vm<TAB> ro rancher.password=rancher
grub> initrd /boot/initrd<TAB>
grub> boot


It's alive! Login with the username and password rancher/rancher. Run the below command to edit cloud-config.yml so we can put your SSH key in. Oh yeah, generate a ssh keypair to use for access. On windows I used puTTY to generate my keypair.
[rancher@rancher ~]$ vi cloud-config.yml

This is what the file should look like, it's REALLY picky about spacing. No tabs. Enter it just like this.

Code:
# cloud-config
ssh_authorized_keys:
- ssh-rsa <your pubkey here>


Now we can kick off the installer to write everything to the VM. It likely won't reboot at the end, and thats ok.
sudo ros install -c cloud-config.yml -d /dev/sda

Make sure the VM actually stopped, need to make a couple changes.
iohyve list

This is to fix the booting issues.
iohyve set RancherOS os=custom
sudo iohyve set RancherOS boot=1


Go to where you setup the pool, and you should see an iohyve folder and inside there a folder for this VM. (ie /mnt/iohyve/RancherOS) You should see a device.map file, edit that and remove everything but the line for hd0.

Create a file called grub.cfg and enter the stuff below, chaning it to match what you used before when you booted the VM. If this stuff ever gets updated, you will have to do the same.

Code:
set root=(hd0,1)
linux /boot/vmlinuz-4.9.24-rancher printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait console=tty0
initrd /boot/initrd-v1.0.1
boot


Set some tuneable entries for FreeNAS to start this VM on a reboot. All of these are rc.local types. Use your network interface obviously.
Code:
ng_ther_load = YES
iohyve_flags = kmod=1 net=<NET>
iohyve_enable = YES


Reboot everything and see if it comes up. You can use the console command up above to get back in and the prompt should show you what the outside IP is of the VM. I then used my router to setup a static DHCP lease for the VM so I could get moving forward. Then I setup SSH with the private key so I could SSH into the VM.

There are some good docs for RancherOS. Check them out, I used them to get all this working. Read them to get a good overview.
https://docs.rancher.com/os/

Now, RancherOS is meant to pair with Rancher, their own GUI for docker. However when I tried it, it was rather overkill for what I was wanting. So I had heard of Portainer and decided to give that a try.

Portainer and NFS Storage

So, Portainer, NFS, and persistent storage for the containers. This is all specific to my configuration, but it should work anywhere. (I think.)

Go create a user for the docker containers on FreeNAS. Use:
Code:
UserID 1101
Username: Docker
Check the box to make a group
Home: /nonexistant
Shell: nologin
Full Name: Docker Containers
Check Disable Password Login


Now, setup a storage location for the configurations of the containers. I created a dataset called docker under my main zvol. So: /MainStorange/docker was mine.

After you make it, edit the permissions for it. I used Apply Owner for user and group and set them both to docker, permission type was unix, and I set it recursively.

Now we need to share it. I went to the sharing tab and then selected UNIX (NFS) up top. Create a new share and click advanced.

Code:
Path: The path you made above, so mine was /mnt/MainStorage/docker
Authorized IP addresses or hosts: I entered the IP of the VM here, since no one else will need this.
Mapall User and Mapall Group: I set these both to docker and left it.
I know this might be a security issue, but damn me if I could get the permissions playing nicely otherwise.

Now to setup NFS and portainer back in the VM.

So here is the file I used to configure my NFS share for the config files. SPACES MATTER! NO TABS! You need to be root to create this file, use whatever text editor you want. File name/location is on the first line. Edit the SERVER and SHARE lines at the bottom to match your FreeNAS and the share you made above.

Code:
#/var/lib/rancher/conf/cloud-config.d/nfs-config.yml
write_files:
  - path: /etc/rc.local
  permissions: "0755"
  content: |
  #!/bin/bash
  [ ! -e /usr/bin/docker ] && ln -s /usr/bin/docker.dist /usr/bin/docker
rancher:
  services:
	nfs-config:
	  image: d3fk/nfs-client
	  labels:
		io.rancher.os.after: console, preload-user-images
		io.rancher.os.scope: system
	  net: host
	  privileged: true
	  restart: always
	  volumes:
		- /usr/bin/iptables:/sbin/iptables:ro
		- /mnt/config:/mnt/config:shared
	  environment:
		SERVER: 192.168.1.20
		SHARE: /mnt/MainStorage/docker
		MOUNTPOINT: /mnt/config

Now we need another file for portainer. Change the TZ below if needed, this is Eastern.

Code:
#/var/lib/rancher/conf/cloud-config.d/portainer.yml
rancher:
  services:
	portainer:
	  image: portainer/portainer
	  container_name: "portainer"
	  environment:
		PGID: '1101'
		PUID: '1101'
		TZ: America/New_York
	  labels:
		io.rancher.os.after: console, preload-user-images
	  privileged: true
	  restart: always
	  ports:
		- "9000:9000"
	  volumes:
		- /var/run/docker.sock:/var/run/docker.sock
		- /mnt/config/portainer:/data

Type reboot and it should restart. After it comes back, you should be able to hit the IP:9000 and see portainer.

Set a password, and it will ask if you want to use a remote docker system or the local one. Select local.

I used some of the following posts/threads to figure a lot of this out.

https://gist.github.com/s1lvester/c082027208bbfb6c80c8d6f5bbf6f047
https://www.cod3r.com/2017/01/docker-in-freenas-9-10/
https://forums.freenas.org/index.php?threads/rancheros.53769/
 

SaskiFX

Dabbler
Joined
Mar 18, 2015
Messages
27
So, its running, now what?

I needed to get my services moved over from my VM. Mainly, I was running stuff that needed access from the outside world to the inside, or stuff than didn't play nice in jails

Currently for me that was nginx, Ombi, Jackett, and some personal pages. I'll give examples on how I granted access to my other shares and Jackett.

For the other shares I fudged things a bit. The NFS containers all talk about the ability to do multiple mounts, or perhaps even do some kind of setupw here I just run the NFS container to give the system access to
the NFS protocol, and mount things elsewhere. This didn't work for me. I'm sure it CAN work, but I wanted this at least stumbling along. So I just created another NFS service container for my other main dataset. The file I did it with is this:

Code:
# /var/lib/rancher/conf/cloud-config.d/nfs-vault.yml
write_files:
  - path: /etc/rc.local
  permissions: "0755"
  content: |
  #!/bin/bash
  [ ! -e /usr/bin/docker ] && ln -s /usr/bin/docker.dist /usr/bin/docker
rancher:
  services:
	nfs-vault:
	  image: d3fk/nfs-client
	  labels:
		io.rancher.os.after: console, preload-user-images
		io.rancher.os.scope: system
	net: host
	privileged: true
	restart: always
	volumes:
	   - /usr/bin/iptables:/sbin/iptables:ro
	   - /mnt/vault:/mnt/vault:shared
	environment:
	  SERVER: 192.168.1.20
	  SHARE: /mnt/MainStorage/Vault
	  MOUNTPOINT: /mnt/vault


There are a couple points above. Obviously this is for my setup, you need to change the SERVER, SHARE, and MOUNTPOINT. But if you notice in a difference from the main post, the entry under services is nfs-vault, not nfs-config. This
is so it would launch a new NFS container and provide access to my particular main dataset. If I needed to do more, I would just clone this file and name it something else, nfs-moreshares, or whatever.

I know this is kludgy, and it likely uses more resources than another method, but it does work. This gets me working until 11.1 and a more elegant solution.

For Jackett, I originally set things up from the command line by hand. I just got it up and running and was happy. But containers are ephemeral and I was concerned about it disappearing at some point, and a service would do better.

This is the command line version I used.

Code:
docker create \
--name=jackett \
-v /mnt/config/jackett:/config \
-v /mnt/config/jackett/downloads:/downloads \
-e PGID=1101 -e PUID=1101 \
-e TZ=America/New_York \
-p 9117:9117 \
linuxserver/jackett


This is what I replaced it with. I used the same location as the services above, /var/lib/rancher/conf/cloud-config.d and created jackett.yml in there. RancherOS should auto-load it at boot.

Code:
rancher:
  services:
	jackett:
	  image: linuxserver/jackett
	  container_name: "jackett"
	  environment:
		PGID: '1101'
		PUID: '1101'
		TZ: America/New_York
	  restart: always
	  labels:
		io.rancher.os.after: portainer
	  ports:
		- "9117:9117"
	  volumes:
		- /mnt/config/jackett:/config
		- /mnt/config/jackett/downloads:/downloads

Couple things on this one, you can see where I map the persistent storage for the container down under volumes. I make sure to put the name of the app on the host side so things stay clean. Other services would work the same way, for example Ombi has /mnt/config/ombi as the first part. I also have this set to not start until after portainer comes up.
 

SaskiFX

Dabbler
Joined
Mar 18, 2015
Messages
27
I just want to say how awesome it was for you to the time to document and post this information.

Hopefully it's helpful for someone else. I knew nothing about Docker other than "it runs containers?" before I started working on this. I'm looking forward to what the team does with docker in 11.1.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,367
set
Code:
root=(hd0,1)
linux /boot/vmlinuz-4.9.24-rancher printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait console=tty0 
initrd /boot/initrd-v1.0.1 boot

You can use wild cards, so vmlinuz* and initrd* will ensure when the next kernel update comes out your grub.cfg will still work.
 

Kleimo

Cadet
Joined
May 22, 2017
Messages
4
Hi, i have been trying to get this to install on my freenas build but i only get this error:

Code:
@freenas:/iohyve # iohyve fetch https://releases.rancher.com/os/latest/rancheros.iso
Fetching https://releases.rancher.com/os/latest/rancheros.iso...
/iohyve/ISO/rancheros.iso  7% of  54 MB 4288 kBps 00m12s
fetch: /iohyve/ISO/rancheros.iso: No space left on device


I have been searching for answer but seems there are any? Any suggestions?
 
Last edited:

SaskiFX

Dabbler
Joined
Mar 18, 2015
Messages
27
When you setup iohyve with :
iohyve setup pool=<storage pool> kmod=1 net=<NIC>

Where did you put the storage pool, and does it have any space left? :) I used my primary zvol to store everything.
 

Kleimo

Cadet
Joined
May 22, 2017
Messages
4
When you setup iohyve with :
iohyve setup pool=<storage pool> kmod=1 net=<NIC>

Where did you put the storage pool, and does it have any space left? :) I used my primary zvol to store everything.
Primary pool also with 2,9TiB free space.
 

SaskiFX

Dabbler
Joined
Mar 18, 2015
Messages
27
That's strange then. When you look under the storage tab in FreeNAS you should see the iohyve dataset, and under that a dataset called Firmware and ISO. Does the free space look odd there?

Try the command again with sudo if you are not root?
 

Kleimo

Cadet
Joined
May 22, 2017
Messages
4
That's strange then. When you look under the storage tab in FreeNAS you should see the iohyve dataset, and under that a dataset called Firmware and ISO. Does the free space look odd there?

Try the command again with sudo if you are not root?
I can see those folders and 2,7TiB available. Running it root already.
 

SaskiFX

Dabbler
Joined
Mar 18, 2015
Messages
27
I did some googling myself, and I'm stuck. I had no issues running the command listed. Maybe a restart to make sure iohyve setup correctly?
 

Kleimo

Cadet
Joined
May 22, 2017
Messages
4
I did some googling myself, and I'm stuck. I had no issues running the command listed. Maybe a restart to make sure iohyve setup correctly?
I think I now know my problem source. Just have to figure out how to increase the /mnt size?

Code:
@freenas# df -h /mnt
Filesystem  Size  Used  Avail Capacity  Mounted on
tmpfs  4.0M  8.0K  4.0M  0%  /mnt


edit: I think i solved it. Changed iohyve mount away from /mnt. Now i am able to proceed with the instructions!
 
Last edited:

Dan Sherman

Cadet
Joined
Apr 28, 2014
Messages
5
Thanks for writing this up!

I found I needed to add # cloud-config as the first line of the .yml files, but after that it's working great.
 

SaskiFX

Dabbler
Joined
Mar 18, 2015
Messages
27
Thanks for writing this up!

I found I needed to add # cloud-config as the first line of the .yml files, but after that it's working great.

You know, some of my files have that and some don't. So when I copied and pasted the files for this tutorial, I left them out thinking they were un-needed or pointless. Now I wonder.
 

xmicks

Dabbler
Joined
Apr 28, 2016
Messages
33
Thanks for the writeup SaskiFX! but I cant get rancher to boot.

Since my freenas box crashed on upgrade I am with a fresh install and tried to get this working both om FreeNAS-11.0-RC3 and on latest 9.10.2-u4

I've got to the point of the yml file. (dont know if i got the ssh key part correct) and then added the grubfile, but when i then try to startup rancher with
iohyve start rancheros
it shows these error messages(and it did too in previously when installing)

Code:
[root@FreeNAS] ~# iohyve start rancheros
Starting rancheros... (Takes 15 seconds for FreeBSD guests)
[root@FreeNAS] ~# GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console rancheros in second terminal to complete GRUB process...
rdmsr to register 0x34 on vcpu 1
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88


When i then go to a second terminal it asks me logincredentials but when I press return, they don't work:
Code:

			   ,		, ______				 _				 _____ _____TM
  ,------------|'------'| | ___ \			   | |			   /  _  /  ___|
 / .		   '-'	|-  | |_/ /__ _ _ __   ___| |__   ___ _ __  | | | \ '--.
 \/|			 |	|   |	// _' | '_ \ / __| '_ \ / _ \ '__' | | | |'--. \
   |   .________.'----'   | |\ \ (_| | | | | (__| | | |  __/ |	| \_/ /\__/ /
   |   |		|   |	 \_| \_\__,_|_| |_|\___|_| |_|\___|_|	 \___/\____/
   \___/		\___/	 Linux 4.9.24-rancher

		 RancherOS v1.0.1 rancher ttyS0
		 docker-sys: 172.134.42.32 eth0: 10.0.8.140 lo: 127.0.0.1
rancher login: rancher
rancher: Command not found.
[root@FreeNAS] ~# rancher
rancher: Command not found.
[root@FreeNAS] ~# rancher
rancher: Command not found.

when I go ssh to the rancher-ip I cant login with rancher/rancher: "Access denied"

Does somebody know what I did wrong? I would really appreciate it



Thanks
 

silverback

Contributor
Joined
Jun 26, 2016
Messages
134
Thanks for the writeup SaskiFX! but I cant get rancher to boot.

Since my freenas box crashed on upgrade I am with a fresh install and tried to get this working both om FreeNAS-11.0-RC3 and on latest 9.10.2-u4

I've got to the point of the yml file. (dont know if i got the ssh key part correct) and then added the grubfile, but when i then try to startup rancher with
iohyve start rancheros
it shows these error messages(and it did too in previously when installing)

Code:
[root@FreeNAS] ~# iohyve start rancheros
Starting rancheros... (Takes 15 seconds for FreeBSD guests)
[root@FreeNAS] ~# GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console rancheros in second terminal to complete GRUB process...
rdmsr to register 0x34 on vcpu 1
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88


When i then go to a second terminal it asks me logincredentials but when I press return, they don't work:
Code:

			   ,		, ______				 _				 _____ _____TM
  ,------------|'------'| | ___ \			   | |			   /  _  /  ___|
/ .		   '-'	|-  | |_/ /__ _ _ __   ___| |__   ___ _ __  | | | \ '--.
\/|			 |	|   |	// _' | '_ \ / __| '_ \ / _ \ '__' | | | |'--. \
   |   .________.'----'   | |\ \ (_| | | | | (__| | | |  __/ |	| \_/ /\__/ /
   |   |		|   |	 \_| \_\__,_|_| |_|\___|_| |_|\___|_|	 \___/\____/
   \___/		\___/	 Linux 4.9.24-rancher

		 RancherOS v1.0.1 rancher ttyS0
		 docker-sys: 172.134.42.32 eth0: 10.0.8.140 lo: 127.0.0.1
rancher login: rancher
rancher: Command not found.
[root@FreeNAS] ~# rancher
rancher: Command not found.
[root@FreeNAS] ~# rancher
rancher: Command not found.

when I go ssh to the rancher-ip I cant login with rancher/rancher: "Access denied"

Does somebody know what I did wrong? I would really appreciate it



Thanks


Well, this helped for me:

$ ssh -i /path/to/private/key rancher@<ip-address>

From the rancheros Docs.

So once you have installed rancheros and are no longer using the iso, you need to ssh directly into your vm
The Path to your private key will be unique to the system you are running to connect.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,466
I created a dataset called docker under my main zvol.
You're using this term a couple of times in this thread, and I think you're using it incorrectly. A zvol is a pseudo-block device that you can create inside a pool or a dataset. I think you mean "pool" where you're using the term "zvol".

I think you also have a couple of places where you're confusing rancheros and RancherOS, unless iohyve is case-insensitive.

But aside from that, thanks! Following these instructions, I was able to get RancherOS and portainer up and running. Now to figure out what they are, and why everybody's so in love with Docker...
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
This would be good to have in the Resources section. Could you please create a new Resource with the content you have here? Once you've done that, I'll link it to this thread, which will serve as the discussion thread for it.
Any questions, just ask!
 

Zwck

Patron
Joined
Oct 27, 2016
Messages
371
Very good guide.

I have been using a docker-machine host in a ubuntu 16.07 vm/iohyve, and it works reasonably well. however i always run into problems when i try to have persistent storage on a network resource. does rancher handle this differently?
 
Top