Like many I have been rather attracted by the Docker possibilities of Corral and the cold shower has been hard. Moreover this is happening at the moment I have to setup home & professional servers and I wanted to use Docker for applications. So I insisted a bit and ended up with a setup based on Rancher & RancherOS which I now found very satisfying. Because it sounds like there is a lot of people around struggling with setting this thing up, I thought I should write something here to help. This is an example on how to setup a Plex server as a container in Rancher running under a RancherOS VM. I used this procedure successfully with two different machines on two very different networks.
1 - Setting up a RancherOS VM
RancherOS is a ultra-lightweight Linux OS specialised for Docker. RancherOS does not support UEFI boot so the VM can't be setup through the FreeNAS 11 GUI but it can be done using iohyve. Open a root shell on your FreeNAS server and start by setting up iohyve:
where
name: iohyve_enable -- value: YES
name: iohyve_flags -- value: kmod=1 net=<nic>
This will insure that iohyve starts correctly when you'll reboot your machine. Now download RancherOS image:
and copy it in iohyve internal folder:
Here I used RancherOS 1.0.1-LTS. Now we will create two virtual disks for the VM. Open the FreeNAS GUI and in storage create two zvols wherever you want, in my case that will be
tank/vdisks/ranch/docker: 20 G
tank/vdisks/ranch/plex-transcode: 20 G
The first disk will contain the /var/lib/docker of the system if something goes wrong and the second is for Plex transcoding. A bit of explanation here: the rest of the data from the FreeNAS box will be shared with containers through NFS which is more convenient because you can browse the files. But Plex will HATE doing some write-intensive transcoding operation through NFS (even locally) so for that a direct disk is better. Let's create the VM
4G is enough to install the system. Set the VM parameters:
You might want to adapt the CPU & memory allocated depending on your hardware. Now go to the VM directory
Use whatever you want to create the three following files in this directory:
device.map:
grub.cd.cfg:
grub.hd.cfg (sorry for the long line, I am not sure these files support \ line breaks):
The file device.map set the system disk and virtual CD, grub.cd.cfg is a Grub configuration to boot RancherOS from the CD and grub.hd.cfg to boot RancherOS from the hard drive with persistence.
Just a comment here, why three virtual disks (the RancherOS drive and the two zvols for docker & Plex)? Well you don't have to, you can just create the VM with a big ~50G drive. Bust experience told me that with this kind of story things can go real wrong real quick if you play too much. Then it is nice to have an easy way to recover. With this setup if you mess up the system you can still reinstall the system drive and then remount the docker drive and every container you had active before will be there. I just like to have the system drive isolated in case of. But again you don't have to, another strategy is to carefully snapshot clean configurations and roll back if something wrong happens.
2 - Installing RancherOS
Let's use the CD boot:
and start the VM with
Installing the system is a bit different if you have a DHCP or static network configuration so I will split the two cases.
2.1 - DHCP network configuration
If you have a DHCP then your VM should have picked up IP, DNS & gateway and you should be able to reach the net. Give Google a ping to check. If you want the MAC of the VM to be fixed for a static lease, do the following in FreeNAS:
where <MAC> is whatever MAC address you want. Now create a cloud-config.yml containing:
Where the XXXXXX... should be replaced by the SSH public key of the machine you want to use to SSH into RancherOS. Here the DNS are the Google ones as an example, don't let Google know your traffic and change it if possible.
2.2 - Static IP network configuration
If you have no DHCP on the network then surely you can't reach the net, and you need to for the installation. Start by writing the following cloud-config.yml:
Where the XXXXXX... should be replaced by the SSH public key of the machine you want to use to SSH into RancherOS. The IPs are just an example, replace them with what you need. Similarly, the DNS are the Google ones as an example, don't let Google know your traffic and change it if possible. Now put the config in the system:
and restart the network service container:
Now you should be able to access Internet. Double check with
2.3 - Installation
Now install the system with
When this is finished the installer will ask you if you want to reboot, say no. Go back to a FreeNAS shell and stop the VM with
and start the VM again
If everything went well no password was asked to you are in.
3 - Setting up Rancher
Spawn a Rancher container in RancherOS with
Wait a bit, you can do
We will now setup the NFS access. In FreeNAS create a dataset that will hosts the container volumes, for me it was tank/cloud-data, within this dataset create two other ones called plex-db and medias. Make some user called plex owner of these datasets. Start the NFS service with the 'Enable NFSv4' option enabled. Add two NFS shares for the two datasets created, map all the permissions to plex and only authorise the RancherOS IP to mount. Back to Rancher go to Catalog > Library and install Rancher NFS. Fill your FreeNAS IP and /mnt/tank/cloud-data as a mount point.
4 - Setting up Plex
In Rancher go to Stack > Add Stack, name the stack the way you like and provide the two following files:
docker-compose.yml
rancher-compose.yml
And click Create. If everything goes well after a bit you should be able to reach Plex web interface at your RancherOS IP on port 32400. Rancher allows you through its web interface to directly see the logs of the containers which might help you know what's going on. With this setup the folder /mnt/tank/cloud-data/medias is supposed to be used to put movies, etc... for Plex. It will be seen inside of Plex as /data.
This is all! In principle if you reboot FreeNAS the VM and Rancher & Plex should restart automatically. I did not write the YML files for Plex, but I exported them from containers I created with Rancher awesome interface, I let you experiment and read the doc to figure that out.
EDIT 1: Added static IP configuration and commented on the virtual disks.
EDIT 2: Typos
1 - Setting up a RancherOS VM
RancherOS is a ultra-lightweight Linux OS specialised for Docker. RancherOS does not support UEFI boot so the VM can't be setup through the FreeNAS 11 GUI but it can be done using iohyve. Open a root shell on your FreeNAS server and start by setting up iohyve:
iohyve setup pool=vm kmod=1 net=igb0where
vm is the ZFS pool you want to use for iohyve and igb0 is the network interface to which the VMs will be connected. Now through FreeNAS GUI, add two tunables to rc.conf :name: iohyve_enable -- value: YES
name: iohyve_flags -- value: kmod=1 net=<nic>
This will insure that iohyve starts correctly when you'll reboot your machine. Now download RancherOS image:
wget https://releases.rancher.com/os/latest/rancheros.isoand copy it in iohyve internal folder:
iohyve cpiso rancheros.isoHere I used RancherOS 1.0.1-LTS. Now we will create two virtual disks for the VM. Open the FreeNAS GUI and in storage create two zvols wherever you want, in my case that will be
tank/vdisks/ranch/docker: 20 G
tank/vdisks/ranch/plex-transcode: 20 G
The first disk will contain the /var/lib/docker of the system if something goes wrong and the second is for Plex transcoding. A bit of explanation here: the rest of the data from the FreeNAS box will be shared with containers through NFS which is more convenient because you can browse the files. But Plex will HATE doing some write-intensive transcoding operation through NFS (even locally) so for that a direct disk is better. Let's create the VM
iohyve create ranch 4G4G is enough to install the system. Set the VM parameters:
iohyve set ranch cpu=8 ram=10G os=custom boot=1 loader=grub-bhyve \
pcidev:1=ahci-hd,/dev/zvol/tank/vdisks/ranch/docker \
pcidev:2=ahci-hd,/dev/zvol/tank/vdisks/ranch/plex-transcodeYou might want to adapt the CPU & memory allocated depending on your hardware. Now go to the VM directory
cd /iohyve/ranchUse whatever you want to create the three following files in this directory:
device.map:
Code:
(hd0) /dev/zvol/vm/iohyve/ranch/disk0 (cd0) /iohyve/ISO/rancheros.iso/rancheros.iso
grub.cd.cfg:
Code:
linux (cd0,msdos1)/boot/vmlinuz-4.9.24-rancher rancher.password=rancher initrd (cd0,msdos1)/boot/initrd-v1.0.1 boot
grub.hd.cfg (sorry for the long line, I am not sure these files support \ line breaks):
Code:
linux (hd0,msdos1)/boot/vmlinuz-4.9.24-rancher printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait console=tty0 initrd (hd0,msdos1)/boot/initrd-v1.0.1 boot
The file device.map set the system disk and virtual CD, grub.cd.cfg is a Grub configuration to boot RancherOS from the CD and grub.hd.cfg to boot RancherOS from the hard drive with persistence.
Just a comment here, why three virtual disks (the RancherOS drive and the two zvols for docker & Plex)? Well you don't have to, you can just create the VM with a big ~50G drive. Bust experience told me that with this kind of story things can go real wrong real quick if you play too much. Then it is nice to have an easy way to recover. With this setup if you mess up the system you can still reinstall the system drive and then remount the docker drive and every container you had active before will be there. I just like to have the system drive isolated in case of. But again you don't have to, another strategy is to carefully snapshot clean configurations and roll back if something wrong happens.
2 - Installing RancherOS
Let's use the CD boot:
cp grub.cd.cfg grub.cfgand start the VM with
iohyve start ranch. Go into the VM serial console with iohyve console ranch and you should see the system booting. When the login screen appear, login with rancher both for the username and the password. Become root with sudo -i. Let's start by formatting the virtual drives. Do fdisk /dev/sdb, type n and Return to accept all the default options then w to create the partition an quit. Format the disk with mkfs.ext4 /dev/sdb1. Repeat the procedure for /dev/sdc.Installing the system is a bit different if you have a DHCP or static network configuration so I will split the two cases.
2.1 - DHCP network configuration
If you have a DHCP then your VM should have picked up IP, DNS & gateway and you should be able to reach the net. Give Google a ping to check. If you want the MAC of the VM to be fixed for a static lease, do the following in FreeNAS:
iohyve set ranch mac_tap0=<MAC>where <MAC> is whatever MAC address you want. Now create a cloud-config.yml containing:
Code:
#cloud-config hostname: ranch.local ssh_authorized_keys: - XXXXXXXXXXXXXXXX mounts: - ["/dev/sdb1", "/var/lib/docker", "ext4", ""] - ["/dev/sdc1", "/mnt/plex-transcode", "ext4", ""] rancher: network: interfaces: eth0: dhcp: true dns: nameservers: - 8.8.8.8 - 8.8.4.4
Where the XXXXXX... should be replaced by the SSH public key of the machine you want to use to SSH into RancherOS. Here the DNS are the Google ones as an example, don't let Google know your traffic and change it if possible.
2.2 - Static IP network configuration
If you have no DHCP on the network then surely you can't reach the net, and you need to for the installation. Start by writing the following cloud-config.yml:
Code:
#cloud-config hostname: cliath-ranch.ph.ed.ac.uk ssh_authorized_keys: - XXXXXXXXXXXXXXXX mounts: - ["/dev/sdb1", "/var/lib/docker", "ext4", ""] - ["/dev/sdc1", "/mnt/plex-transcode", "ext4", ""] rancher: network: interfaces: eth0: addresses: - 192.168.0.5/24 - 192.168.0.6/24 gateway: 192.168.0.1 dhcp: false dns: nameservers: - 8.8.8.8 - 8.8.4.4
Where the XXXXXX... should be replaced by the SSH public key of the machine you want to use to SSH into RancherOS. The IPs are just an example, replace them with what you need. Similarly, the DNS are the Google ones as an example, don't let Google know your traffic and change it if possible. Now put the config in the system:
ros config merge -i cloud-config.ymland restart the network service container:
system-docker restart networkNow you should be able to access Internet. Double check with
ifconfig and ping some website.2.3 - Installation
Now install the system with
ros install -d /dev/sda -c cloud-config.ymlWhen this is finished the installer will ask you if you want to reboot, say no. Go back to a FreeNAS shell and stop the VM with
iohyve stop ranch. Now we will boot from our fresh install, change the Grub config with cp grub.hd.cfg grub.cfgand start the VM again
iohyve start ranch. Wait a bit and figure out which IP the VM is connected on and SSH into RancherOS ssh rancher@<IP>If everything went well no password was asked to you are in.
3 - Setting up Rancher
Spawn a Rancher container in RancherOS with
sudo docker run -d --restart=unless-stopped --name='rancher-server' -p 8080:8080 rancher/server:stableWait a bit, you can do
docker logs -f rancher-server to see Rancher starting. At some point you should be able to see the web interface at http://<RancherOS IP>:8080. Go to Infrastructure > Hosts and click Add Host. There will be a command on the page to copy & paste into the RancherOS shell, do it. After it started, you should see your machine in the Rancher hosts. Now secure the login to Rancher by going to Admin > Access Control and set things up the way you like.We will now setup the NFS access. In FreeNAS create a dataset that will hosts the container volumes, for me it was tank/cloud-data, within this dataset create two other ones called plex-db and medias. Make some user called plex owner of these datasets. Start the NFS service with the 'Enable NFSv4' option enabled. Add two NFS shares for the two datasets created, map all the permissions to plex and only authorise the RancherOS IP to mount. Back to Rancher go to Catalog > Library and install Rancher NFS. Fill your FreeNAS IP and /mnt/tank/cloud-data as a mount point.
4 - Setting up Plex
In Rancher go to Stack > Add Stack, name the stack the way you like and provide the two following files:
docker-compose.yml
Code:
version: '2' volumes: medias: external: true driver: rancher-nfs plex-db: external: true driver: rancher-nfs services: plex-server: image: plexinc/pms-docker environment: CHANGE_CONFIG_DIR_OWNERSHIP: 'false' stdin_open: true tty: true volumes_from: - plex-storage ports: - 32400:32400/tcp - 32413:32413/udp - 32414:32414/udp - 32469:32469/tcp - 3005:3005/tcp - 32410:32410/udp - 32412:32412/udp - 8324:8324/tcp - 1900:1900/udp labels: io.rancher.container.pull_image: always io.rancher.sidekicks: plex-storage plex-storage: image: busybox stdin_open: true volumes: - medias:/data - plex-db:/config - /mnt/plex-transcode/transcode:/transcode tty: true labels: io.rancher.container.pull_image: always
rancher-compose.yml
Code:
version: '2' services: plex-server: scale: 1 start_on_create: true plex-storage: scale: 1 start_on_create: true
And click Create. If everything goes well after a bit you should be able to reach Plex web interface at your RancherOS IP on port 32400. Rancher allows you through its web interface to directly see the logs of the containers which might help you know what's going on. With this setup the folder /mnt/tank/cloud-data/medias is supposed to be used to put movies, etc... for Plex. It will be seen inside of Plex as /data.
This is all! In principle if you reboot FreeNAS the VM and Rancher & Plex should restart automatically. I did not write the YML files for Plex, but I exported them from containers I created with Rancher awesome interface, I let you experiment and read the doc to figure that out.
EDIT 1: Added static IP configuration and commented on the virtual disks.
EDIT 2: Typos
