RancherOS 1.5.5 + Docker Swarm 19.0.3.5 + NFS4.... 10 step program

[vinistois]

I've spent too many hours getting this working to not share.

This is an example of:

• Freenas 11.2U7
• 3 nodes RancherOS deployed the 'docker host' way.
• RancherOS upgraded to 1.5.5 by following this very helpful post
• Docker version upgraded to 19.0.3.5
• Docker Swarm deploy 3 node
• No Rancher server. I use Portainer instead.
• NFS shares mounted from Freenas host directly as named volumes into deployed services, not on the host.
• NFS shares mounted via docker-compose, not pre-declared as docker volumes
• NFS shares auto-created on the other nodes when scaled to them by the stack with no 'prep'

steps

I'm not going to type all the commands verbatim here, you need to do your due diligence and verify the commands you're typing. Only run commands as root if the systems ask. Don't just sudo su and go forth, because, chances are you will do things as root that leave them inaccessible to other users. Where things are between $dollar$ signs, replace that with your own input, obviously without the $.

1. Create 3 nodes using 'docker host', suggest 8GB for the manager node and 4GB for the worker nodes
   1. Before you start them go into vm devices, and edit the 'raw' device, change the password.
   2. If you're going to host public-facing services, attach a nic sourced from your dmz vlan now to each vm.
   3. start the first 'manager' vm
2. All these vms will have the same default hostname 'rancher'. Rename them, I chose 'manager' 'worker1', and 'worker2
   1. ssh into the manager host and login as rancher:$password$ (from step1)
   2. sudo ros config set hostname $worker$
   3. sudo ros config get hostname (should return your hostname)
   4. sudo reboot
   5. ssh into the 2nd vm and repeat steps 2 - 4
3. when they boot back up, verify hostnames, and set static ip and dns records
   1. hostname (should return your hostname)
   2. I use PFsense and hit the button in the dhcp users list to quickly create the static ip entry and put each nic in the vlan I want (management and dmz)
   3. I then create a hostname override in my dns resolver to point the fqdn to the 3 host vms.
   4. I take this opportunity to also create hostname overrides for the various services I will host and add them as additional hostnames pointing to the manager node
4. Now you should be able to ssh in by fqdn
   1. ssh rancher@$hostname$.$domain$
   2. make sure you can ssh into each vm by hostname this way
5. Now lets upgrade rancher OS to latest
   1. sudo ros os list (this will return a list of available upgrade versions)
   2. sudo ros os upgrade (this will automatically select the newest version. At time of writing, 1.5.5.
   3. DO NOT say 'y' to reboot. say no.
   4. drop back to shell
6. Now we have to fool freenas into letting us upgrade (ffs)
   1. ssh into your freenas system
   2. nano /usr/local/lib/python3.6/site-packages/middlewared/plugins/vm.py
   3. using nano, you can use ctl-W to search for a string. search for "vmlinuz"
   4. you need to change two version numbers. the vmlinuz number, and the initrd number. If you look at the rancherOS releases page, you'll see how they are related. Edit the numbers carefully, mine look like this:
      1. grub_additional_args = {
         'RancherOS': ['linux /boot/vmlinuz-4.14.138-rancher rancher.password={0} printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE
         rancher.state.wait rancher.resize_device=/dev/sda'.format(quote(password)),
         'initrd /boot/initrd-v1.5.5']
   5. nano /mnt/$<POOL>$/.bhyve_containers/configs/$<CONTAINER>$/grub/grub.cfg (you'll have to edit this for your pool and vm names)
      1. same thing in this file, change the vmlinuz and initrd version numbers to 4.14.138 and v1.5.5, respectively.
   6. Make the above changes to the grub.cfg file for each of the 3 vms
   7. Reboot all 3 vms
   8. OK! now do a sudo ros os list again on each host, and verify they are all updated to the latest RancherOS 1.5.5
7. Docker Swarm Init!
   1. Do not deploy rancher/rancher (unless you really want to) I could never get a swarm deployment working properly that way.
   2. Just init and join the other nodes and check the status of the swarm
   3. docker info has a section with Swarm status
8. add NFS capabilities to rancheros (I'm not sure this step is required, I have not tried without it)
   1. sudo ros service list (should see a disabled volume.nfs service in this list.)
   2. as an aside, the services volume.nfs, volume.cifs, volume.efs, used to be part of "docker-volume-netshare". Now its broken into 3 services.
   3. sudo ros enable volume.nfs
   4. sudo ros start volume.nfs
   5. sudo reboot
9. In Freenas, create a dataset and an NFS share.
   1. do not do anything special to the dataset / permissions / etc. Just create it w/ defaults and move on
   2. services --> nfs --> enable
   3. services --> nfs --> edit -->turn on NFSv4 and NFSv3 ownership
   4. sharing --> nfs --> add
   5. turn on All dirs
   6. maproot User > root
   7. maproot Group > wheel
10. Ok now you are ready for your first service. I use 'docker stack deploy' or I paste my stack deploy compose files into Portainer. I don't create volumes, containers, networks, or touch anything outside of the 'docker stack deploy'.
    1. do not mount any shares on the host
    2. do not use ros config set mount ____
    3. do not specify any different storage drivers

gotchas:

- the named volume should not pre-exist, otherwise it will be attached and the share will not mount. if it exists, delete it

- if you make a mistake in the compose yml, and that mistake has something to do with the volumes, delete all the existing services before deploying again. Portainer makes this easy, just go into stacks view, delete the whole stack, then go into the editor and make your changes and deploy again (don't forget to go delete the volume first!)

- the above two gotchas will get the hell out of you and waste your time if you don't heed them

- I just use a single NFS share, and then the services mount subfolders of that share. That works fine for me for ~20 services. You could also make separate NFS shares for each service if you want.

- if you're creating special users and groups in freenas and then trying to chmod folders and figuring out what uid/gid has permissions, you're gonna be in for a hard time. I tried those methods. I tried them over and over again. not only did they not work, they are too much work. This method, once you have it setup, you can forget about how you got there and just work in portainer.

- I'm not a security specialist. I would recommend, once your shares are mounted and the containers are running, ssh to freenas and do a cat /var/log/messages. You will see something like: mount request succeeded from 192.168.70.7 for ... from each node mounting a share. Add those ip addresses into the 'Authorized Hosts and IP addresses' field in the NFS share settings.

- Just because you can run replicas of containers and they can all share common storage, doesn't mean you should. Not all services are built this way, so do your research. Further, even services that support clustering of multiple instances, sometimes don't support it this way, so do your research.

finally, here is a compose file for a stack-deploy operation

Code:

version: "3.7"

services:

  telegraf:
    image: "telegraf:latest"
    networks:
      - metrics
    volumes:
      - telegraf-nfs:/etc/telegraf    #you can also use the long syntax here but it isn't required
    deploy:
      replicas: 1  #after deployment, you can turn this up to 2 or 3.  The volume will mount to each host automagically.

volumes:
  telegraf-nfs:   #this must match the volume claim in the service above
    driver_opts:
      type: nfs     #you see many people put nfs4 here.  Didn't work for me, also, the server is nfs4 only anyways.
      o: "addr=freenas.malonehome.net,nolock,soft,rw"    # if you put 'hard' here instead of 'soft', when things go wrong with NFS it will lock up your vms.
      device: ":/mnt/QuietPool/rancheros/telegraf"     #here put in the mount path.  you should be able to specify any subfolder underneath the share and mount it.

networks:
  metrics: