Rancher VM Suggestions (for 11.1)

[Nasb Klamb]

I'm trying to understand how to best use Docker VMs on 11.1. I've successfully created the rancher UI and another VM host node. But I have a few things (so far) that I can't find info about:

1. Upgrading the Rancher OS in a VM. The installed version is 1.1.0, but 1.1.1 is available. The documented "sudo ros os upgrade" process works and the system reboots. But it is still at 1.1.0. Is there a way to do this?

2. Mounting/sharing storage with containers. From what I can read, using nfs shares is the best way to go. I have shares defined in my freenas. But when I try to mount one of them in my rancher vm, it hangs. E.g. "mount -t nfs alpha:/mnt/vol1/media:/mnt/host_media" hangs.

3. Related to (1), how does booting work with the docker vm? It is created to use grub. And the logs suggest that /boot is where I should find the grub stuff, but the booted system has no /boot directory.

Thanks!

 

[KrisBee]

1. If you knew what kernel was being used in rancher v1.1.1, then you could trying editing the grub.cfg in the hidden directory in your zpool which holds the DockerVM boot config, e.g.:

Code:

root@freenas:/mnt/NasPool/.bhyve_containers/configs/docker2/grub # cat grub.cfg
set timeout=0
set default=RancherOS
menuentry "bhyve-image" --id RancherOS {
set root=(hd0,msdos1)
linux /boot/vmlinuz-4.9.45-rancher rancher.password=rancher printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait rancher.state.autoform
at=[/dev/sda] rancher.resize_device=/dev/sda
initrd /boot/initrd-v1.1.0
}root@freenas:/mnt/NasPool/.bhyve_containers/configs/docker2/grub #

2. If you intend to use the RancherUI ( rancheros + rancher-server etc. ) I would think rancher-nfs from the catalogue is one solution to using NFS mounts. Otherwise, follow this:

http://rancher.com/docs/os/v1.1/en/storage/additional-mounts/

Your example would equate to:

ros config set mounts '[["alpha:/mnt/vol1/media","/mnt/host_media","nfs", ""]]'

Note you need that 4th field.

3. You'd have to unpick the precise boot sequence to see how grub is used and why there appears to be no /boot once rancher is running. Unpacking initrd may be instructive, together with reading dmesg and rancher docs.

 

[Nasb Klamb]

Thanks for the answers. I'm still learning and it's still not working.

I think I have a problem with mounting an nfs share in my docker vm. The Rancher-NFS server was not working, so I tried the ros config command, and it kills the whole host. The docker host stops at a message to the console saying RancherOS is started. But no sshd, no web ui. Nothing.

So after recreating the vm, I ssh'd into the vm and tried a mount:

sudo mount alpha:/mnt/vol1/media /mnt/host_media


And that hangs.

FWIW, I can mount alpha:/mnt/vol1/media from other linux hosts in my network. So I wonder what's with ros?

 

[blacs30]

1. If you knew what kernel was being used in rancher v1.1.1, then you could trying editing the grub.cfg in the hidden directory in your zpool which holds the DockerVM boot config, e.g.:

Code:

root@freenas:/mnt/NasPool/.bhyve_containers/configs/docker2/grub # cat grub.cfg
set timeout=0
set default=RancherOS
menuentry "bhyve-image" --id RancherOS {
set root=(hd0,msdos1)
linux /boot/vmlinuz-4.9.45-rancher rancher.password=rancher printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait rancher.state.autoform
at=[/dev/sda] rancher.resize_device=/dev/sda
initrd /boot/initrd-v1.1.0
}root@freenas:/mnt/NasPool/.bhyve_containers/configs/docker2/grub #

I tried that but it seems the grub.cfg is overwritten on every reboot. :(
Is there another way to set the initrd and vmlinuz kernel version?

 

[Nasb Klamb]

Thanks for the answers. I'm still learning and it's still not working.

I think I have a problem with mounting an nfs share in my docker vm. The Rancher-NFS server was not working, so I tried the ros config command, and it kills the whole host. The docker host stops at a message to the console saying RancherOS is started. But no sshd, no web ui. Nothing.

So after recreating the vm, I ssh'd into the vm and tried a mount:

sudo mount alpha:/mnt/vol1/media /mnt/host_media


And that hangs.

FWIW, I can mount alpha:/mnt/vol1/media from other linux hosts in my network. So I wonder what's with ros?

Update: I saw a syslog message when trying to mount the share:

Code:

docker kernel: [ 3621.712512] svc: failed to register lockdv1 RPC service (errno 512).

When I try the mount command using the nolock option, it doesn't hang. And specifying "nolock" in the ros config set command now works.

 

[KrisBee]

Update: I saw a syslog message when trying to mount the share:

Code:

docker kernel: [ 3621.712512] svc: failed to register lockdv1 RPC service (errno 512).

When I try the mount command using the nolock option, it doesn't hang. And specifying "nolock" in the ros config set command now works.

Would strongly advise against that. If your FreeNAS NFS server is set to use NFSv4 then your mount command in rancher failed as you needed to specify that the client should connect using v4, eg:

mount -t nfs -o vers=4 alpha:/mnt/vol1/media /mnt/host_media

If your FreeNAS server is set to NFSv3, the mount is still going to fail in your try to use a mount command in the default rancher console because the client side rpc progam in not running as it would if you had used a standard linux distro with nfs-utils or nfs-comon package installed.

My advice would be use your FreeNAS NFS server with both "Enable NFSv4" & "NFSv3 ownership model for NFSv4 checked" to avoid id mapping problems and on your NFS share settings in FreeNAS have maproot user set to "root" and maproot group set to "wheel".

Rancher-NFS should work, as does adding mounts to the cloud-config.

 

[duckworth]

Any luck figuring out how to upgrade Rancher OS? I am hitting some bugs in 1.1.0 and would like to upgrade as well. running 'ros os upgrade' and rebooting doesn't stick. Changing the gurb.cfg .bhyve_containers gets overridden as well.

 

[KrisBee]

You can't upgrade rancheros in the new DockerVM function. Every time you start or restart a "DockerVM" its associated grub.cfg file is re-created, so as you've seen any change you might make manually is over written. It seems to be baked into the underlying logic of the FreeNAS middleware. As rancheros doesn't use grub itself, you'd never see any kind of grub prompt which would allow the user to pick which kernel to boot.

For users of rancheros, which is pretty bleeding edge with a spate of recent releases ( 1.1.2 is the latest at https://github.com/rancher/os/releases ), this looks like a severe limitation. Currently you'd have to revert to setting up rancheros by hand with iohyve to get round this, an issue has been raised: https://redmine.ixsystems.com/issues/27484. Was DockerVM function squeezed into 11.1 release too early?

But are the bugs you've hit definitely in rancheros, rather than the version of docker that it's running or, if you're using it, in the version of rancher-server/agent?

 

[duckworth]

Thanks, Looks like the bug was just fixed and targeted for 11.2. I will keep my eye for the release.

 

[KrisBee]

Thanks, Looks like the bug was just fixed and targeted for 11.2. I will keep my eye for the release.

If this is the fix then it looks like just another harded coded grub.cfg which is only good until the next upgrade of rancheros.

https://redmine.ixsystems.com/proje...ff149b558fda102e9bfb716561ef1ef4c6acf97a/diff

 

[duckworth]

True, will this even work with an existing image or will I have to create a new VM?

 

[KrisBee]

True, will this even work with an existing image or will I have to create a new VM?

AFAIK , as implemented the grub.cfg is tied to a particular rancher image as the grub.cfg is used by grub-byhve to load the specified kernel. So I'd say you'd have to create a new VM.

I'd be interested to see what happened if the grub.cfg used wildcards for the vmlinuz and initrd.

Rancheros does not support EFI, so only grub-bhyve can be used in conjunction with bhyve in a two step boot process ( see: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html ). So it's all manual config at the CLI even if you used tools like iohyve or vm-bhyve.

If you installed the rancher iso in virtualbox you'd see rancheros is using syslinux to boot, e.g:

 

[blacs30]

Is there a way to specify where the bhyve config folder .bhyve_containers will be places?
For me it was placed automatically on an encrypted drive which I on reboot first have to unlock that leads me to not being able to autostart the rancher vm.

Couldn't find anything for this in the guide or forum so far.

 

[blacs30]

If this is the fix then it looks like just another harded coded grub.cfg which is only good until the next upgrade of rancheros.

https://redmine.ixsystems.com/proje...ff149b558fda102e9bfb716561ef1ef4c6acf97a/diff

It's actually possible to take over this commit manually by editing this file:
/usr/local/lib/python3.6/site-packages/middlewared/plugins/vm.py

I have rebooted the FreeNAS machine after the change as I don't know how to reload the middleware. Then the new image is used and new grub config written.

Probably not the recommended way but that might enable a manual upgrade of RancherOS (assuming the base image is only required for the first run)

 

[KrisBee]

It's actually possible to take over this commit manually by editing this file:
/usr/local/lib/python3.6/site-packages/middlewared/plugins/vm.py

I have rebooted the freenas machine after the change as I don't know how to reload the middleware. Then the new image is used and new grub config written.

Probably not the recommended way but that might enable a manual upgrade of RancherOS (assuming the base image is only required for the first run)

Contrary to my previous post, this does allow an existing rancheros v.1.1.0 to be upgraded without the need to re-create the VM. But, if you really need to do this you must upgrade any existing v.1.1.0 DockerVMs before making the changes to the vm.py file.

1. In existing DockerVM upgrade os with:

sudo ros os upgrade

2. Apply vm.py fix (will be writing to a system file, possibly on usb stick.)

3. Restart existing DockerVM

Code:

[root@rancher ~]# ros os version  
v1.1.2
[root@rancher ~]# ros os list	
rancher/os:v1.1.2 local latest running
rancher/os:v1.1.1 remote available 
rancher/os:v1.1.0 remote available 
rancher/os:v1.0.4 remote available 
rancher/os:v1.0.3 remote available 
rancher/os:v1.0.2 remote available 
rancher/os:v1.0.1 remote available 
rancher/os:v1.0.0 remote available 
rancher/os:v0.9.2 remote available 
rancher/os:v0.8.1 remote available 
rancher/os:v0.7.1 remote available 
rancher/os:v0.6.1 remote available 
rancher/os:v0.5.0 remote available 
rancher/os:v0.4.5 remote available

But it's only good until the next rancheros release.

 

[diskdiddler]

Contrary to my previous post, this does allow an existing rancheros v.1.1.0 to be upgraded without the need to re-create the VM. But, if you really need to do this you must upgrade any existing v.1.1.0 DockerVMs before making the changes to the vm.py file.

1. In existing DockerVM upgrade os with:

sudo ros os upgrade

2. Apply vm.py fix (will be writing to a system file, possibly on usb stick.)

3. Restart existing DockerVM

Code:

[root@rancher ~]# ros os version 
v1.1.2
[root@rancher ~]# ros os list	
rancher/os:v1.1.2 local latest running
rancher/os:v1.1.1 remote available
rancher/os:v1.1.0 remote available
rancher/os:v1.0.4 remote available
rancher/os:v1.0.3 remote available
rancher/os:v1.0.2 remote available
rancher/os:v1.0.1 remote available
rancher/os:v1.0.0 remote available
rancher/os:v0.9.2 remote available
rancher/os:v0.8.1 remote available
rancher/os:v0.7.1 remote available
rancher/os:v0.6.1 remote available
rancher/os:v0.5.0 remote available
rancher/os:v0.4.5 remote available

But it's only good until the next rancheros release.

Does this still work?
https://redmine.ixsystems.com/issues/51102#change-389643
Is this information useful for this?