Recover Rancher VM

[vinistois]

I was on 11.2B2 and had a Rancher VM running nicely.

I updated to B3 and some things were broken so I had to reset the config.

I imported the storage pool and managed to recover my other VM by re-creating it and attaching the disk. But what is the process for doing this with a Rancher VM?

I still have the "rancherdata" dataset. How do I rebuild?

Thanks!

 

[KrisBee]

The configuration of "Docker Host" VMs is stored on your pool in a hidden directory at /mnt/<pool name>/.bhyve_containers

Recovering a "Docker Host" VM may be possible by copying an existing config to a newly created "Docker Host". It's the device.map and grub.cfg that need to be copied.

You cannot do the recovery via the webUI alone, old or new. You cannot pick a pre-exiting RAW file when creating a "Docker Host" VM, or simply edit the RAW device after using the wizard. The "Docker Host" VM configuration and RAW file are not created on the pool until the VM is started for the first time.

So in order to copy the underlying config of your inaccessible docker VM you have to go through all steps of creating and starting once a new "Docker Host".

First check your RancherVM config is still on you pool. e.g. with:
ll -R /mnt/<pool name>/.bhyve_containers

Here's an exmaple:

Code:

root@freenas:/ # ll -R /mnt/NasPool/.bhyve_containers
total 1
drwxr-xr-x  3 root  wheel  3 Sep 15 09:35 configs/
drwxr-xr-x  2 root  wheel  4 Sep 12 12:20 iso_files/

/mnt/NasPool/.bhyve_containers/configs:
total 1
drwxr-xr-x  3 root  wheel  4 Sep 15 09:35 27_dockerA/

/mnt/NasPool/.bhyve_containers/configs/27_dockerA:
total 1
-rw-r--r--  1 root  wheel  38 Sep 15 09:35 device.map
drwxr-xr-x  2 root  wheel   3 Sep 15 09:35 grub/

/mnt/NasPool/.bhyve_containers/configs/27_dockerA/grub:
total 5
-rw-r--r--  1 root  wheel  294 Sep 15 09:35 grub.cfg

/mnt/NasPool/.bhyve_containers/iso_files:
total 117713
-rw-r--r--  1 root  wheel  47958299 Jul 28 15:47 rancheros-bhyve-v1.1.3.img.gz
-rw-r--r--  1 root  wheel  73924582 Sep 12 12:20 rancheros-bhyve-v1.4.0.img.gz

If it exists, then check the contents of device.map and that it references a RAW file that is still on your pool. In my example it is:

Code:

root@freenas:/mnt/NasPool/.bhyve_containers/configs/27_dockerA # cat device.map
(hd0) mnt/NasPool2/VM/dockerA_dockerA

IIRC, BETA2 was still using the rancheros-bhyve-v1.1.3.img.gz. Is that still on your pool?

 

[vinistois]

wow Krisbee, possibly the most helpful reply I've ever received on the internet.

The old Rancher files are still there as you describe them. I installed a new Rancher instance and booted it up. The device.map points to a .img that exists, and I have rancheros-bhyve-v1.4.0.img.gz on my pool:

Code:

root@freenas[~]# ll -R /mnt/QuietPool/.bhyve_containers								
total 10
drwxr-xr-x  4 root  wheel  uarch 4 Sep 11 00:29 ./
drwxrwxr-x+ 6 root  wheel  uarch 9 Sep 16 22:24 ../
drwxr-xr-x  4 root  wheel  uarch 4 Sep 16 22:24 configs/
drwxr-xr-x  2 root  wheel  uarch 3 Sep 11 00:30 iso_files/

/mnt/QuietPool/.bhyve_containers/configs:
total 2
drwxr-xr-x  4 root  wheel  uarch 4 Sep 16 22:24 ./
drwxr-xr-x  4 root  wheel  uarch 4 Sep 11 00:29 ../
drwxr-xr-x  3 root  wheel  uarch 4 Sep 16 22:24 5_RacherSyslog/
drwxr-xr-x  3 root  wheel  uarch 4 Sep 11 00:36 6_Rancher/

/mnt/QuietPool/.bhyve_containers/configs/5_RacherSyslog:
total 2
drwxr-xr-x  3 root  wheel  uarch  4 Sep 16 22:24 ./
drwxr-xr-x  4 root  wheel  uarch  4 Sep 16 22:24 ../
-rw-r--r--  1 root  wheel  uarch 51 Sep 16 22:24 device.map
drwxr-xr-x  2 root  wheel  uarch  3 Sep 16 22:24 grub/

/mnt/QuietPool/.bhyve_containers/configs/5_RacherSyslog/grub:
total 6
drwxr-xr-x  2 root  wheel  uarch   3 Sep 16 22:24 ./
drwxr-xr-x  3 root  wheel  uarch   4 Sep 16 22:24 ../
-rw-r--r--  1 root  wheel  uarch 294 Sep 16 22:24 grub.cfg

/mnt/QuietPool/.bhyve_containers/configs/6_Rancher:
total 2
drwxr-xr-x  3 root  wheel  uarch  4 Sep 11 00:36 ./
drwxr-xr-x  4 root  wheel  uarch  4 Sep 16 22:24 ../
-rw-r--r--  1 root  wheel  uarch 52 Sep 14 11:37 device.map
drwxr-xr-x  2 root  wheel  uarch  3 Sep 11 00:36 grub/

/mnt/QuietPool/.bhyve_containers/configs/6_Rancher/grub:
total 6
drwxr-xr-x  2 root  wheel  uarch   3 Sep 11 00:36 ./
drwxr-xr-x  3 root  wheel  uarch   4 Sep 11 00:36 ../
-rw-r--r--  1 root  wheel  uarch 294 Sep 11 00:36 grub.cfg

/mnt/QuietPool/.bhyve_containers/iso_files:
total 71566
drwxr-xr-x  2 root  wheel  uarch		3 Sep 11 00:30 ./
drwxr-xr-x  4 root  wheel  uarch		4 Sep 11 00:29 ../
-rw-r--r--  1 root  wheel  uarch 73924582 Sep 11 00:30 rancheros-bhyve-v1.4.0.img.gz

I copied device.map and grub.cfg from /mnt/QuietPool/.bhyve_containers/configs/6_Rancher (my old instance) to /mnt/QuietPool/.bhyve_containers/configs/5_RacherSyslog (the new instance), and started it back up, but none of my stacks are there, it's the same as the fresh boot. Is there a way to get the stacks/containers back?

 

[KrisBee]

As the answer was no to the last question in my first post, perhaps you should have paused at that point. I'm pretty sure your old Rancher VM was built from rancheros-bhyve-v1.1.3.img.gz. The contents of your file /mnt/QuietPool/.bhyve_containers/configs/6_Rancher/grub/grub.cfg would show that.

I don't think you can get your stacks/containers back as it seems your original docker img file may have been overwritten. And if the rancher/server docker is missing, then it almost certainly has been overwritten.

I should have added a warning to backup or snapshot your original docker img file. I did only say you may be able to recover your old Rancher VM as this is a hack.

On reflection, it may have been safer to restart your old RancherVM from the command line, but then it may never have been visible again in the web UI.

Creating/editing "Docker Host" VMs in Beta2/3 is pretty buggy. How extensive is your use of docker apps?
Beta3 uses rancheros 1.4.0 as it base, which is meant to be used in conjunction with rancher2. Rancher2 has moved to kubernetes. Rancher2 is designed for large scale development and deployment of dockers over multiple hosts. That's not exactly the typical use case for a FreeNAS home user.

I'd encourage FreeNAS users to use jails before docker. If you must use docker then consider a combination of a debian/ubuntu VM with docker compose and/or portainer for small scale docker app management.

 

[diskdiddler]

Kris : is this still the case?

I think I may have been stung by this tonight. Tried pointing to the existing raw file.

Wonder if it's possible to recover the VM in the UI with some work, so users don't need to fiddle. I may log a job to the team

 

[diskdiddler]

I've logged a job for this now, I may need to do this in the near future. We need a comprehensive way to backup docker and containers.
https://redmine.ixsystems.com/issues/54987