- Joined
- Apr 24, 2020
- Messages
- 5,399
There was very little documentation on how to create a VM that boots from GRUB in the GUI. I had to go digging in /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py to figure out how to do this successfully. I found there are 2 required parameters that aren't accessible from the GUI, which have to be set via the REST 2.0 API to achieve a successful installation.
1. First, starting at line 142 of /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py:
This means that grub VMs can only boot from a RAW device, and the device needs to have the attribute "boot"="true" set. If the RAW device doesn't have the "boot"="true" attribute set, you'll get the error "There is no boot disk for vm: <name of grub VM>" when you try to boot the VM in the GUI, as described starting at line 241 of /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py:
Unfortunately, the only way to set this attribute is via the REST 2.0 API:
The device ID for the RAW file is visible in the GUI, and the VM ID can be found in the number of the COM port /dev/nmdm<VM ID>B. Note the path to the RAW file is a mandatory field for the API call. Also, password login to root has to be enabled for the API call to succeed, as you'll authenticate the API call with root's password.
What this also means is that it's not possible to install from a CD-ROM device to a RAW file. You'll need to export a temporary zvol via iSCSI to an external hypervisor, like VirtualBox or Hyper-V, and install from the ISO to the zvol from outside FreeNAS. Once the install is complete, and the temporary external VM is verified to boot from the zvol, then you can use dd to convert the temporary zvol to the RAW file:
2. Second, you'll need to make grub-bhyve happy. As described in /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py, starting at line 253:
The VM itself needs to have the "grubconfig" attribute set. This is either a path to the grub.cfg, starting from /mnt, or the contents of a working grub.cfg. This is where I found the documentation for grub-bhyve contradictory. Grub-bhyve does NOT use the grub.cfg inside the RAW file, even though it can navigate within the RAW file and load the kernel and initrd; rather, grub-bhyve relies on a grub.cfg on the host.
I recommend using a path to grub.cfg for the "grubconfig" attribute, as it's easier to modify the grub.cfg in vi instead of having to edit the attribute via API:
The simplest thing here is to just cut & paste the grub.cfg from inside the RAW file to a new grub.cfg on the host. Note, use single quotes instead of parentheses for the grub set root command.
If you've typoed anything inside the grub.cfg, you'll get the error "grub-bhyve timed out, please check your grub config." when you try to boot the VM.
1. First, starting at line 142 of /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py:
Code:
# Get grub devices to be used in grub-bhyve if device['dtype'] == 'RAW' and device['attributes'].get('boot'): grub_devices.append(device['attributes']['path'])
This means that grub VMs can only boot from a RAW device, and the device needs to have the attribute "boot"="true" set. If the RAW device doesn't have the "boot"="true" attribute set, you'll get the error "There is no boot disk for vm: <name of grub VM>" when you try to boot the VM in the GUI, as described starting at line 241 of /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py:
Code:
# grub-bhyve support device_map_file = tempfile.NamedTemporaryFile() grub_dir = None if self.vm['bootloader'] == 'GRUB': if not grub_devices: raise CallError(f'There is no boot disk for vm: {self.vm["name"]}') for i, device in enumerate(grub_devices): device_map_file.write(f'(hd{i}) {device}\n'.encode()) device_map_file.flush()
Unfortunately, the only way to set this attribute is via the REST 2.0 API:
Code:
curl --basic -u root -k -X PUT "http://<FreeNAS GUI IP>/api/v2.0/vm/device/id/<device id>" -H "accept: */*" -H "Content-Type: application/json" -d '{"dtype":"RAW","vm":<vm id>,"attributes":{"boot":true,"path":"<path to image>"}}'
The device ID for the RAW file is visible in the GUI, and the VM ID can be found in the number of the COM port /dev/nmdm<VM ID>B. Note the path to the RAW file is a mandatory field for the API call. Also, password login to root has to be enabled for the API call to succeed, as you'll authenticate the API call with root's password.
What this also means is that it's not possible to install from a CD-ROM device to a RAW file. You'll need to export a temporary zvol via iSCSI to an external hypervisor, like VirtualBox or Hyper-V, and install from the ISO to the zvol from outside FreeNAS. Once the install is complete, and the temporary external VM is verified to boot from the zvol, then you can use dd to convert the temporary zvol to the RAW file:
Code:
dd if=<path to zvol> of=<path to RAW file>
2. Second, you'll need to make grub-bhyve happy. As described in /usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py, starting at line 253:
Code:
if ( self.vm['grubconfig'] and self.vm['grubconfig'].startswith('/mnt/') and os.path.exists(self.vm['grubconfig']) ): grub_dir = os.path.dirname(self.vm['grubconfig']) else: grub_dir = f'/tmp/grub/{self.vm["id"]}_{self.vm["name"]}' os.makedirs(grub_dir, exist_ok=True) grub_file = os.path.join(grub_dir, 'grub.cfg') with open(grub_file, 'w') as f: f.write(self.vm['grubconfig'])
The VM itself needs to have the "grubconfig" attribute set. This is either a path to the grub.cfg, starting from /mnt, or the contents of a working grub.cfg. This is where I found the documentation for grub-bhyve contradictory. Grub-bhyve does NOT use the grub.cfg inside the RAW file, even though it can navigate within the RAW file and load the kernel and initrd; rather, grub-bhyve relies on a grub.cfg on the host.
I recommend using a path to grub.cfg for the "grubconfig" attribute, as it's easier to modify the grub.cfg in vi instead of having to edit the attribute via API:
Code:
curl --basic -u root -k -X PUT "http://<FreeNAS GUI IP>/api/v2.0/vm/id/<vm id>" -H "accept: */*" -H "Content-Type: application/json" -d '{"grubconfig":"<path to grub.cfg>/grub.cfg"}'
The simplest thing here is to just cut & paste the grub.cfg from inside the RAW file to a new grub.cfg on the host. Note, use single quotes instead of parentheses for the grub set root command.
If you've typoed anything inside the grub.cfg, you'll get the error "grub-bhyve timed out, please check your grub config." when you try to boot the VM.
Last edited: