TrueNAS SCALE - Virtualization Plugin - File/qcow2 support for QEMU/KVM instead of using zvol

kc.yinger

Cadet
Joined
Dec 30, 2020
Messages
1
Everything is working great thus far. The only exception, I was trying to figure out how to use a "qcow2" disk image as the boot source for a VM within the angular ui. The native qemu image, "qcow2", is certainly more flexible then a block device when it comes to snapshots, migrations, sizing, and pausing VM state. These file based images work great being stored within a zfs dataset with snapshots enabled.

I tried to create a "raw" device and proceeded to modify the markup (/etc/libvirt/qemu/<vm name>.xml) as shown below, but an exception is raised due to the type not conforming to a block device.

Will file based images be considered for "vm" plugin? Looks like a few middlewared files need to be tweaked to make that happen as all python bindings for libvirt are in libvirt.py.

E.g. of attempting to modify the VM metadata to support file image:
<disk type='file' device='disk'>​
<driver name='qemu' type='qcow2' />​
<source file='/mnt/zfs_storage_pool/dataset_vm_images/oracleApex19c.qcow2'/>​
<target dev='vda' bus='virtio'/>​
<boot order='1'/>​
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>​
</disk>​

Error raised on frontend:
Error: Traceback (most recent call last):​
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor_base.py", line 152, in start​
if self.domain.create() < 0:​
File "/usr/lib/python3/dist-packages/libvirt.py", line 1234, in create​
if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)​
libvirt.libvirtError: internal error: process exited while connecting to monitor: 2020-12-30T22:24:18.784926Z qemu-system-x86_64: -blockdev {"driver":"host_device","filename":"/mnt/zfs_storage_pool/dataset_vm_images/nvr.qcow2","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: 'host_device' driver expects either a character or block device​
During handling of the above exception, another exception occurred:​
Traceback (most recent call last):​
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 137, in call_method​
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,​
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1215, in _call​
return await methodobj(*prepared_call.args)​
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1021, in nf​
return await f(*args, **kwargs)​
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 46, in start​
await self.middleware.run_in_thread(self._start, vm['name'])​
File "/usr/lib/python3/dist-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread​
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))​
File "/usr/lib/python3/dist-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run​
result = self.fn(*self.args, **self.kwargs)​
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 61, in _start​
self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))​
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor_base.py", line 161, in start​
raise CallError('\n'.join(errors))​
middlewared.service_exception.CallError: [EFAULT] internal error: process exited while connecting to monitor: 2020-12-30T22:24:18.784926Z qemu-system-x86_64: -blockdev {"driver":"host_device","filename":"/mnt/zfs_storage_pool/dataset_vm_images/nvr.qcow2","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: 'host_device' driver expects either a character or block device​
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
I can't tell you for 100% sure, but as I don't think gluster can handle zvols...
So it would mean for IX to implement the things they want, they eventually will need filestorage.

But to give you an idea: They also introduced filebacked iscsi on SCALE, so I guess it's comming.
You could make a suggestion on the Jira tracker, if you want to be sure it gets read by a developer eventually.
(or a PR, Or a slack shoutout)

A PR should be doableI think...
 

inman.turbo

Contributor
Joined
Aug 27, 2019
Messages
149
qcow2 should be the default when creating a vm. And we shouldn't stop there, there should be support for LVM as well. And why not? This is KVM after all.
 

PackElend

Explorer
Joined
Sep 23, 2020
Messages
60
could it be, that there will be a support of other virtual disk formats as there is truenas/pkg-qemu: https://salsa.debian.org/qemu-team/qemu (github.com) but it hasn't been updated for some months.
Otherwise this package could be used to include KVM in SCALE as KVN and QEMU share history as explaiendin What Is the Difference between QEMU and KVM? (packetflow.co.uk) .

If I understand correctly, qcow2 is superior to zvol, isn't. it? At least that is my understanding after reading to numerous discussions:

So basically, to create a VM around an existing virtual disk I still need to do:
  1. qemu-img convert: raw, qcow2, qed, vdi, vmdk, vhd to raw
  2. dd if=drive.raw of=/dev/zvol/volume2/zvol
migrate qcow2 image to zfs volume (github.com) won't work, will it?
What about creating a zero size zvol and add the into raw converted virtual disk as an additional disk device 16. Virtual Machines — TrueNAS®11.3-U5 User Guide Table of Contents (ixsystems.com)


I aks as I would like to Installing Home Assistant OS - Home Assistant (home-assistant.io) using a virtual appliance image as even QCOW2 are provided.
 

dquercus

Dabbler
Joined
Dec 31, 2020
Messages
13
could it be, that there will be a support of other virtual disk formats as there is truenas/pkg-qemu: https://salsa.debian.org/qemu-team/qemu (github.com) but it hasn't been updated for some months.
Otherwise this package could be used to include KVM in SCALE as KVN and QEMU share history as explaiendin What Is the Difference between QEMU and KVM? (packetflow.co.uk) .

If I understand correctly, qcow2 is superior to zvol, isn't. it? At least that is my understanding after reading to numerous discussions:

So basically, to create a VM around an existing virtual disk I still need to do:
  1. qemu-img convert: raw, qcow2, qed, vdi, vmdk, vhd to raw
  2. dd if=drive.raw of=/dev/zvol/volume2/zvol
migrate qcow2 image to zfs volume (github.com) won't work, will it?
What about creating a zero size zvol and add the into raw converted virtual disk as an additional disk device 16. Virtual Machines — TrueNAS®11.3-U5 User Guide Table of Contents (ixsystems.com)


I aks as I would like to Installing Home Assistant OS - Home Assistant (home-assistant.io) using a virtual appliance image as even QCOW2 are provided.

Did you make any progress on this? I'm in the same situation. Thanks.
 

PackElend

Explorer
Joined
Sep 23, 2020
Messages
60
progress on what?
Install the HA VM?

Nope I'm currently busy with replacing hardware and pulling cables and there is one cable conduit what drives me nuts...
 

dquercus

Dabbler
Joined
Dec 31, 2020
Messages
13
progress on what?
Install the HA VM?

Nope I'm currently busy with replacing hardware and pulling cables and there is one cable conduit what drives me nuts...
Yep. HA is my last VM running on ESXi and I'm really looking forward to have everything on the new Hardware with TrueNas Scale.

KVM + ZFS is new for me so I'm still learning about that and didn't try yet with HA.
 

Norumen

Dabbler
Joined
Dec 23, 2020
Messages
18
Anybody have a good step by step on how to convert a qcow2 file to something I can use in TrueNAS SCALE?
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
I got HomeAssistant running by using
Code:
sudo qemu-img convert -O raw hassos_ova-5.11.qcow2 /dev/zvol/main/HasOSS-f11jpf

on the zvol created for the VM. Still testing but at least the VM was able to boot.
That should work just fine :)
 

Norumen

Dabbler
Joined
Dec 23, 2020
Messages
18
I got HomeAssistant running by using
Code:
sudo qemu-img convert -O raw hassos_ova-5.11.qcow2 /dev/zvol/main/HasOSS-f11jpf

on the zvol created for the VM. Still testing but at least the VM was able to boot.

Tried that after downloading the file and unpacked the file.
Created a virtual machine and a new vdev, copied the file with the qemu-img command, and then mounted the drive to the virtual machine.

Get this on boot:
1613478230092.png
 

raudraido

Cadet
Joined
May 25, 2021
Messages
8
Tried that after downloading the file and unpacked the file.
Created a virtual machine and a new vdev, copied the file with the qemu-img command, and then mounted the drive to the virtual machine.

Get this on boot:
View attachment 45186
How it is done actually, I have my VM set up and I want to change VM zvol with my raw image
 

Ian Tamplin

Cadet
Joined
Jun 14, 2017
Messages
1
I got HomeAssistant running by using
Code:
sudo qemu-img convert -O raw hassos_ova-5.11.qcow2 /dev/zvol/main/HasOSS-f11jpf

on the zvol created for the VM. Still testing but at least the VM was able to boot.

Thanks mate this worked for me.
 

Ixian

Patron
Joined
May 11, 2015
Messages
218
Importing/exporting disks through the VM GUI is already a Jira enhancement, and it's been accepted. So it will likely show up in an RC down the line.

Whether that means native file/qcow2 or they are just going to automate the process to convert them via the GUI I don't know. Good question.

This and USB passthrough support are what's missing from the current beta (I'd also add in the ability to order VM startup, as well as gracefully shut them down when TrueNAS is rebooted).
 

Zain

Contributor
Joined
Mar 18, 2021
Messages
124
Hello,

Any idea why I might see the error below? Based on what is visible, I presumptuously increased the amount of ram available to the VM but it has no effect on alleviating the issue.

I'm asking here because I used the method above for the <attempted> install.

TIA

1646921705906.png
 
Top