Can't mount ISO to VM: Permission denied

[Ruuddie]

I have uploaded an ISO file to a SMB share, via SMB. When I try to mount the ISO, I get a 'Permission denied' error. My assumption is the VM runs under some 'weird' service user, which I need to allow on the SMB datastore.

Any clue how this works?

Full error log:

Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor_base.py", line 166, in start
if self.domain.create() < 0:
File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create
raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-01-28T15:22:02.263506Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/mnt/Pool-5TB/Pool-5TB/ISO/virtio-win-0.1.215.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/mnt/Pool-5TB/Pool-5TB/ISO/virtio-win-0.1.215.iso': Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 160, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1281, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1269, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1137, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 42, in start
await self.middleware.run_in_thread(self._start, vm['name'])
File "/usr/lib/python3/dist-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 62, in _start
self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name))
File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor_base.py", line 175, in start
raise CallError('\n'.join(errors))
middlewared.service_exception.CallError: [EFAULT] internal error: process exited while connecting to monitor: 2022-01-28T15:22:02.263506Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/mnt/Pool-5TB/Pool-5TB/ISO/virtio-win-0.1.215.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/mnt/Pool-5TB/Pool-5TB/ISO/virtio-win-0.1.215.iso': Permission denied

 

[sretalla]

Seems to me (from looking at htop) that VMs run as a user called libvirt-q

You could either chmod the .iso to 777 (since I guess there aren't any secrets on it that everyone should be prevented from seeing) or chown it to that user.

 

[anodos]

The user in question will need execute across every path component for ISO file (each dir), and read permissions on the ISO itself.

 

[dirtyfreebooter]

i did find it weird that i tried a VM the other day and once i started the VM, it changed the ownership of the ISO file to libvirt-q:libvirt-q which for a read-only thing seems a bit odd.

 

[anodos]

i did find it weird that i tried a VM the other day and once i started the VM, it changed the ownership of the ISO file to libvirt-q:libvirt-q which for a read-only thing seems a bit odd.

This is normal behavior for QEMU IIRC.

 

[dirtyfreebooter]

Bug #1784001 “Libvirt changes ownership (user/group) of my ISO i...” : Bugs : libvirt package : Ubuntu

Libvirt should NOT be modifying ownership of my ISO images when mounting them to install a KVM instance. This causes issues later when I use zsync to update the images from the latest bits at cdimages.u.c. More importantly, my files should not arbitrarily be stolen from me when they're used BY...

bugs.launchpad.net

Code:

Looking in /etc/libvirt/qemu.conf, this setting would fix this... why is the default not 0 here:

# Whether libvirt should dynamically change file ownership
# to match the configured user/group above. Defaults to 1.
# Set to 0 to disable file ownership changes.
#dynamic_ownership = 1

The reason stated for that default is

for dynamic_ownership "why is the default not 0 here"
=> Because that is the upstream default and actually fixing issues with image permissions for a lot of people.

 

[anodos]

Bug #1784001 “Libvirt changes ownership (user/group) of my ISO i...” : Bugs : libvirt package : Ubuntu

Libvirt should NOT be modifying ownership of my ISO images when mounting them to install a KVM instance. This causes issues later when I use zsync to update the images from the latest bits at cdimages.u.c. More importantly, my files should not arbitrarily be stolen from me when they're used BY...

bugs.launchpad.net

Code:

Looking in /etc/libvirt/qemu.conf, this setting would fix this... why is the default not 0 here:

# Whether libvirt should dynamically change file ownership
# to match the configured user/group above. Defaults to 1.
# Set to 0 to disable file ownership changes.
#dynamic_ownership = 1

The reason stated for that default is

Can you file a jira ticket with that information? That way the person responsible for that area of the product can see this info / make a decision.

 

[dirtyfreebooter]

https://jira.ixsystems.com/browse/NAS-114532

 

[Ruuddie]

Seems to me (from looking at htop) that VMs run as a user called libvirt-q

You could either chmod the .iso to 777 (since I guess there aren't any secrets on it that everyone should be prevented from seeing) or chown it to that user.

Thanks for the tip! Unfortunately, I can't seem to change chmod of the ISO directory and it's files:

root@truenas[/mnt/Pool-5TB/Pool-5TB]# chmod -R 777 ISO
chmod: changing permissions of 'ISO': Operation not permitted
chmod: changing permissions of 'ISO/en-us_windows_server_2022_updated_jan_2022_x64_dvd_f7ca3012.iso': Operation not permitted
chmod: changing permissions of 'ISO/virtio-win-0.1.215.iso': Operation not permitted
root@truenas[/mnt/Pool-5TB/Pool-5TB]# ls -l
total 25
drwx------ 2 root root 4 Jan 28 12:36 ISO

I see the user 'vibvirt-qemu' is owner of the ISO. I am no Linux guru, but I think I changed it to root, but still I couldn't change the permissions:

root@truenas[/mnt/Pool-5TB/Pool-5TB/ISO]# ls -l
total 5030309
-rwx------ 1 root root 5034006528 Jan 27 20:34 en-us_windows_server_2022_updated_jan_2022_x64_dvd_f7ca3012.iso
-rwx------ 1 libvirt-qemu libvirt-qemu 541001728 Jan 28 12:00 virtio-win-0.1.215.iso
root@truenas[/mnt/Pool-5TB/Pool-5TB/ISO]# chmod 777 virtio-win-0.1.215.iso
chmod: changing permissions of 'virtio-win-0.1.215.iso': Operation not permitted
root@truenas[/mnt/Pool-5TB/Pool-5TB/ISO]# chown root virtio-win-0.1.215.iso
root@truenas[/mnt/Pool-5TB/Pool-5TB/ISO]# ls -l
total 5030309
-rwx------ 1 root root 5034006528 Jan 27 20:34 en-us_windows_server_2022_updated_jan_2022_x64_dvd_f7ca3012.iso
-rwx------ 1 root libvirt-qemu 541001728 Jan 28 12:00 virtio-win-0.1.215.iso
root@truenas[/mnt/Pool-5TB/Pool-5TB/ISO]# chmod 777 virtio-win-0.1.215.iso
chmod: changing permissions of 'virtio-win-0.1.215.iso': Operation not permitted

 

[boggie1688]

I'm running into this exact issue too.

I'm not a VM guru or anything, I just wanted to try to run Zoneminder in a VM seeing as how there isn't an app for it on Scale.

I also tried to create a VM using a Win10 ISO, and kept running into the same issue. At first I thought it was a user/group permission problem, so I kept trying to apply the correct user/group to the dataset where the iso was stored. I then realized that for some reason, after trying to start the VM, the permission was changing to libvirt-gemu as mentioned above.

I have nothing insightful to add besides the fact that I am able to replicate this same issue.

 

[titust1]

I have the same issue in Truenas Scale (released version) today. I thought this issue was fixed

 

[anodos]

I have the same issue in Truenas Scale (released version) today. I thought this issue was fixed

From the linked jira ticket above:

i have been playing around with toggling dynamic ownership flag and we run into other issues when starting / stopping vms as libvirt uses that flag extensively to access zvols, create/manage files/dirs it creates temporarily.

After talking with the team, we think it's best if this is handled/requested upstream - so please feel free to create an issue upstream as changing that flag produces lots of problems in other places where on zfs libvirt is not able to manage the vms. Thanks!

This means that the libvirt behavior has been kept the same since it introduced other issues. This means that you must ensure that the libvirt-qemu user has appropriate access to the path you are trying to use as an iso repository.

 

[taaangy234]

I have the same problem updating from TrueNAS Core 12 few days ago. Must be a bug when upgrading. Does anyone having same problem install a fresh copy of TrueNAS Scale then restore, and importing previous ZFS? I initially upload the ISO to my /mnt/NAS/Data1/AppLib/ISO/ubuntu-22.04-desktop-amd64.iso, it was able to upload the file to it, but can't read. Does that mean it uses different account to upload then libvirt-qemu to read the ISO?

I followed the kb gave libvirt-qemu modify/full access without any luck.
I checked /etc/libvirt/qemu.conf #dynamic_ownership = 1 is the default.
I've Strip ACL, and reset ACL

I'm stuck! :(

 

[nferocious76]

I have the very same problem and can't get it to work. Any luck on these issues?

 

[anodos]

What are permissions on each path component leading to the file? If the user lacks execute on anything (for instance /mnt/NAS or /mnt/NAS/Data1/AppLib) then access will be denied by the OS.

 

[taaangy234]

My ACL was set at /mnt/NAS/Data1

I'm currently backing up all my data to an external HD then start fresh. But my hope is a fresh TrueNAS Scale install, able to import the ZFS, if permission still stuck to that ZFS pool, I might just have to blow it away

 

[taaangy234]

I created a new dataset /mnt/NAS/ISO then upload an iso to it still same error.

 

[skoop]

Hi,

Had the same issue today with the ISO permission and also when i tried to "chmod" with the root user got "Operation not permitted".

I have tired what @anodos said and it worked.

used the GUI, attached screenshot.
In my ACL permission i have "Group - builtin_users" with Full control.

What i did, searched for the "libvirt-qemu" user in users credentials and add this user to "builtin_users" as an "Auxiliary Groups".

Then it worked.

Hope this help

 

[nferocious76]

Thanks. It really is really a permission problem. I have it working

 

[hristo.mirchev]

Hello, I found a solution for myself. I needed to include user libvirt-qemu to all of the parent datasets and then it worked!