Freenas - to - TrueNas Virtual machine failed to connect to libvirt

S

Sommerer

Cadet
Joined
Jul 19, 2020
Messages
2
Good day,

I decided to upgrade my FreeNas to TrueNas beta and have run into an odd issue. When going to Virtual Machines and trying to start one of my Windows 10 Virtual machines I get the message "CallError" [EFAULT] Failed to connect to libvirt"

Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1198, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1102, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1511, in start
    self.ensure_libvirt_connection()
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1494, in ensure_libvirt_connection
    raise CallError('Failed to connect to libvirt')
middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt


My first though was that there was an issue with the Virtual Machine Config after the upgrade so decide I'd try just a clean Virtual machine Windows 10 to see if that would boot. Even creating a new Virtual machine results in the "[EFAULT] Failed to connect to libvirt"

Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1187, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 449, in create
    rv = await self.middleware._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1187, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1207, in do_create
    await self.middleware.call('vm.ensure_libvirt_connection')
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1229, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1198, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1102, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1494, in ensure_libvirt_connection
    raise CallError('Failed to connect to libvirt')
middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt



Didn't know if anyone else has run into this?
 
W

warllo

Contributor
Joined
Nov 22, 2012
Messages
117
I'm having the same troubles with all of my VM's as of this morning. Haven't figured out why yet. I've been running the 12.0 Beta for weeks now without any issues. After a reboot now I'm getting this. The odd thing is my VM is actually running even though the UI shows it isn't
 
Last edited:
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Have you cleared the browser cache? Various bits often go out of sync with these updates.

If you have, file a bug report and post the link here.
 
H

Hendrixx

Dabbler
Joined
Jul 6, 2020
Messages
32
Still have this problem on my TrueNAS-12.0-RC1 (Beta) system.
Most of the time a restart of the server fixes the problem, but not always.
Yesterday i had same problem again and after 3 reboots i still had the problem that my 2 VMs didn't autostart.

When i disabled the autostart en rebooted the server then i finally could start my 2 VMs manually.
 
H

Hendrixx

Dabbler
Joined
Jul 6, 2020
Messages
32
I had to restart my TrueNAS-12.0-RC1 Beta server again and this time when i tried to manually start my 2 VMs i got the same error again :(

Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1202, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1106, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1511, in start
    self.ensure_libvirt_connection()
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1494, in ensure_libvirt_connection
    raise CallError('Failed to connect to libvirt')
middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt


So this issue still exists for me, any ideas?
 
S

Strub3l

Cadet
Joined
Oct 20, 2020
Messages
1
Having the same issue


Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1202, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1106, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1511, in start
    self.ensure_libvirt_connection()
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1494, in ensure_libvirt_connection
    raise CallError('Failed to connect to libvirt')
middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt


Any solutions for it?
 
O

olly86

Dabbler
Joined
Dec 27, 2012
Messages
12
I'm seeing the same error after updating to TrueNAS 12 Release when trying to start my VM. The VM is also set to auto start and isn't, I've tried multiple reboots and cleared my browser cache.

Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1202, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1106, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1511, in start
    self.ensure_libvirt_connection()
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1494, in ensure_libvirt_connection
    raise CallError('Failed to connect to libvirt')
middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt
 
H

Hendrixx

Dabbler
Joined
Jul 6, 2020
Messages
32
I've updated to the TrueNAS 12 Release (Stable) and it seems the problem is solved for me.
Restarted my TrueNAS server 2 times and the 2 VM's started automatically for me without problems.
 
S

seb101

Contributor
Joined
Jun 29, 2019
Messages
142
I am also running into the same error preventing me from stopping/restarting VMs.
 
J

Jon Moog

Dabbler
Joined
Apr 24, 2017
Messages
21
Seeing what appears to be the same error on 12 Release trying to submit changes for a new VM. The GUI for VM status shows others as off now despite them clearly running. I've tried creating VMs with and without Virt interfaces with the same effect. The two running VMs where created prior to the update to 12 and have started at boot.

Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 455, in create
    rv = await self.middleware._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1207, in do_create
    await self.middleware.call('vm.ensure_libvirt_connection')
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1233, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1202, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1106, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1494, in ensure_libvirt_connection
    raise CallError('Failed to connect to libvirt')
middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt
 
H

HannesM

Dabbler
Joined
Jul 4, 2017
Messages
24
i've upgraded from FreeNAS 11.3-U5 to TrueNAS 12.0 Release a view days ago. now, the VM systems (not the Jails) are not starting after a reboot. they are offline and show me the error message
"middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt"

any idea what happend and how to fix it ?

thx and regards
Hannes
 
K

Kirk_K

Cadet
Joined
Oct 14, 2020
Messages
3
I am super satisfied with the functionality, and I am very grateful to the developers and the people helping them.

Regarding this issue, In my case, VM starts automatically very rare, so I turned off the autostart.
Now after powering on, I am turning it on manually, but sometimes I need to restart my storage a few times so I can turn my VM on.
With FreeNAS 11.3, I haven't had this problem, but I had a problem with autostart of SMART and SMB, and that was the reason why I updated to TrueNas12 release.

Have no idea what's happening. It was perfect with FreeNAS 11.2, but now it is too late to go back as I updated my pool in TrueNas 12

btw, the error is: [EFAULT] Failed to connect to libvirt

Thanks,
Kirk
 
H

HannesM

Dabbler
Joined
Jul 4, 2017
Messages
24
Hi Kirk,
i agree :smile: until 11.3-U5 the VMs are workling well and with the upgrade the hypervisor is works according to the random principle. :confused:
what this is caused by - would now be too much speculation :wink:

i try to disable the autostart and hope my productive vms are comming up and online.

Regards
Hannes
 
E

ertank

Explorer
Joined
Sep 16, 2018
Messages
66
Ericloewe said:
Have you cleared the browser cache? Various bits often go out of sync with these updates.

If you have, file a bug report and post the link here.
Click to expand...
I got exact same problem. I was installing VM after my upgrade to TrueNAS 12.0-RELEASE. After a few days problem disappeared. I did not clear my browser cache though. Probably it is cleared automatically.
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Poor cache invalidation, you mean? No, that's outside my field of expertise. I do know that there have been several attempts over the years to correct this issue, but they never seem to have been fully satisfactory.
 
H

HannesM

Dabbler
Joined
Jul 4, 2017
Messages
24
on my system (32GB ECC RAM) the ZFS use more that 50% (18 GB of 32GB) - that's a bit heavy and perhapse the ZFS of FreeNAS was better implemented and do not crache VM Systems or do nit start them with my reported issue.

Hope iX will fix the issue.

Regards
Hannes
 
H

HannesM

Dabbler
Joined
Jul 4, 2017
Messages
24
Hi all,
i've restarted an VM without rebooting the TureNAS host completly :grin:
but what happend is a challange for iX Systems engineering :wink:

until i got the reported error i've connected trough SSH to the TrueNAS directly and ckecked the following

$ service middlewared status
middlewared is running as pid xxxx (PID from host).

$ service middlewared restart
Stopping middlewared.
Waiting for PIDS: xxxx,yyyy.

Now, the issue is fixed and my VM is starting until the next crash by middleware service on the TrueNAS host.

Regards
Hannes
 
O

olly86

Dabbler
Joined
Dec 27, 2012
Messages
12
Hi

The tips Hannes posted worked for a short time, but within about 10 mins I'm getting the "Failed to connect to libvirt" error:

# service middlewared status
middlewared is running as pid 213

# service middlewared restart
Stopping middlewared.
Waiting for PIDS: 213.
Middleware startup is idle for more than 240 seconds
##############################################################
MIDDLEWARED FAILED TO START, SYSTEM WILL NOT BEHAVE CORRECTLY!
##############################################################

# service middlewared status
middlewared is running as pid 69611

EDIT: restarting the service before rebooting the VM for updates seems to work.
 
Last edited:
You must log in or register to reply here.

Similar threads

julian35152
VM remove error: "dataset does not exist" after data loss
Replies
3
Views
2K
julian35152
julian35152
F
12.0-U1 Deleting VM results in error
Replies
1
Views
947
Flashbin
F
D
Swap out defective HDD
Replies
4
Views
2K
fillg1
F
L
Cannot create NFS share (sqlite3.OperationalError)
Replies
1
Views
1K
lpwevers
L
J
Win 10 VM won't start after migration to TrueNas
Replies
2
Views
1K
jasonsansone
J
Top