Freenas - to - TrueNas Virtual machine failed to connect to libvirt

[HannesM]

Hi

The tips Hannes posted worked for a short time, but within about 10 mins I'm getting the "Failed to connect to libvirt" error:

# service middlewared status
middlewared is running as pid 213

# service middlewared restart
Stopping middlewared.
Waiting for PIDS: 213.
Middleware startup is idle for more than 240 seconds
##############################################################
MIDDLEWARED FAILED TO START, SYSTEM WILL NOT BEHAVE CORRECTLY!
##############################################################

# service middlewared status
middlewared is running as pid 69611

EDIT: restarting the service before rebooting the VM for updates seems to work.

Hi Olly86,
i've the issue a view days ago.... the restart option was not really working in this case. i've used
$ service middlewared stop
wait a view seconds
$ service middlewared start

after this my VMs can restarted. I'm looking forward when the issue is comeing back... knock-knock-knoc 5 day's without broken middleware service.

Regards
Hannes

 

[olly86]

Cheers Hannes, that doesn't seem to help mine either I'm needing to do a full system reboot and then restart the service. The VM seems to stay running ok, until it needs a reboot for an update, hopefully a fix is in the works.

 

[fahadshery]

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1195, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 455, in create
    rv = await self.middleware._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1195, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1229, in do_create
    await self.middleware.call('vm.ensure_libvirt_connection')
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1206, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1516, in ensure_libvirt_connection
    raise CallError('Failed to connect to libvirt')
middlewared.service_exception.CallError: [EFAULT] Failed to connect to libvirt

having same issue. TrueNAS-12.0-U1

 

[fahadshery]

resolved it by:

Code:

 service middlewared restart

 

[ChrisRJ]

Poor cache invalidation, you mean? No, that's outside my field of expertise. I do know that there have been several attempts over the years to correct this issue, but they never seem to have been fully satisfactory.

The three most underestimated problems in software development:

• Choosing good names
• Dependency management
• Cache invalidation

 

[MGYVR]

Dunno if discussion of this has continued in another thread or what, but as of now I am still having this issue on TrueNAS-12.0-U2.1.

Dell R730xt
2x Xeon CPU E5-2680 v3 @ 2.50GHz
64GB RAM

System will run for one to several weeks at a time before middlewared gets wonky, and stops showing correct VM statuses.

The restart command does not really work for me, as the service won't automatically start again after shutting down for some reason.

Code:

system middlewared restart

But manually stopping and starting it does the trick.

Code:

system middlewared stop
system middlewared start

 

[hescominsoon]

Dunno if discussion of this has continued in another thread or what, but as of now I am still having this issue on TrueNAS-12.0-U2.1.

Dell R730xt
2x Xeon CPU E5-2680 v3 @ 2.50GHz
64GB RAM

System will run for one to several weeks at a time before middlewared gets wonky, and stops showing correct VM statuses.

The restart command does not really work for me, as the service won't automatically start again after shutting down for some reason.

Code:

system middlewared restart

But manually stopping and starting it does the trick.

Code:

system middlewared stop
system middlewared start

I have reported this bug as wlel in jira. I have an r520(noted below) and it happens even with the u2.1 update on here.

 

[olly86]

I can't remember which update resolved it, but mine has stopped acting up and is working normally.