Sudden issue to start VM

[impestrator]

Hello,

for about two weeks ago I´ve migrated my home assistant instance from Rasberry PI to virtual machine on Truenas Scale. It took some work, but it was runnning smoothly. Till today morning the virtual machine is off. IfI try to start it I´ll get the following error message:

[EFAULT] Unable to define domain for HAOS: cannot serialize None (type NoneType)

Code:

Call Error
[EFAULT] Unable to define domain for HAOS: cannot serialize None (type NoneType)
Fehler: Traceback (most recent call last):
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1052, in _escape_attrib
    if "&" in text:
TypeError: argument of type 'NoneType' is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 45, in _check_add_domain
    self._add(self._vm_from_name(vm_name)['id'])
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 16, in _add
    self._add_with_vm_data(vm)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 19, in _add_with_vm_data
    self.vms[vm['name']] = VMSupervisor(vm, self.middleware)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 30, in __init__
    self.update_domain()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 60, in update_domain
    self.__define_domain()
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 92, in __define_domain
    vm_xml = etree.tostring(self.construct_xml()).decode()
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1109, in tostring
    ElementTree(element).write(stream, encoding,
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 748, in write
    serialize(write, self._root, qnames, namespaces,
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 913, in _serialize_xml
    _serialize_xml(write, e, qnames, None,
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 913, in _serialize_xml
    _serialize_xml(write, e, qnames, None,
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 913, in _serialize_xml
    _serialize_xml(write, e, qnames, None,
  [Previous line repeated 1 more time]
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 906, in _serialize_xml
    v = _escape_attrib(v)
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1075, in _escape_attrib
    _raise_serialization_error(text)
  File "/usr/lib/python3.9/xml/etree/ElementTree.py", line 1029, in _raise_serialization_error
    raise TypeError(
TypeError: cannot serialize None (type NoneType)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 184, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1317, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1247, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 46, in start
    await self.middleware.run_in_thread(self._start, vm['name'])
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1234, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1231, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 67, in _start
    self._check_add_domain(vm_name)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 47, in _check_add_domain
    raise CallError(f'Unable to define domain for {vm_name}: {e}')
middlewared.service_exception.CallError: [EFAULT] Unable to define domain for HAOS: cannot serialize None (type NoneType)

Does anyone knows what happend and how can I fix it? My last backup is 14 days old I don´t want to setup a new machine.

Regards
Martin

 

[chuck32]

Hello,

for about two weeks ago I´ve migrated my home assistant instance from Rasberry PI to virtual machine on Truenas Scale.

Just for clarification: You posted under Virtualized Truenas. Is your truenas instance a bare metal installation or virtualized? Not that I suspect that that matters here, but still.

How did you install HAOS? I followed this guide so I have created a zvol that I used rather than using an installation image during VM creation. I'm not 100% clear on how to describe this setup: HAOS as an appliance? If you have the same / similiar install: Do not despair, your non existing backup does not matter!

First of all to get you running again:

1. for good measure, make a backup of your HAOS zvol
2. detach the disk from the HAOS VM
3. Go to devices and note your MAC address of the attached NIC
4. delete the VM (do not forget to detach the disk before to avoid the disk beeing deleted here, because I checked the destroy data button then)
5. now you have the zvol, you old NIC mac address and no VM
6. Create a new VM, choose use existing zvol and point it to your previously detached zvol
7. During creation, when attaching a NIC: enter your previously copied mac address
8. Start the VM

If all went well you should be up and running like nothing happened. Since you reused the mac address (and that's really the beauty here) your VM should get the same fixed IP address as before -> no reconfiguring of mqtt devices or of client devices. The best part, if you use the HA app on android for example: You do not need to log in again or something like that, everything works as before.

Then two important things: create more recent backups of HAOS! Under settings, storagae (:8123/config/storage) you can attach a SMB share. Create an automation that will trigger the backup service (available in the newer HAOS versions). Or do it differently, however you want, but make sure to have recent backups.
See this issue as for the naming and retention of the backups.

Code:

alias: auto-backup
description: automatisierte backups
trigger:
  - platform: time
    at: "07:00:00"
condition:
  - condition: time
    weekday:
      - mon
      - fri
action:
  - service: hassio.backup_full
    data:
      compressed: true
mode: single

Maybe use the yaml for the automation but use the graphical interace afterwards to make sure you choose your SMB location as the destination for the backup.

Secondly: figure out why the VM stopped. Except for some custom integrations misbehaving (with no impact on everyday usage for me) HAOS has been rock stable on truenas.

And on a third point: I face the exact same issue as you. You can look up details here.

I suspected my hardware upgrade (and the fact that my systemdata set lived on my boot pool) was the cause for the issues. I have not resolved it yet. What i can tell you though is this, usually HAOS will not survive a reboot. It may though. So my theory is, that maybe it was working / stable before upgrading to a certain scale version. I cannot for sure say which one, because I rarely reboot my server and as I said, there is a slim chance
that HAOS will survive a reboot or two.

Thank you for posting, maybe there is a wider spread issue with a recent update in scale rather than an individual problem on my site.

Good luck and I hope we can get the issue resolved alltogether. My proposed fix for getting HA OS should at least not put pressure on us. It's an extra 5 min after a reboot to recreate the VM and reattach the USB devices (unfortunately you cannot do that on creation so you have to reboot the VM one time in between).

 

[impestrator]

Hi Chuck32,

first I need to apologize to raise this thread within the wrog topic. My Truenas instance run on bare metal (HP GEN8). I followed the exact guide you posted.

Second: Thank you very much for helping me out with that one. I followed your instructions (except my screen capturing tool didn´t work when I wanted to capture the MAC address, so I´ve to set up network configuration again - doesn´t matter) and Home Assistant back again.

Third: You´re absolutely right, I´ve to create more recent backups. In the last 14 days I just added some KNX device (just adding group addresses) and add then to the dashboard, so no major changes. I just pluged in my Conbee2 stick yesterday which ran over 4 years within my PI flawless. So maybe this could be the root cause. But I didn´t think that would break my installation, so I didn´t create a backup. I´ll try that again later, then I´ll have more information about that.

Thanks for sharing your backup automation. I´ll try that later that day.

 

[chuck32]

First I need to apologize to raise this thread within the wrog topic. My Truenas instance run on bare metal (HP GEN8). I followed the exact guide you posted.

No need, it's just folks around here like to have all the correct information necessary and whether you are running baremetal or not often plays a huge role.

(except my screen capturing tool didn´t work when I wanted to capture the MAC address, so I´ve to set up network configuration again - doesn´t matter) and Home Assistant back again.

Wait on that. Go to your router and find your Network. Somewhere should be option to see all devices known to your Network. Look for your old HAOS instance, if you click on it it should display the mac Adress. I had to go this route the first time ;) I wanted to save you the hassle.

Repeat the VM recreation (if you already began making changes use the backuped zvol).

Or reconfigure everything, that's up to you, but I would have had to update the IP on all my separate mqtt devices and I was too lazy for that.

just pluged in my Conbee2 stick yesterday which ran over 4 years within my PI flawless. So maybe this could be the root cause.

You would need to check the logs but I doubt that it would have caused a crash.


Maybe one of us should open a Jira ticket?

 

[mindstormsguy]

Interestingly, the exact same thing happened to me last night when I allowed HAOS to update itself from the HA web GUI.

[EFAULT] Unable to define domain for haos: cannot serialize None (type NoneType)

I've update HAOS in my TrueNAS vm many times in the past with no problems. I've also rebooted it countless times (ok, maybe not _that_ often, but as often as might be "normal" for VM or TrueNAS OS updates), and never had a problem until now.

Perhaps there's something weird in the new HAOS update? (Though I guess I don't understand enough about virtualization to understand why the guest OS could prevent the VM itself from even starting).

 

[chuck32]

Truenas: Jira Ticket

Homeassistant: Github Issue

 

[stergio]

Got the same issue message with a running Ubuntu install. Same error message see pic

Ubuntu was running great, did some rsync commands and shut down the VM from Ubuntu. Power off.

Seperate issue noticed my files date times were off, and found another thread to update my bios time to reflect UTC time. Restarted the machine changed the bios time to match utc, and logged back into truenas.

Was curious how my VMs would see this time change and noticed this issue.

Had the same issue before, this is the 2nd time Ive installed ubuntu on Truenas. This is cant be normal?!

 

[chuck32]

Haven't encountered the issue anymore on 23.10.1

Did you check your USB devices?

 

[stergio]

Hey Chuck, yes, that did it, and good to know. I deleted the USB device in the devices menu, and the VM fired right up. Is this expected behavior or something we need to workaround when no longer using a usb device? (From the picture attached deleted device 12-USB from the devices list)

Some history on how I got to my sitution... I had added an external hard drive and in devices for the VM added a USB device with passthrough to transfer some files using terminal rsync commands. Great transfer speeds. After finishing that task i did "power off" in ubuntu, and disconnected the drive from the server.

Very curious why this behavior is this way. Is there documentation you are aware of that you can point to that talks about this?

Apollogies for my newb questions. I am still new to Trunas, upgraded hardware to a supermicro, a for real server homelab setup, and upgraded the software from the old freenas from 2009, a lot simpler times . Big learning curve from the freenas 0.7.2 version.

Thank you very much!!

 

[tprelog]

I deleted the USB device in the devices menu, and the VM fired right up. Is this expected behavior

I don't know if it's a bug or not, but the way TrueNAS handles everything from the UI, I'm not surprised you would have to delete the USB from VM settings if there is no longer a USB device attached.

 

[Darren David]

This helped me track down a similar issue. Not a very helpful error from TrueNAS, but thank you for pointing me at the USB devices.