VM Hang on install of certain OSes

[acoushawk]

Hello,

I recently updated to 23.10.1 and I've noticed a strange issue when trying to install certain OSes in virtual machines. When I hit a certain point in the installer it completely locks up the VM and qemu-system-x86 spikes to 100% for that instance. I have tried to check the stack trace and it seems to be getting errors on read however I'm not able to determine the issue beyond that point. I've tested this using various combinations of disk locations, nic types, disk types, and cpu/mem confirmations and nothing seems to make any difference.

The weird part is if I install rocky linux it will execute fine however trying to install ubuntu 22.04 or 23.10 as well as nixos all seem to hang during install.

I also noticed that my system did an update around December 18th for libvirt and qemu so maybe that is the issue? I honestly haven't attempted to create a VM since then so that could be valid.

Another strange thing is that any previously created instances, regardless of OS, are running fine and are able to reboot etc.. without issue.

Any ideas, help, guidance on where to look next would be super appreciated!

Specs:
OS Version:TrueNAS-SCALE-23.10.1
Product:B760M PG Lightning
Model:12th Gen Intel(R) Core(TM) i9-12900K
Memory:62 GiB

 

[Whattteva]

What kind of storage are you installing to? You might be running into IOPS issues and are just crawling, creating a "hung" impression.
Consumer grade SSD's are especially notorious for having terrible IOPS once the write cache runs out in a write-heavy situation and can end up even slower than HDD's, which includes installing an OS.

 

[acoushawk]

@Whattteva - So I have a RAIDz with spinning disks and a NVMe drive I use for the VMs. I thought the same thing as you that it could be storage contention but I tried to installed to the RAIDz and there was no difference. I also shut down a bunch of the other VMs I have running and it still wouldn't allow the install to continue.

What is also strange is that if I install rocky linux it goes through ok. There seems to be something in the way that the ubuntu installer installs that hits this issue.. but only with the update to qemu or libvirt.

Not sure if it's possible but I wonder if I revert back to the previous version if that will work. I'll look into the docs and see if that is possible with truenas. I did revert back to the previous version of scale but that didn't revert back the packages.

 

[Whattteva]

Can you install other OS's like Windows and have it succeed just to be sure that it's a Rocky Linux thing?

 

[acoushawk]

@Whattteva Thanks for the suggestion! So I did try a Windows VM and it worked fine. So I went back and started adjusting the CPU/MEM and it looks like ubuntu and nixos stall out when there is less than 2 CPUs. I think like you said, it doesn't quite die but just gets so slowed down with CPU contention that it looks like it.

Thanks again for the responses!!

 

[Whattteva]

Glad you worked that out.