Boot crash-loop with VM PCI pass-through

[s0_]

Hello,

I have a VM set up on my TrueNAS Scale Bluefin box. it is set to start on boot automatically.
The problem is, I attempted to pass-through a PCI device of a graphics card to it.
It seems I selected the wrong PCI device, because after saving the change and restarting the VM, the entire server crashed.
Unfortunately, because the VM is set to start on boot, now it's in a loop where TrueNAS reboots, starts the VM, and crashes again.

I've shut it down and I'm attempting to mount the boot disk on my Debian machine now but I'd appreciate any suggestions!
All I need is to either reach a shell with the boot disk mounted but not storage (can I do this from the grub boot arguments or something?), and find where the VM XML is stored, or to boot in such a way as to prevent the VM from starting. Or mount the boot disk on my other machine and edit the same XML.

 

[s0_]

Solved by:

1. Removed boot disk (SATA SSD).
2. Connected disk to my Debian machine with USB adapter.
3. Installed zfs and zfsutils-linux from apt.
4. Forced zfs to import pool from device - sudo zpool import -d /dev/sdb boot-pool -f
5. Changed mountpoint of correct version to usable mountpoint on my machine (not necessary -- should have just created /mnt/<host>/bootpool/ROOT and used original mountpoint!) - sudo zfs mountpoint=/mnt/<tempdir> boot-pool/ROOT/22.12.0
6. mounted (might have been automatic?) - sudo zfs mount boot-pool/ROOT/22.12.0
7. browsed the root drive in /mnt/<tempdir>. Found /data/freenas-v1.db -- promising!
8. Opened /data/freenas-v1.db in sqlite-browser.
9. Changed autostart to 0 in the row of my VM in the vm-vms table, and committed.
10. Unmounted boot-pool/ROOT/22.12.0
11. Changed back the zfs mountpoints as they were originally (I didn't do this the first time but had to come back and fix it later!)
12. Exported the pool - sudo zfs export boot-pool
13. Disconnected the disk.
14. Replaced boot disk in server.

As noted, the first time the server wouldn't boot again correctly because I'd changed the zfs mountpoints, not realising they were persistent.
I also had to unplug the graphics card in the server because the BIOS was trying to use it for boot console, so I couldn't see the boot output on the mainboard VGA out/ILO console display. I fixed these issues and TrueNAS booted back up again successfully, with the VM disabled.

In retrospect, I should have just removed the graphics card in the first place, and tried booting, in case that worked!

 

[kevin4324234]

I'm having the same issue, but I don't have a Linux machine.

Is there any way to fix this without one?

 

[s0_]

You’ll need to use the ZFS packages/tools for your OS distribution. Alternatively you could simply make a Ubuntu (or other linux) bootable USB, and boot that on your server, then run similar commands to my list, without having to remove the boot disk physically.