Cannot boot or edit VM configuration after upgrade to 13.0-U2

cb88

cb88

Dabbler
Joined
May 11, 2022
Messages
24
I rolled back my install to 12.0-U8.1 and it continued working as it had.

I have 2 VMs a pi-hole VM and a pfSense VM with ports passed through to the latter.

The Pi Hole VM works now in 13.0-U2 but the pfSense VM still fails to boot. And yes I am aware that pfSense is misspelled... but that should not be why it fails since it works fine in 12.0.

1661957327169.png


It also fails to edit the VM configuration.

1661957434710.png
 
J

Juan Manuel Palacios

Contributor
Joined
May 29, 2017
Messages
146
Hi @cb88,

I'm also running a pfSense VM on my TrueNAS system, and I just upgraded to 13.0-U2 and everything worked fine. There is an important difference, though, and that is that I run my pfSense VM outside of the TrueNAS GUI, controlling it with my own command-line scripts (the TrueNAS GUI does execute those as Init/Shutdown Scripts), to be able to perform NIC passthrough to the VM (the GUI doesn't support that, as far as I can see). And that setup didn't experience any problems when upgrading to 13.0-U2 just now.

So, perhaps what's failing for you is the TrueNAS VM middleware, but not the VM itself, so you might still be able to control it via de command-line.

HTH!
 
cb88

cb88

Dabbler
Joined
May 11, 2022
Messages
24
Juan Manuel Palacios said:
Hi @cb88,

I'm also running a pfSense VM on my TrueNAS system, and I just upgraded to 13.0-U2 and everything worked fine. There is an important difference, though, and that is that I run my pfSense VM outside of the TrueNAS GUI, controlling it with my own command-line scripts (the TrueNAS GUI does execute those as Init/Shutdown Scripts), to be able to perform NIC passthrough to the VM (the GUI doesn't support that, as far as I can see). And that setup didn't experience any problems when upgrading to 13.0-U2 just now.

So, perhaps what's failing for you is the TrueNAS VM middleware, but not the VM itself, so you might still be able to control it via de command-line.

HTH!
Click to expand...
That seems likely and I did tag it with middleware in the ticket.... I was aware of that much at least last time I mentioned this issue for the 13.0-Release version.
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Hmm. I didn't have any problems with my VMs after the upgrade U1.1->U.2.
 
cb88

cb88

Dabbler
Joined
May 11, 2022
Messages
24
Note this problem is between 12 and 13... of any version. I originally had the problem with 13.0-Release then downgraded back to 12.0-U8 where the problem continues to not exist and both VMs in question boot without issue.

I have not been running 13 at all regularly and this was an upgrade from 12-U8.1 -> 13.0-U2 this time around.
 
J

Juan Manuel Palacios

Contributor
Joined
May 29, 2017
Messages
146
Could it be related to VNC settings? The bhyve args for VNC changed between FreeBSD 12 & FreeBSD 13, and when first upgrading to TrueNAS 13 I did have to adjust my scripts to pass them correctly, because otherwise I wouldn't have VNC into my pfSense VM. I did start successfully, though, but without VNC, and it wouldn't surprise me to see the TrueNAS middleware throwing an exception because of that.
 
cb88

cb88

Dabbler
Joined
May 11, 2022
Messages
24
Juan Manuel Palacios said:
Could it be related to VNC settings? The bhyve args for VNC changed between FreeBSD 12 & FreeBSD 13, and when first upgrading to TrueNAS 13 I did have to adjust my scripts to pass them correctly, because otherwise I wouldn't have VNC into my pfSense VM. I did start successfully, though, but without VNC, and it wouldn't surprise me to see the TrueNAS middleware throwing an exception because of that.
Click to expand...
No they fixed that bug, and it was preventing PiHole from starting.

If you look at my original comment here, you can see that the middleware is falling flat on it's face for some reason.
 
J

Juan Manuel Palacios

Contributor
Joined
May 29, 2017
Messages
146
Aha, this is what I was referring to:

[NAS-114595] - iXsystems TrueNAS Jira

ixsystems.atlassian.net ixsystems.atlassian.net
github.com

NAS-114595 / 13.0 / Fix vnc args in 13 by sonicaj · Pull Request #8964 · truenas/middleware

This commit brings change to middleware to provide new/updated args to bhyve for vnc as this changed when switching from 12->13.
github.com github.com
github.com

NAS-114595 / 13.0 / Fix vnc args in 13 by sonicaj · Pull Request #8964 · truenas/middleware

This commit brings change to middleware to provide new/updated args to bhyve for vnc as this changed when switching from 12->13.
github.com github.com
 
J

Juan Manuel Palacios

Contributor
Joined
May 29, 2017
Messages
146
cb88 said:
No they fixed that bug, and it was preventing PiHole from starting.

If you look at my original comment here, you can see that the middleware is falling flat on it's face for some reason.
Click to expand...
OK, it was worth a shot ;) Where does following that Python stack trace take you?
 
J

Juan Manuel Palacios

Contributor
Joined
May 29, 2017
Messages
146
cb88 said:
No idea... not exactly my cup of tea. I'm more of an embedded controller engineer.
Click to expand...
Well, I get'cha on that one, I'm a coder, but I absolutely despise Python with a burning passion! So the more I can avoid it, the better, it gives me heartburn!
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Please report the output of midclt call vm.query | jq. The error indicates a name mismatch.
 
cb88

cb88

Dabbler
Joined
May 11, 2022
Messages
24
I've written a little pythong (and a moderate amount of Lisp ) but... like I said. Nothing serious so I would just be stumbling through it to figure out whatever it is. It appears to be unable to find the VM for some reason though on 13... so perhaps not too difficult to figure out for someone familiar with that.

@Samuel Tai
Code:
FreeBSD 13.1-RELEASE-p1 n245406-814eb095751 TRUENAS

<Snip>

root@truenas[~]# vm query
root@truenas[~]# midclt call vm.query | jq
[
  {
    "id": 1,
    "name": "DebianPiHole",
    "description": "Pi Hole",
    "vcpus": 1,
    "memory": 256,
    "autostart": true,
    "time": "LOCAL",
    "grubconfig": null,
    "bootloader": "UEFI",
    "cores": 1,
    "threads": 1,
    "shutdown_timeout": 60,
    "devices": [
      {
        "id": 5,
        "dtype": "NIC",
        "attributes": {
          "type": "VIRTIO",
          "mac": "00:a0:98:27:ea:16",
          "nic_attach": "lagg0"
        },
        "order": 1003,
        "vm": 1
      },
      {
        "id": 6,
        "dtype": "DISK",
        "attributes": {
          "path": "/dev/zvol/Data/Debian-w1ss58",
          "type": "VIRTIO",
          "physical_sectorsize": null,
          "logical_sectorsize": null
        },
        "order": 1001,
        "vm": 1
      },
      {
        "id": 8,
        "dtype": "VNC",
        "attributes": {
          "vnc_port": 52488,
          "wait": false,
          "vnc_resolution": "800x600",
          "vnc_bind": "0.0.0.0",
          "vnc_password": "",
          "vnc_web": true
        },
        "order": 1002,
        "vm": 1
      }
    ],
    "status": {
      "state": "RUNNING",
      "pid": 1740,
      "domain_state": "RUNNING"
    }
  },
  {
    "id": 2,
    "name": "pfSenese",
    "description": "pfSense",
    "vcpus": 1,
    "memory": 1024,
    "autostart": false,
    "time": "LOCAL",
    "grubconfig": null,
    "bootloader": "UEFI",
    "cores": 2,
    "threads": 1,
    "shutdown_timeout": 90,
    "devices": [
      {
        "id": 9,
        "dtype": "NIC",
        "attributes": {
          "type": "VIRTIO",
          "mac": "00:a0:98:5b:10:d6",
          "nic_attach": "lagg0"
        },
        "order": 1005,
        "vm": 2
      },
      {
        "id": 10,
        "dtype": "DISK",
        "attributes": {
          "path": "/dev/zvol/Data/pfSenese-xtmr",
          "type": "VIRTIO",
          "physical_sectorsize": null,
          "logical_sectorsize": null
        },
        "order": 1001,
        "vm": 2
      },
      {
        "id": 11,
        "dtype": "VNC",
        "attributes": {
          "vnc_port": 6988,
          "wait": false,
          "vnc_resolution": "800x600",
          "vnc_bind": "0.0.0.0",
          "vnc_password": "",
          "vnc_web": true
        },
        "order": 1002,
        "vm": 2
      },
      {
        "id": 13,
        "dtype": "PCI",
        "attributes": {
          "pptdev": "5/0/0"
        },
        "order": 1004,
        "vm": 2
      },
      {
        "id": 14,
        "dtype": "PCI",
        "attributes": {
          "pptdev": "6/0/0"
        },
        "order": 1003,
        "vm": 2
      }
    ],
    "status": {
      "state": "ERROR",
      "pid": null,
      "domain_state": "ERROR"
    }
  }
]
root@truenas[~]# ls /dev/zvol/Data/pfSenese-xtmr
/dev/zvol/Data/pfSenese-xtmr
root@truenas[~]#
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Are your passthrough tunables still in place?
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Did you verify the PCI addresses are still the same?
 
cb88

cb88

Dabbler
Joined
May 11, 2022
Messages
24
It would appear that they are the same.

root@truenas[~]# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.5 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.6 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:01.7 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 PCIe GPP Bridge [6:0]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus A
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Internal PCIe GPP Bridge 0 to Bus B
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Raven/Raven2 Device 24: Function 7
01:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02)
01:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
03:00.0 Non-Volatile memory controller: Intel Corporation NVMe Optane Memory Series
04:00.0 Non-Volatile memory controller: Intel Corporation NVMe Optane Memory Series
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
06:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
07:00.0 SATA controller: ASMedia Technology Inc. Device 1166 (rev 02)
08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
08:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
08:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
08:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Raven USB 3.1
08:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor
08:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
08:00.7 Non-VGA unclassified device: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/Renoir Non-Sensor Fusion Hub KMDF driver
09:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 61)
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Does the VM start if you try midclt call vm.start 2?
 
You must log in or register to reply here.

Similar threads

cb88
Bhyve VM broken after upgrade to 13.0-RELEASE
Replies
4
Views
1K
cb88
cb88
W
noVNC "Failed to connect to server" after upgrade to 13.0
Replies
2
Views
3K
wuxia
W
C
Veeam copy jobs failing after upgrade to TrueNAS 13 - Failed to decompress LZ4 block
Replies
7
Views
2K
ljt3705
L
Agent92
Kernel panic when upgrading TrueNAS 12 to 13
2
Replies
32
Views
3K
Agent92
Agent92
E
Trouble connecting to VM, it only shows black screen with TianoCore logo?
Replies
3
Views
2K
FelipeFR
FelipeFR
Top