VM won't start anymore after adding a SAS HBA PCI passthrough device

[eldxmgw]

I was happy to see that TN Scale was listing my SAS HBA, an HP SAS9207-8e H3-25280-01c which i want to use to connect my HP Tape Library.

So i reconfigured a VM with a PCI passthrough device.

The VM won't start anymore. I tried to reconfigure the device order.
My Problem is, i don't really understand the error message:

In detail:

Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 172, in start if self.domain.create() < 0: File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create raise libvirtError('virDomainCreate() failed') libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2024-01-20T23:15:38.141317Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9] 2024-01-20T23:15:38.141321Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7] 2024-01-20T23:15:38.141805Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-register [bit 8] 2024-01-20T23:15:38.141810Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9] 2024-01-20T23:15:38.141814Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7] 2024-01-20T23:15:39.132668Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio 0000:03:00.0: group 1 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/main.py", line 184, in call_method result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1317, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1247, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 46, in start await self.middleware.run_in_thread(self._start, vm['name']) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1234, in run_in_thread return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1231, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 68, in _start self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name)) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 181, in start raise CallError('\n'.join(errors)) middlewared.service_exception.CallError: [EFAULT] internal error: qemu unexpectedly closed the monitor: 2024-01-20T23:15:38.141317Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9] 2024-01-20T23:15:38.141321Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7] 2024-01-20T23:15:38.141805Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-register [bit 8] 2024-01-20T23:15:38.141810Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9] 2024-01-20T23:15:38.141814Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7] 2024-01-20T23:15:39.132668Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio 0000:03:00.0: group 1 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Could somebody please help me in what to do to get this functional?
Thank you.

 

[morganL]

I was happy to see that TN Scale was listing my SAS HBA, an HP SAS9207-8e H3-25280-01c which i want to use to connect my HP Tape Library.

So i reconfigured a VM with a PCI passthrough device.

View attachment 74891

The VM won't start anymore. I tried to reconfigure the device order.
My Problem is, i don't really understand the error message:

View attachment 74892​

In detail:

Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 172, in start if self.domain.create() < 0: File "/usr/lib/python3/dist-packages/libvirt.py", line 1353, in create raise libvirtError('virDomainCreate() failed') libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2024-01-20T23:15:38.141317Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9] 2024-01-20T23:15:38.141321Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7] 2024-01-20T23:15:38.141805Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-register [bit 8] 2024-01-20T23:15:38.141810Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9] 2024-01-20T23:15:38.141814Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7] 2024-01-20T23:15:39.132668Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio 0000:03:00.0: group 1 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/main.py", line 184, in call_method result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1317, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1247, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_lifecycle.py", line 46, in start await self.middleware.run_in_thread(self._start, vm['name']) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1234, in run_in_thread return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1231, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/vm_supervisor.py", line 68, in _start self.vms[vm_name].start(vm_data=self._vm_from_name(vm_name)) File "/usr/lib/python3/dist-packages/middlewared/plugins/vm/supervisor/supervisor.py", line 181, in start raise CallError('\n'.join(errors)) middlewared.service_exception.CallError: [EFAULT] internal error: qemu unexpectedly closed the monitor: 2024-01-20T23:15:38.141317Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9] 2024-01-20T23:15:38.141321Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7] 2024-01-20T23:15:38.141805Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-register [bit 8] 2024-01-20T23:15:38.141810Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9] 2024-01-20T23:15:38.141814Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7] 2024-01-20T23:15:39.132668Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio 0000:03:00.0: group 1 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Could somebody please help me in what to do to get this functional?
Thank you.

Suggest you follow the forum rules.... document all the hardware and clarify that everything else in the hardware is working.
I assume VMs without the pass-thru device work?
Which version of SCALE is being used?

After that, you can report a bug... or we might suggest testing with Dragonfish.

 

[eldxmgw]

Suggest you follow the forum rules.... document all the hardware and clarify that everything else in the hardware is working.
I assume VMs without the pass-thru device work?
Which version of SCALE is being used?

After that, you can report a bug... or we might suggest testing with Dragonfish.

The MLB is an Intel S1200BTL with an E3-1270 Xeon, 32GB ECC RAM, a Fujitsu D2755 Dual Port 10G SFP+, a 6 port SATA controller card with compatible Chip, a HPE H221 SAS HBA (also known as LSI 9207-8e) in IT Mode, 6x 12TB Seagate Ironwolf HDDs, 2x Samsung 500GB 860 Evo SSDs, 650W PSU.

HDDs are unified in a z2, and both SSDs in a mirror pool.
My release train is Bluefin, latest available (stable) release.
All is working fine, also the VM, except when i add a passthrough device like mentioned before.

I already tested a fresh installed Cobia (latest stable), the behavior is the same.
I was told that TN Core and its type 2 Hypervisor Bhyve in conjunction with the kernel implementation never had this issues.
It seems that it has been forgotten since to the same with TN Scale.
Replacing Scale with Core isn't an option for me.
I own two almost identical TN Scale systems that only differ by their MLB and the quantity of SSDs. This #2 server replicates with server #1.
But the SAS HBA only exist as an interconnect for the LTO tape library in server #2.

The SAS HBA serves as a connector for a tape library that is to be managed on a W2k19 VM including VBR.
To do this, the SAS HBA must be looped through via passthrough.

This is a Scale problem that has been known for years. The community forum backlog is full of this malfunction.
Over the past years tickets has also been applied by some users to the triage team but the issues has been closed mostly without comment.
The topic is usually discussed around passthrough GPUs, but as you can see here, it also affects other PCIe passthrough devices.
As far as I was able to research, this is due to a missing kernel implementation that allows an "isolate mode" of the hardware for the passthrough.
The VM works perfectly. Only as soon as a passthrough PCIe device is added does the Scale type 2 Hypervisor (QEMU as far as I know) attest to the error shown.
In TN Scale there is a widget that offers a GPU control option.
However, it doesn't recognize anything other than GPUs which is almost useless in my or other cases.
Because it fails when the VM starts, the underlying OS layer is completely irrelevant here. Because it doesn't even get that far.

It would be a nice move to fix this issue ASAP because this already lasts so long.
And i would be finally able to get use of my tape library as it was ment to be.

 

[morganL]

The MLB is an Intel S1200BTL with an E3-1270 Xeon, 32GB ECC RAM, a Fujitsu D2755 Dual Port 10G SFP+, a 6 port SATA controller card with compatible Chip, a HPE H221 SAS HBA (also known as LSI 9207-8e) in IT Mode, 6x 12TB Seagate Ironwolf HDDs, 2x Samsung 500GB 860 Evo SSDs, 650W PSU.

HDDs are unified in a z2, and both SSDs in a mirror pool.
My release train is Bluefin, latest available (stable) release.
All is working fine, also the VM, except when i add a passthrough device like mentioned before.

I already tested a fresh installed Cobia (latest stable), the behavior is the same.
I was told that TN Core and its type 2 Hypervisor Bhyve in conjunction with the kernel implementation never had this issues.
It seems that it has been forgotten since to the same with TN Scale.
Replacing Scale with Core isn't an option for me.
I own two almost identical TN Scale systems that only differ by their MLB and the quantity of SSDs. This #2 server replicates with server #1.
But the SAS HBA only exist as an interconnect for the LTO tape library in server #2.

The SAS HBA serves as a connector for a tape library that is to be managed on a W2k19 VM including VBR.
To do this, the SAS HBA must be looped through via passthrough.

This is a Scale problem that has been known for years. The community forum backlog is full of this malfunction.
Over the past years tickets has also been applied by some users to the triage team but the issues has been closed mostly without comment.
The topic is usually discussed around passthrough GPUs, but as you can see here, it also affects other PCIe passthrough devices.
As far as I was able to research, this is due to a missing kernel implementation that allows an "isolate mode" of the hardware for the passthrough.
The VM works perfectly. Only as soon as a passthrough PCIe device is added does the Scale type 2 Hypervisor (QEMU as far as I know) attest to the error shown.
In TN Scale there is a widget that offers a GPU control option.
However, it doesn't recognize anything other than GPUs which is almost useless in my or other cases.
Because it fails when the VM starts, the underlying OS layer is completely irrelevant here. Because it doesn't even get that far.

It would be a nice move to fix this issue ASAP because this already lasts so long.
And i would be finally able to get use of my tape library as it was ment to be.

GPUs and SAS HBAs are different

They share one thing in common - the TrueNAS host typically uses them to provide services.

Feel free to report it as a bug...

 

[eldxmgw]

GPUs and SAS HBAs are different

They share one thing in common - the TrueNAS host typically uses them to provide services.

Feel free to report it as a bug...

Done.

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net

 

[eldxmgw]

@morganL it's useless!

This Caleb didn't even read the bug ticket fully.
He advised to do something what i already stated i already did. And lastly just closed the bug ticket with obvious desinterest.
This is a well known behavior i already found in almost all cases when sombeody firstly opened a thread in the community forum, then being advised to open a bug ticket, and then this ticket is being closed faster than it was opened with force.
Just search the backlog of the past years. The forum is full of that.
When people like him just hitting the close button because somebody is showing a bug that is well known, nobody will fix that missing kernel implementation in a decade because nobody will never know, or better said, nobody wants to know.

 

[eldxmgw]

This is the current status:

I installed TN Core on the same system
With this explanation I got the SAS controller passed through on a Windows 2019 server: https://wiki.freebsd.org/bhyve/pci_passthru
In short... since TN Core and its Type 2 Hypervisor (Bhyve) do not list PCIe passthrough devices like TN Scale, two tunables are important to loop through the desired PCIe card to the VM:
vmm_load
pptdevs
This is of course followed by a restart.
It is then possible to add the SAS card to the VM as a PCIe passthrough device and start the VM smoothly. The card is recognized as a storage controller in the device manager without any problems.
From this point of view, instead of the Windows driver, the manufacturer's driver and then VBR can be installed and the tape library can be put into operation.
There is a short video about the tunables here that will help: https://www.youtube.com/watch?v=i0MAZBX7P-U
The only problems I found, if you can even call it that, was that after a restart, the VM configured in this way could no longer be started.
I suspect that it has something to do with the fact that the isolated PCIe card then flies out and the hypervisor can no longer manage it somehow.
Workaround is to either restart the entire TN Core system (bad) or remove the PCIe passthrough device from the VM config.
The bottom line is that I see it as an acceptable disadvantage. Because if the VM is provided with autostart, it starts automatically with boot of the TN Core. For me there is no reason to restart the server VM dozens of times after installation.
Even if the TN system only boots up when necessary, there is no reason to do so.
Normally the VM starts and ends its service when the TN system is started up or shut down.
Only after a patch day when cumulative updates are installed, otherwise not.
I haven't found another solution yet.

Today I tried somehow to bridge the gap to TN Scale using this knowledge.
TN Scale uses a different Type 2 hypervisor with Qemu, but that shouldn't be the issue.
A good supporting document (including the topic of bypassing IOMMU groups) is here: https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
The IOMMU groups can be displayed wonderfully with the following:
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
echo "IOMMU Group ${g##*/}:"
for d in $g/devices/*; do
echo -e "\t$(lspci -nns ${d##*/})"
done;
done;

Unfortunately for me the SAS card is in a group where other devices are listed that are not allowed to be provided with a passthrough:
IOMMU Group 1:
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0159] (rev 09)
01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
01:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
03:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 01)

Since, as described, an IOMMU group bypass (ACS patch) is not provided on the kernel side, and replugging the PCIe card on the MLB is of course not mechanically possible, the topic of TN scale and passthrough is at an end at this point due to the lack of kernel implementations.

Ironically, one can ask the question why TN Core is able to do this with its Type 2 hypervisor, but TN Scale cannot, despite years of advice? :)

The issue, however, is and remains the problem that iX has ignored for years and, on the one hand, its unwillingness to add a missing kernel implementation in order to isolate PCIe devices and bypass IOMMU groups (ACS patch).
There is probably also a living two-class society where community users are used as cheap beta testers for the paying enterprise clientele, and are only brought in when they need features or bug fixes that serve them.
The previously described phenomenon of having unqualified tickets immediately closed by iX staff on the subject has been proven to be explainable for years.
Even though this might not be company policy, the behavior damages public trust through the listless proletariat they represent at that very moment.
In the end, the community users are knowingly left alone, as they have been for years.