GPU passthrough fails with third PCIE slot in use

[Wexy101]

Just moved over to Truenas Scale as host (had as VM in proxmox which worked great) and got GPU passthrough working for libreelec connected to truenas Plex app using plexkodiconnect all working very well.

The next step was to implament a PFsence VM so i installed a 4port nic adapter in my third PCIE port (GA-Z97X-UD5H-BK. fist two PCIE ports have GPUs). With that nic card in, the ibreelec wont start withe following error.

Ive reisntalled, sived through the bios, added verious items to "/etc/default/grub.d/truenas.cfg" and "/usr/local/bin/truenas-grub.py" but i cannot get group one to split up into their own individual groups:

Any help would be appriciated.

 

[Wexy101]

Update: after replacing the nic card and trying again it failed.

But, when i add both nic card ports (with the GT710 which is what i'm trying to get working) via "PCI Passthru Device" the VM boots! if I use the fist gpu (gtx580) instead of the GT710, the VM boots. but if i remove even one nic port, it failes with above error.

So i can add remove the GPUs but I always have to have the NIC cards present. Their all in the same iommu group (1). So how can i get it that i can remove the NIC card like i can remove the GPUs? ( I just tested, VM boots with only the NIC ports pased through)

There must be a setting in a file somewhere that lets the NIC card act the same as the two GPUs within iommu group 1.

 

[Wexy101]

If I add the NICs to a second VM, that VM, VM2, boots fine but i get the following error when I start the first VM that has the GPU, Because their in the same iommu group its possible to get this working as I can add the first GPU with VM2 and it boots fine.

The clarify the objective boot GPUs and NICs on different VMs

 

[morganL]

If I add the NICs to a second VM, that VM, VM2, boots fine but i get the following error when I start the first VM that has the GPU, Because their in the same iommu group its possible to get this working as I can add the first GPU with VM2 and it boots fine.

The clarify the objective boot GPUs and NICs on different VMs

View attachment 57136

Interesting problem... I'd suggest turning into 1 simple and easily reproducible bug and then report it.
If you are prepared to work with engineering they may be able to help.

Can you also confirm that the issue occurs with a simpler VM than PFSense. PFSense may have some unusual requirements that are not KVM friendly.

You should report your hardware and BIOS and PCIe card types.. just in case this is platform specific.

 

[Wexy101]

So basically

3 PCIe ports
2 GPUs 1 Nic card
their in the same iommu group
i can use the GPUs seperatly
But the NIC card cant be used on its own, needs either GPU with it!!!

 

[Wexy101]

Update:

So with both a GPU (Fisrt or second) and the NIC ports passed through, i can boot a VM (nothing intalled).

when i enter the following that allow the GRUB additions to not be wiped after reboot, I can boot a VM with either GPU without having to add the NIC ports.
midclt call system.advanced.update '{"kernel_extra_options": "firstoption=value secondoption=value thirdoption=value"}'
but, then, when I add the NIC ports to a second VM it says the GPU is in use by a different domain (VM)

The saga continues!