Single VM w/ nvidia GPU passthrough and USB3 PCI passthrough

E2zin

Dabbler
Joined
Mar 25, 2021
Messages
16
Hi everyone, I have a workstation machine that houses all my drives. It used to be my NAS but now that I upgraded the motherboard, CPU and RAM, and moved to a smaller office, I need to be able to make that machine work more for me, with less hassle. TrueNas SCALE was a godsent when I heard about it, but I have not been able, so far, to make it work for my (very simple) use case.

I would like to run a VM with GPU passthrough (there are 2 discrete GPUs in my machine, on top of the integrated graphics used by the console output). I would also like to passthrough my USB3 PCIe card to that VM. I can also definitely settle on a Ubuntu VM (rather than dealing with the complications of Nvidia on Windows VM). I can't seem to figure out what is wrong in my configuration and there is simply not much options that should go wrong but still I was not able to get the VM to properly start with the monitor plugged to my GPU displaying the ubuntu install screen.

On some other tries, I had the VNC display work properly, installed Ubuntu but the GPU would never be detected or displaying anything. Finally on most tries, I just end up with the web interface freezing, then becoming unavailable, and the whole server not responding at all (even when plugging in the keyboard to the ports from the motherboard). The only options is to force power down then restart the server.

I would really like if someone could help me figure out how to provide more information to get a better answer.

Thank you very much.
 

E2zin

Dabbler
Joined
Mar 25, 2021
Messages
16
Some more information:

TrueNAS-SCALE-22.02-RC.1
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
Is the issue that you cannot get 2 NVidia GPU/graphics cards recognized by SCALE.. Do either of them work?
 

E2zin

Dabbler
Joined
Mar 25, 2021
Messages
16
Thank you for your quick answer @morganL

Actually I do see both GPUs separately in the GPU field for the VM, so I only select the one I want and it does get added properly to the list of PCI passthrough devices (one is the GPU pci_4_0, the other one the HDMI audio pci_4_1). But I don't know if there is something else I'm missing or doing incorrectly. I unchecked "Enabled display" so the GPU would be used and did not configure any VNC/Spice display, and still nothing loading on my monitor plugged to the selected GPU.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
@E2zin I would suggest focussing your description on this post on getting that specific Graphics card working with a VM....
Maybe someone else has that model and can help you...... but solving the generic problem with 2 random GPUs and USB pass thru is more complex. One problem at a time.
 

E2zin

Dabbler
Joined
Mar 25, 2021
Messages
16
I appreciate the suggestion. Do you mean taking the other GPU and card out from the computer or just not try to configure it to the VM.
So is there anything I should be doing prior to getting the passthrough setup? Any commands I would need to run manually that I'm not aware?
Or should TrueNAS be supposed to handle all of this on it's own?

Would passing in a VNC display automatically make issues with the GPU or should it just show up as a secondary display?

Thank you.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
I appreciate the suggestion. Do you mean taking the other GPU and card out from the computer or just not try to configure it to the VM.
So is there anything I should be doing prior to getting the passthrough setup? Any commands I would need to run manually that I'm not aware?
Or should TrueNAS be supposed to handle all of this on it's own?

Would passing in a VNC display automatically make issues with the GPU or should it just show up as a secondary display?

Thank you.
Lets assume we are either trying to find a solution (from someone with knowledge) or a bug. The simpler you can cam the problem statement, the more likely you will be to find these. The problem is "VM of type X can't use GPU of type Y and drive a display." Has someone been succcessful with something similar?
 

E2zin

Dabbler
Joined
Mar 25, 2021
Messages
16
Ok, I realize what you mean. I would really like if I could be pointed in a good support direction. I understand that SCALE is still in development and as such the documentation might not be 100% complete. But when I look at this page (and it's child page), I do not feel I have all the necessary knowledge to make the best decisions for each options of a VM. And because of that lack of knowledge, I do not know if some options are incompatible between each other. I tried looking up for those options online and there's still very little useful results.

I would like to figure out what I'm doing wrong, or what is wrong in my configuration. I will be trying some more over the weekend, try to be more methodical in "what" I try, one change at a time, etc. But the issue remains that I feel a lack of knowledge to really understand what is going on. Every time I look online for some help it turns out results that are either not valid anymore (from earlier versions of SCALE) or are for other distros. I understand that TrueNAS uses a middleware to make sure all configuration is in one location. So a lot of tools that are usually available to "make it work" on other distros do not work on TrueNAS. I just wish I could find some "easier" tutorials about running a Ubuntu or Windows VM (I'll settle for the simplest!) with a GPU passthrough and a PCI USB passthrough. About the latter, it seems to be the one crashing the most often, and I don't understand how other users do it to have a keyboard and mouse in their VM if it isn't by passing a whole PCI USB controller?

Sorry for the rant, I just feel exasperated after 3 days of trying different things, being "so close" so many times and not doing what to do next to try and figure out the issue. I will leave this thread open if someone can reply with some good documentation that would help me gain more knowledge. I'm still searching for a tutorial that a nice member of the community might have written after managing to do it themselves. I would be surprised that no one has created a VM that is used with a passthrough GPU + monitor and a passthrough PCI USB card with input devices?

If it's not against the forum rules, I will open new threads when I hit more specific issues in my next, more methodical trial and error.
Thank you in advance, I'm really sorry for feeling so overwhelmed in trying to run what I feel is a "main use case" for TrueNAS SCALE for hobbyists.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
I think the 1st call is just to see if anyone has worked with that specific GPU.... we can then determine whether it should or should not work.
Unfortunately our software team doesn't have a full library of hardware options and the time to try each one. It really about working with the community.
 

E2zin

Dabbler
Joined
Mar 25, 2021
Messages
16
Thank you very much for being patient with me @morganL

I will post a new thread soon when I have specific questions.

For the community at large, did anyone manage to use an EVGA GeForce GTX 670 FTW in a GPU passthrough configuration?
Did anyone manage to passthrough a USB3 PCIe card? Did you have to "unregister it" manually or blacklist it before passing it through?
 
Top