Guide: NVidia Quadro P400 GPU Passthrough to VM and Docker for Jellyfin Transcoding

[craftycraft]

I recently obtained a new box for my TrueNAS CORE server with hardware that supports PCI-E passthrough. I then spent hours troubleshooting the various barriers to successfully utilizing a NVidia Quadro P400 graphics card for Jellyfin transcoding via Docker running in an Ubuntu VM. I've put together the working solution here in the hope that it may be helpful to others in the future. The end result is that I have TrueNAS Core running on bare metal with Ubuntu Server inside a VM which hosts a Jellyfin Docker container able to offload transcoding tasks to the Quadro P400. It works beautifully transcoding 1080p video - GPU utilization peaks at about 95% while the CPU utilization remains under 3%. Mods - please feel free to relocate this thread if another forum would be more suitable.

For reference, my hardware is as follows:

• Dell Precision T7910
• 1x Intel Xeon E5-2667 v3
• 128GB DDR4 ECC RAM
• NVidia Quadro P400
• 4x WD RED 10TB HDDs for spinning storage
• 2x Corsair 2TB NVME SSDs for high-speed VM storage
• 2x Silicon Power 128GB NVME SSDs for boot drive
• TrueNAS Core 13.0 U2

Many of my headaches were caused by incompatibilities between the particular version of Ubuntu Server I was initially running and the version of the NVidia drivers which would play nice with a passed-through card. The steps and suggestions offered here worked well for me - YMMV.

1. Set up your PCI-E passthrough by finding the correct IDs via the pciconf -lv command in the shell and then setting the required tunables. There are many threads covering this topic - the one I referenced is here.
2. Once I set the required tunables and rebooted my system, it would continually hang at startup. I eventually figured out that I needed to reinstall TrueNAS and select the legacy BIOS boot option instead of UEFI - not sure why this is the case, but it worked for me. If you happen to get stuck after rebooting, you can select boot options when the ASCII TrueNAS logo appears and enable safe mode to get back into your system. In my case, I passed through both the Quadro P400 GPU and audio devices.
3. After successfully passing through the devices, go ahead and create your VM within TrueNAS. It wasn't possible to spin up a VM with the latest version of Ubuntu Server since that version isn't compatible with the required driver version - the solution is to install Ubuntu Server 20.04 LTS. Complete the Ubuntu installation as usual.
4. Once your Ubuntu VM is up and running, head to the shell and CD into /etc/modprobe.d/
   cd /etc/modprobe.d/
5. Create a new file:
   sudo nano blacklist-nvidia-nouveau.conf
6. Add the following to the new file then save/close:
   blacklist nouveau
options nouveau modeset=0
7. Create another new file:
   sudo nano nvidia.conf
8. Add the following to the new file then save/close:
   options nvidia NVreg_OpenRmEnableUnsupportedGpus=1
9. Update kernel init ram fs:
   sudo update-initramfs -u
10. Reboot the system.
11. Install "gcc"
    sudo apt update && sudo apt install gcc -y
12. Install "make"
    sudo apt install make -y
13. Download version 470.57.02 of the NVidia x64 video driver for Linux - the version # is particularly important here:
    wget https://us.download.nvidia.com/XFree86/Linux-x86_64/470.57.02/NVIDIA-Linux-x86_64-470.57.02.run
14. Run the NVidia driver installer:
    sudo bash NVIDIA-Linux-x86_64-470.57.02.run
15. At this point, you should be able to verify that the correct driver has been installed and that the GPU has been successfully passed through by running the following command - if all went well, you should see a table with the current utilization stats for the Quadro P400:
    nvidia-smi
16. Install Docker as usual - there are many guides for this process online, so I won't outline them here. Be sure to ensure that your Docker installation is working as expected before moving on to the next steps.
17. Set up the NVidia Container Toolkit - the required commands are included below and you can reference this page from NVidia for more details if you run into trouble:
    Code:

    distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
        && curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add - \
        && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

    
    
    Code:

    sudo apt-get update \
        && sudo apt-get install -y nvidia-container-toolkit

18. Test that the Quadro P400 can be passed through to a Docker container by running the following command - if successful, you'll see the table displayed with the current utilization stats for the GPU:
    
    Code:

    sudo ctr image pull docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04

    
    
    Code:

    sudo ctr run --rm -t \
        --runc-binary=/usr/bin/nvidia-container-runtime \
        --env NVIDIA_VISIBLE_DEVICES=all \
        docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04 \
        cuda-11.0.3-base-ubuntu20.04 nvidia-smi

19. Finally, setup the Jellyfin container with the directions on their site for hardware accelerated transcoding - the key is to set the --gpus all option in the Docker run command.
    
    Code:

    docker run -d \ --name=jellyfin \
     --gpus all \
     -p 8096:8096 \
     -p 8920:8920 \
     -v /config:/config \
     -v /media:/media \
     -v /cache:/cache \
     --restart unless-stopped \
     jellyfin/jellyfin

20. To verify that Jellyfin can indeed access the GPU, run the following command from your Ubuntu Server shell - if successful, then you'll again see the table with the current utilization stats:
    docker exec -it jellyfin nvidia-smi
21. You'll then go into the Jellyfin administration dashboard in the web UI to enable hardware transcoding. Play a video from Jellyfin which requires transcoding. You can run the nvidia-smi from the Ubuntu Server shell while the video is playing to confirm that the GPU is being utilized for transcoding.

That's it - enjoy! I hope these steps are of use to others. Cheers.

 

[raidflex]

Is it possible to pass through to a Windows 10/11 VM as well?

 

[elforesto]

Just tried this with a Quadro P2000 on a Quanta D51B-2U server. I had to pass both Nvidia devices to the VM (display/VGA and multimedia/HDA), then the card was recognized in an Ubuntu 20.04 LTS VM and available to Plex. I used the latest driver without an issue (535.104.05 as of this writing) and verified it was transcoding 4KHDR to 1080p. Thanks for doing all of this legwork!