I can't speak to that ticket specifically, but reviewing the linked thread it might be a similar issue if there's still artifacts left over from a previous passthrough setup, and/or the middleware isn't in sync with what the on-disk modprobe configuration is.
Do you have a file existing at /etc/modprobe.d/vfio.conf and if so what are the contents?
@Sparx if you're open to a DM I can see if I have availability to troubleshoot this real-time with you.
For @Sparx it seems like the middleware and the kernel weren't on the same page as far as the vfio-pci usage.
Big thanks to @Sparx for swinging the mallet as we played a game of Whack-A-Mole to sort this out!
Important Edit - These instructions were written for resetting GPU/VFIO claims in TrueNAS SCALE 22.12 "Bluefin" and may not longer be valid for future versions, as GPU mapping may have changed.
First off, check to see if your device is claimed by the vfio-pci driver by investigating the "Kernel driver in use" part of the lspci -v output:
Code:
0b:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
DeviceName: pciPassthru0
Subsystem: NVIDIA Corporation GP104GL [Tesla P4]
Physical Slot: 192
Flags: bus master, fast devsel, latency 248, IRQ 19
Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at e4000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: vfio-pci ### this line indicates the passthrough driver has claimed your GPU
Kernel modules: nouveau, nvidia_current_drm, nvidia_current
Code:
0b:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
DeviceName: pciPassthru0
Subsystem: NVIDIA Corporation GP104GL [Tesla P4]
Physical Slot: 192
Flags: bus master, fast devsel, latency 248, IRQ 19
Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at e4000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_current_drm, nvidia_current
If you have the vfio-pci driver in use, but you have no isolated GPU (and have rebooted) then you probably have a stuck file somewhere.
Warning - This doesn't preserve any existing passthroughs, and resets all GPUs to host-owned. No VMs were running during this. Fixing this requires you to go mucking with kernel options, manually deleting files, and multiple reboots. Have backups, and obligatory warning:
Ensure you've removed all isolated GPUs from the webUI; reboot the system if any changes were necessary.
Open a root shell (SSH, use sudo -s or prepend commands with sudo to elevate if needed) on your TrueNAS SCALE machine.
Back up the contents of the following files (save them to your pool, copy them as a local text file, wherever you'd like):
These files will all likely contain the PCI vendor and device IDs of a passthrough GPU, eg: 10DE:1BB3 as well as references to vfio. Once you've backed up the content, remove these files.
Run update-initramfs -k all -u - a number of grep errors will likely be logged. Reboot once more, and you should have your nvidia-smi functionality back.
Note that this isn't a fix for the "two identical GPUs in a system" issue - that one's a bit more complex - nor will it do anything to enable a pre-Maxwell NVIDIA GPU in Bluefin.
For @Sparx it seems like the middleware and the kernel weren't on the same page as far as the vfio-pci usage.
Big thanks to @Sparx for swinging the mallet as we played a game of Whack-A-Mole to sort this out!
First off, check to see if your device is claimed by the vfio-pci driver by investigating the "Kernel driver in use" part of the lspci -v output:
Code:
0b:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
DeviceName: pciPassthru0
Subsystem: NVIDIA Corporation GP104GL [Tesla P4]
Physical Slot: 192
Flags: bus master, fast devsel, latency 248, IRQ 19
Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at e4000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_current_drm, nvidia_current
Code:
0b:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
DeviceName: pciPassthru0
Subsystem: NVIDIA Corporation GP104GL [Tesla P4]
Physical Slot: 192
Flags: bus master, fast devsel, latency 248, IRQ 19
Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at e4000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_current_drm, nvidia_current
If you have the vfio-pci driver in use, but you have no isolated GPU (and have rebooted) then you probably have a stuck file somewhere.
Warning - This doesn't preserve any existing passthroughs, and resets all GPUs to host-owned. No VMs were running during this. Fixing this requires you to go mucking with kernel options, manually deleting files, and multiple reboots. Have backups, and obligatory warning:
Ensure you've removed all isolated GPUs from the webUI; reboot the system if any changes were necessary.
Open a root shell (SSH) on your TrueNAS SCALE machine.
Back up the contents of the following files (save them to your pool, copy them as a local text file, wherever you'd like):
These files will all likely contain the PCI vendor and device IDs of a passthrough GPU, eg: 10DE:1BB3 as well as references to vfio. Once you've backed up the content, remove these files.
Run update-initramfs -k all -u - a number of grep errors will likely be logged. Reboot once more, and you should have your nvidia-smi functionality back.
Note that this isn't a fix for the "two identical GPUs in a system" issue - that one's a bit more complex - nor will it do anything to enable a pre-Maxwell NVIDIA GPU in Bluefin.
For @Sparx it seems like the middleware and the kernel weren't on the same page as far as the vfio-pci usage.
Big thanks to @Sparx for swinging the mallet as we played a game of Whack-A-Mole to sort this out!
First off, check to see if your device is claimed by the vfio-pci driver by investigating the "Kernel driver in use" part of the lspci -v output:
Code:
0b:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
DeviceName: pciPassthru0
Subsystem: NVIDIA Corporation GP104GL [Tesla P4]
Physical Slot: 192
Flags: bus master, fast devsel, latency 248, IRQ 19
Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at e4000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_current_drm, nvidia_current
Code:
0b:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
DeviceName: pciPassthru0
Subsystem: NVIDIA Corporation GP104GL [Tesla P4]
Physical Slot: 192
Flags: bus master, fast devsel, latency 248, IRQ 19
Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at e4000000 (64-bit, prefetchable) [size=32M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_current_drm, nvidia_current
If you have the vfio-pci driver in use, but you have no isolated GPU (and have rebooted) then you probably have a stuck file somewhere.
Warning - This doesn't preserve any existing passthroughs, and resets all GPUs to host-owned. No VMs were running during this. Fixing this requires you to go mucking with kernel options, manually deleting files, and multiple reboots. Have backups, and obligatory warning:
Ensure you've removed all isolated GPUs from the webUI; reboot the system if any changes were necessary.
Open a root shell (SSH) on your TrueNAS SCALE machine.
Back up the contents of the following files (save them to your pool, copy them as a local text file, wherever you'd like):
These files will all likely contain the PCI vendor and device IDs of a passthrough GPU, eg: 10DE:1BB3 as well as references to vfio. Once you've backed up the content, remove these files.
Run update-initramfs -k all -u - a number of grep errors will likely be logged. Reboot once more, and you should have your nvidia-smi functionality back.
Note that this isn't a fix for the "two identical GPUs in a system" issue - that one's a bit more complex - nor will it do anything to enable a pre-Maxwell NVIDIA GPU in Bluefin.
Thanks a lot @HoneyBadger
I have "Kernel driver in use: vfio-pci" and no Isolated GPU Devices, and have done several reboots after I have removed the Isolated GPU.
If things goes sideways after removing the files and the last reboot, how do I revert the process? Is it that simple that I still will have access to login in to TrueNAS and copy the files back to it's location and run update-initramfs -k all -u and do another reboot and I am back where a started?
I am fearly new to TrueNAS and Linux, so I just want to be sure that I have a way back to my data, if everything goes bad :)
No. I just have never (Angelfish through current) been able to load a driver...thus nvidia-smi returns nothing.
Code:
nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
@mgoulet65 .. so thats probably different. Since my setup was working in angelfish. Whats your GPU?
Is it the K20c stated in your specs? That doesnt work with drivers newer than 460.106 on linux(?)
@mgoulet65 .. so thats probably different. Since my setup was working in angelfish. Whats your GPU?
Is it the K20c stated in your specs? That doesnt work with drivers newer than 460.106 on linux(?)
If things goes sideways after removing the files and the last reboot, how do I revert the process? Is it that simple that I still will have access to login in to TrueNAS and copy the files back to it's location and run update-initramfs -k all -u and do another reboot and I am back where a started?
I am fearly new to TrueNAS and Linux, so I just want to be sure that I have a way back to my data, if everything goes bad :)
The files being changed are only used in relation to GPU passthrough, and only change the modules loaded in the kernel at boot-time. Even if the boot device was completely unusable, your TrueNAS data disks could be imported to a fresh install.
I will suggest that you take a configuration backup through the UI, especially if you are using pool or dataset encryption.
What's the desired use case for the GPU at home? The K20c was a pretty beefy card back in its day (around a GTX780 if I recall correctly) but if you just need to transcode a bit of video, then you won't need that sort of horsepower. Depending on number of concurrent streams something as little as a Quadro P400 could be enough.
The files being changed are only used in relation to GPU passthrough, and only change the modules loaded in the kernel at boot-time. Even if the boot device was completely unusable, your TrueNAS data disks could be imported to a fresh install.
I will suggest that you take a configuration backup through the UI, especially if you are using pool or dataset encryption.
What's the desired use case for the GPU at home? The K20c was a pretty beefy card back in its day (around a GTX780 if I recall correctly) but if you just need to transcode a bit of video, then you won't need that sort of horsepower.
Yes it it trying to add HW transcode to Plex and/or Jellyfin (once I decide how I want to go). Is there a diagnostic path I should be on for the K20c or should I change out to some known working GPU? If the latter, which?
Yes it it trying to add HW transcode to Plex and/or Jellyfin (once I decide how I want to go). Is there a diagnostic path I should be on for the K20c or should I change out to some known working GPU? If the latter, which?
The NVIDIA driver was changed between Angelfish (470.103.01) and Bluefin (515.65.01) which removed support from the Kepler-based GPUs, including your K20c.
I've seen the following page referenced - it allows you to select source and destination resolutions for transcoding, as well as filtering out GPUs by generation. A Quadro P400 will work fine for a home server that's transcoding one or two streams, but if you have a heavier workload and/or are using 4K source material you may need something stronger. Note that NVIDIA has introduced a limit of 3 concurrent NVENC streams on consumer and entry-level prosumer cards as well - if you're going beyond that, look for a card with unlimited stream support.
can you, or are there plans to, allow for some kind of driver version selection by the user? sure it would require a reboot to change the driver. but better than forcing a user to choose between major TN release versions just for a different nvidia driver.
i would recommend 3 versions to choose from. ~340 or 390 for ancient legacy cards, ~470 for middle aged cards, and latest driver version (currently 525 branch) to support the newest cards and CUDA features. that should cover pretty much everyone, and major TN software releases could only focus on updating whatever the latest stable version is.
can you, or are there plans to, allow for some kind of driver version selection by the user? sure it would require a reboot to change the driver. but better than forcing a user to choose between major TN release versions just for a different nvidia driver.
i would recommend 3 versions to choose from. ~340 or 390 for ancient legacy cards, ~470 for middle aged cards, and latest driver version (currently 525 branch) to support the newest cards and CUDA features. that should cover pretty much everyone, and major TN software releases could only focus on updating whatever the latest stable version is.
No plans unfortunately... AFAIK, we would would have to build three separate images. There would need to be separate software trains to allow easy updates. That's a lot of extra cost and complexity.
We are following your advice with the latest TrueNAS Release (22.12) only focussing on the latest stable version. Nvidia has the power to support the older GPUs if they wish (but they are probably happy to force an update).
You can still pass-thru old GPUs to VMs.
Otherwise, its time to update the GPU (even if its an Ebay unit that is 5 years old).
The files being changed are only used in relation to GPU passthrough, and only change the modules loaded in the kernel at boot-time. Even if the boot device was completely unusable, your TrueNAS data disks could be imported to a fresh install.
I will suggest that you take a configuration backup through the UI, especially if you are using pool or dataset encryption.
What's the desired use case for the GPU at home? The K20c was a pretty beefy card back in its day (around a GTX780 if I recall correctly) but if you just need to transcode a bit of video, then you won't need that sort of horsepower. Depending on number of concurrent streams something as little as a Quadro P400 could be enough.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
