Nvidia GPU not appearing for use with SCALE

Sasquatch · Feb 24, 2023

jhavron said:
OS Version:TrueNAS-SCALE-22.12.0
Product:PowerEdge T110 II
Model:Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
Memory:16 GiB

root@truenasscale[~]# lspci | grep VGA
01:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 630 Rev. 2] (rev a1)
05:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
root@truenasscale[~]#

Working on upgrading to TrueNAS-22.12.1 right now though.

There is a Jira bug for this bug(my doing) marked for 22.12.2 release. I didn't check if it's fixed in 22.12.1 yet( maybe by accident?) But I doubt it would.

Sasquatch · Feb 24, 2023

aramis0221 said:
Same problem here running on a Quadro 6000. Does anybody knows how to activate the nvidia legacy drivers or something related??

Nope you're out of luck, attempts to install drivers manually by apt result in TrueNas not booting, at least from what I've read. P400 is like $50 maybe you could stretch for that?

morganL · Feb 25, 2023

If the issue is evident on 22.12.1, please report a bug.

It would be useful to know if there is a Nvidia driver that work with old and new GPUs?
Or does Nvidia recommend 2 different drivers....
Any knowledge on this issue would be useful to pass along..... its not an area of deep expertise on our side.

Sparx · Feb 25, 2023

Bug reported:
Jira created

Ukjent1 · Feb 26, 2023

Hi everyone,
I have the same issue with my Scale 22.12.1.
I have a T400 GPU in my setup (motherboard is Supermicro X10DRG and CPU is Intel E5-2670v3)

I am quite new to Truenas and have basic level experience with Linux:)
I thought I should start to use the Plex app (official version, for now :) ) in Scale for my media. But I can't add the GPU in the installation.

This is what I have tested/done:
For some reason I had added the GPU to a VM I am running (dont' need it there), and thought it was because of that. I removed the GPU from Isolated GPU devices, removed the PCI devices under devices in the VM and unselected the GPU from the VM, and restarted the Truenas server - It didn't help.

If I edit setting in a VM, I can add the GPU, but it is named NVIDIA Corporation Device 1fb2 (I don't won't to use it in a VM)

If I run lspci -v | grep NVIDIA, this is what I get:
root@homeserv[~]# lspci -v | grep NVIDIA
04:00.0 VGA compatible controller: NVIDIA Corporation Device 1fb2 (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 1489
04:00.1 Audio device: NVIDIA Corporation Device 10fa (rev a1)
Subsystem: NVIDIA Corporation Device 1489

If I run nvidia-smi, this is what I get:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

I found something about running this command:
systemctl status systemd-modules-load.service

And it shows this:
● systemd-modules-load.service - Load Kernel Modules
Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static)
Drop-In: /etc/systemd/system/systemd-modules-load.service.d
└─override.conf
Active: active (exited) since Fri 2023-02-24 15:16:46 CET; 1 day 19h ago
Docs: man:systemd-modules-load.service(8)
man:modules-load.d(5)
Main PID: 4242 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 154037)
Memory: 0B
CPU: 0
CGroup: /system.slice/systemd-modules-load.service

Feb 24 15:16:46 homeserv systemd[1]: Starting Load Kernel Modules...
Feb 24 15:16:46 homeserv systemd-modules-load[4242]: Failed to find module 'vfio_pci ids=10DE:1FB2,10DE:10FA'
Feb 24 15:16:46 homeserv systemd-modules-load[4242]: Failed to find module 'nvidia-drm'
Feb 24 15:16:46 homeserv systemd-modules-load[4242]: Inserted module 'ntb_split'
Feb 24 15:16:46 homeserv systemd-modules-load[4242]: Inserted module 'ntb_netdev'
Feb 24 15:16:46 homeserv systemd[1]: Finished Load Kernel Modules.

cat /etc/modprobe.d/nvidia.conf

root@homeserv[~]# cat /etc/modprobe.d/nvidia.conf
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidia* pre: vfio-pci

I found a thread about disabling CSM and enable Above 4G decoding, which I have tested with no luck.
Also in the same thread, it said to enalbe Re-Size BAR support, but this setting I can't find in my BIOS.

Does anyone have something else I can do to get this to work?
Thank you for any advice!

Sparx · Feb 26, 2023

So that looks like the same issue I have @Ukjent1. There is a bug report on it now. Hopefully we can get some help soon.
You could reinstall 22.02.4 instead and give that a go. In that version GPU worked for me.

Ukjent1 · Feb 26, 2023

Sparx said:
So that looks like the sam issue I have @Ukjent1. There is a bug report on it now. Hopefully we can get some help soon.
You could reinstall 22.02.4 instead and give that a go. In that version GPU worked for me.

Hopefully they come with a fix very soon. I don't dear to reinstall, in case everything goes south :)

HoneyBadger · Feb 27, 2023

SCALE 22.12.1 with a Tesla P4 seems to connect properly with nvidia-smi

Code:

root@bluefin01[/home/admin]# lspci -v | grep NVIDIA
04:00.0 3D controller: NVIDIA Corporation GP104GL [Tesla P4] (rev a1)
        Subsystem: NVIDIA Corporation GP104GL [Tesla P4]
root@bluefin01[/home/admin]# nvidia-smi
Mon Feb 27 07:20:45 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P4            Off  | 00000000:04:00.0 Off |                  Off |
| N/A   33C    P8     6W /  75W |      0MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@bluefin01[/home/admin]#

Note that we're using the 515.65.01 driver in Bluefin. ~~but the list of supported chips hasn't significantly changed.~~ Edit: A Maxwell chip or newer is required as everything older was deemed "legacy" by NVIDIA and stopped with the 470.xx Linux driver in Angelfish.

Sparx · Feb 27, 2023

Interesting. I wonder why its different. Thats what i expect it to look like. And it did work in 22.02 for me. So maybe something was broken in upgrade.

HoneyBadger · Feb 27, 2023

Sparx said:
Interesting. I wonder why its different. Thats what i expect it to look like. And it did work in 22.02 for me. So maybe something was broken in upgrade.

Assuming you filed NAS-120509 - is your GPU still isolated? nvidia-smi won't find anything if that's the case.

Can I get an lspci -v ?

Sparx · Feb 27, 2023

Yeah thats me. It was the same when i had it out of isolation. Or without isolation however to say that correctly :)
I can remove the isolation and reboot if you want. Here is lspci -v (many IOMMUs removed to fit the size):

Code:


root@freenas:~# lspci -v
00:00.0 Host bridge: Intel Corporation Sky Lake-E DMI3 Registers (rev 06)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: fast devsel, NUMA node 0, IOMMU group 0
        Capabilities: [90] Express Root Port (Slot-), MSI 00
        Capabilities: [e0] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
        Capabilities: [144] Vendor Specific Information: ID=0004 Rev=1 Len=03c <?>
        Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
        Capabilities: [250] Secondary PCI Express
        Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
        Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
        Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>

00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
        Flags: bus master, fast devsel, latency 0, IRQ 400, NUMA node 0, IOMMU group 1
        Memory at 383ffff1c000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3
        Kernel driver in use: ioatdma
        Kernel modules: ioatdma

00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
        Flags: bus master, fast devsel, latency 0, IRQ 402, NUMA node 0, IOMMU group 2
        Memory at 383ffff18000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3
        Kernel driver in use: ioatdma
        Kernel modules: ioatdma

00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
        Flags: bus master, fast devsel, latency 0, IRQ 400, NUMA node 0, IOMMU group 3
        Memory at 383ffff14000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3
        Kernel driver in use: ioatdma
        Kernel modules: ioatdma

00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
        Flags: bus master, fast devsel, latency 0, IRQ 402, NUMA node 0, IOMMU group 4
        Memory at 383ffff10000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3
        Kernel driver in use: ioatdma
        Kernel modules: ioatdma

00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
        Flags: bus master, fast devsel, latency 0, IRQ 400, NUMA node 0, IOMMU group 5
        Memory at 383ffff0c000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3
        Kernel driver in use: ioatdma
        Kernel modules: ioatdma

00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
        Flags: bus master, fast devsel, latency 0, IRQ 402, NUMA node 0, IOMMU group 6
        Memory at 383ffff08000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3
        Kernel driver in use: ioatdma
        Kernel modules: ioatdma

00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
        Flags: bus master, fast devsel, latency 0, IRQ 400, NUMA node 0, IOMMU group 7
        Memory at 383ffff04000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3
        Kernel driver in use: ioatdma
        Kernel modules: ioatdma

00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
        Flags: bus master, fast devsel, latency 0, IRQ 402, NUMA node 0, IOMMU group 8
        Memory at 383ffff00000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [80] MSI-X: Enable+ Count=1 Masked-
        Capabilities: [90] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3
        Kernel driver in use: ioatdma
        Kernel modules: ioatdma

00:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers
        Flags: fast devsel, NUMA node 0, IOMMU group 9
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

00:05.2 System peripheral: Intel Corporation Sky Lake-E RAS (rev 06)
        Flags: fast devsel, NUMA node 0, IOMMU group 10
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

00:05.4 PIC: Intel Corporation Sky Lake-E IOAPIC (rev 06) (prog-if 20 [IO(X)-APIC])
        Subsystem: Intel Corporation Sky Lake-E IOAPIC
        Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 11
        Memory at aa21a000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3


00:11.5 SATA controller: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode] (rev 09) (prog-if 01 [AHCI 1.0])
        Subsystem: Super Micro Computer Inc C620 Series Chipset Family SSATA Controller [AHCI mode]
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 41, NUMA node 0, IOMMU group 15
        Memory at aa216000 (32-bit, non-prefetchable) [size=8K]
        Memory at aa219000 (32-bit, non-prefetchable)
        I/O ports at 0800
        I/O ports at 0810
        I/O ports at 0820
        Memory at aa180000 (32-bit, non-prefetchable) [size=512K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [70] Power Management version 3
        Capabilities: [a8] SATA HBA v1.0
        Capabilities: [b0] PCI Advanced Features
        Kernel driver in use: ahci
        Kernel modules: ahci

00:14.0 USB controller: Intel Corporation C620 Series Chipset Family USB 3.0 xHCI Controller (rev 09) (prog-if 30 [XHCI])
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: bus master, medium devsel, latency 0, IRQ 40, NUMA node 0, IOMMU group 16
        Memory at aa200000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [70] Power Management version 2
        Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci

00:14.2 Signal processing controller: Intel Corporation C620 Series Chipset Family Thermal Subsystem (rev 09)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: fast devsel, IRQ 18, NUMA node 0, IOMMU group 16
        Memory at 383ffff23000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 3
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
        Kernel driver in use: intel_pch_thermal
        Kernel modules: intel_pch_thermal

00:16.0 Communication controller: Intel Corporation C620 Series Chipset Family MEI Controller #1 (rev 09)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: bus master, fast devsel, latency 0, IRQ 11, NUMA node 0, IOMMU group 17
        Memory at 383ffff22000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 3
        Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
        Kernel modules: mei_me

00:16.1 Communication controller: Intel Corporation C620 Series Chipset Family MEI Controller #2 (rev 09)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: bus master, fast devsel, latency 0, IRQ 11, NUMA node 0, IOMMU group 17
        Memory at 383ffff21000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 3
        Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+

00:16.4 Communication controller: Intel Corporation C620 Series Chipset Family MEI Controller #3 (rev 09)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: bus master, fast devsel, latency 0, IRQ 11, NUMA node 0, IOMMU group 17
        Memory at 383ffff20000 (64-bit, non-prefetchable) [size=4K]
        Capabilities: [50] Power Management version 3
        Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+

00:17.0 SATA controller: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] (rev 09) (prog-if 01 [AHCI 1.0])
        Subsystem: Super Micro Computer Inc C620 Series Chipset Family SATA Controller [AHCI mode]
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 399, NUMA node 0, IOMMU group 18
        Memory at aa214000 (32-bit, non-prefetchable) [size=8K]
        Memory at aa218000 (32-bit, non-prefetchable)
        I/O ports at 0840
        I/O ports at 0850
        I/O ports at 0860
        Memory at aa100000 (32-bit, non-prefetchable) [size=512K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [70] Power Management version 3
        Capabilities: [a8] SATA HBA v1.0
        Capabilities: [b0] PCI Advanced Features
        Kernel driver in use: ahci
        Kernel modules: ahci

00:1c.0 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #1 (rev f9) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 24, NUMA node 0, IOMMU group 19
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: [disabled]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Super Micro Computer Inc C620 Series Chipset Family PCI Express Root Port
        Capabilities: [a0] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Access Control Services
        Capabilities: [220] Secondary PCI Express
        Kernel driver in use: pcieport

00:1c.5 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #6 (rev f9) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 25, NUMA node 0, IOMMU group 20
        Bus: primary=00, secondary=02, subordinate=03, sec-latency=0
        I/O behind bridge: 00003000-00003fff [size=4K]
        Memory behind bridge: a9000000-aa0fffff [size=17M]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [40] Express Root Port (Slot-), MSI 00
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Super Micro Computer Inc C620 Series Chipset Family PCI Express Root Port
        Capabilities: [a0] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Access Control Services
        Capabilities: [220] Secondary PCI Express
        Kernel driver in use: pcieport

00:1f.0 ISA bridge: Intel Corporation C622 Series Chipset LPC/eSPI Controller (rev 09)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: fast devsel, NUMA node 0, IOMMU group 21
        Kernel modules: lpc_ich

00:1f.2 Memory controller: Intel Corporation C620 Series Chipset Family Power Management Controller (rev 09)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: fast devsel, NUMA node 0, IOMMU group 21
        Memory at aa210000 (32-bit, non-prefetchable) [disabled] [size=16K]

00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 09)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: medium devsel, IRQ 16, NUMA node 0, IOMMU group 21
        Memory at 90000000 (64-bit, non-prefetchable)
        I/O ports at 0780
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801

00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series Chipset Family SPI Controller (rev 09)
        Subsystem: Super Micro Computer Inc X11SPM-TF
        Flags: fast devsel, NUMA node 0, IOMMU group 21
        Memory at fe010000 (32-bit, non-prefetchable) [size=4K]

02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 17, NUMA node 0, IOMMU group 22
        Bus: primary=02, secondary=03, subordinate=03, sec-latency=32
        I/O behind bridge: 00003000-00003fff [size=4K]
        Memory behind bridge: a9000000-aa0fffff [size=17M]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
        Capabilities: [c0] Subsystem: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge
        Capabilities: [100] Virtual Channel
        Capabilities: [800] Advanced Error Reporting

03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41) (prog-if 00 [VGA controller])
        DeviceName: ASPEED Video AST2500
        Subsystem: Super Micro Computer Inc ASPEED Graphics Family
        Flags: medium devsel, IRQ 17, NUMA node 0, IOMMU group 22
        Memory at a9000000 (32-bit, non-prefetchable) [size=16M]
        Memory at aa000000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 3000
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
        Kernel driver in use: ast
        Kernel modules: ast

16:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 06) (prog-if 00 [Normal decode])
        Physical Slot: 1
        Flags: bus master, fast devsel, latency 0, IRQ 27, NUMA node 0, IOMMU group 23
        Bus: primary=16, secondary=17, subordinate=17, sec-latency=0
        I/O behind bridge: 00007000-00007fff [size=4K]
        Memory behind bridge: c5c00000-c5efffff [size=3M]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port A
        Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
        Capabilities: [90] Express Root Port (Slot+), MSI 00
        Capabilities: [e0] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
        Capabilities: [110] Access Control Services
        Capabilities: [148] Advanced Error Reporting
        Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
        Capabilities: [250] Secondary PCI Express
        Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
        Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
        Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
        Kernel driver in use: pcieport

16:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 06)
        Subsystem: Intel Corporation Sky Lake-E VT-d
        Flags: fast devsel, NUMA node 0, IOMMU group 24
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

16:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 06)
        Flags: fast devsel, NUMA node 0, IOMMU group 25
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

16:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 06) (prog-if 20 [IO(X)-APIC])
        Subsystem: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers
        Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 26
        Memory at c5f00000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3

16:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E CHA Registers
        Flags: fast devsel, NUMA node 0, IOMMU group 27


--------------


16:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 06)
        Subsystem: Intel Corporation Sky Lake-E PCU Registers
        Flags: fast devsel, NUMA node 0, IOMMU group 36

17:00.0 Serial Attached SCSI controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)
        Subsystem: Broadcom / LSI MegaRAID SAS-3 3108 [Invader]
        Flags: bus master, fast devsel, latency 0, IRQ 39, NUMA node 0, IOMMU group 37
        I/O ports at 7000
        Memory at c5e00000 (64-bit, non-prefetchable) [size=64K]
        Memory at c5d00000 (64-bit, non-prefetchable) [size=1M]
        Expansion ROM at c5c00000 [disabled] [size=1M]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [d0] Vital Product Data
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [c0] MSI-X: Enable+ Count=97 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [1e0] Secondary PCI Express
        Capabilities: [1c0] Power Budgeting <?>
        Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: megaraid_sas
        Kernel modules: megaraid_sas

64:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 06) (prog-if 00 [Normal decode])
        Physical Slot: 5
        Flags: bus master, fast devsel, latency 0, IRQ 29, NUMA node 0, IOMMU group 38
        Bus: primary=64, secondary=65, subordinate=65, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: df000000-dfffffff [size=16M]
        Prefetchable memory behind bridge: 000038b000000000-000038b801ffffff [size=32800M]
        Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port A
        Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
        Capabilities: [90] Express Root Port (Slot+), MSI 00
        Capabilities: [e0] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
        Capabilities: [110] Access Control Services
        Capabilities: [148] Advanced Error Reporting
        Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
        Capabilities: [250] Secondary PCI Express
        Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
        Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
        Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
        Kernel driver in use: pcieport

64:05.0 System peripheral: Intel Corporation Sky Lake-E VT-d (rev 06)
        Subsystem: Intel Corporation Sky Lake-E VT-d
        Flags: fast devsel, NUMA node 0, IOMMU group 39
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

64:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 06)
        Flags: fast devsel, NUMA node 0, IOMMU group 40
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

64:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 06) (prog-if 20 [IO(X)-APIC])
        Subsystem: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers
        Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 41
        Memory at e0000000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3

64:08.0 System peripheral: Intel Corporation Sky Lake-E Integrated Memory Controller (rev 06)
        Subsystem: Intel Corporation Sky Lake-E Integrated Memory Controller
        Flags: fast devsel, NUMA node 0, IOMMU group 42
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
        Kernel driver in use: skx_uncore



-----------




64:0d.3 System peripheral: Intel Corporation Sky Lake-E LMDP Channel 2 (rev 06)
        Subsystem: Intel Corporation Sky Lake-E LMDP Channel 2
        Flags: fast devsel, NUMA node 0, IOMMU group 67
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

65:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] (rev a1)
        Subsystem: NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB]
        Flags: fast devsel, IRQ 410, NUMA node 0, IOMMU group 68
        Memory at df000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 38b000000000 (64-bit, prefetchable) [size=32G]
        Memory at 38b800000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [ac0] Designated Vendor-Specific: Vendor=10de ID=0001 Rev=1 Len=12 <?>
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau, nvidia_current_drm, nvidia_current

b2:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 06) (prog-if 00 [Normal decode])
        Physical Slot: 9
        Flags: bus master, fast devsel, latency 0, IRQ 31, NUMA node 0, IOMMU group 69
        Bus: primary=b2, secondary=b3, subordinate=b5, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: fbc00000-fbefffff [size=3M]
        Prefetchable memory behind bridge: 000038c000000000-000038c003ffffff [size=64M]
        Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port A
        Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
        Capabilities: [90] Express Root Port (Slot+), MSI 00
        Capabilities: [e0] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
        Capabilities: [110] Access Control Services
        Capabilities: [148] Advanced Error Reporting
        Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
        Capabilities: [250] Secondary PCI Express
        Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
        Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
        Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
        Kernel driver in use: pcieport

b2:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port C (rev 06) (prog-if 00 [Normal decode])
        Physical Slot: 11
        Flags: bus master, fast devsel, latency 0, IRQ 32, NUMA node 0, IOMMU group 70
        Bus: primary=b2, secondary=b6, subordinate=b9, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: fb900000-fbbfffff [size=3M]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [40] Subsystem: Super Micro Computer Inc Sky Lake-E PCI Express Root Port C
        Capabilities: [60] MSI: Enable+ Count=1/2 Maskable+ 64bit-
        Capabilities: [90] Express Root Port (Slot+), MSI 00
        Capabilities: [e0] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>
        Capabilities: [110] Access Control Services
        Capabilities: [148] Advanced Error Reporting
        Capabilities: [1d0] Vendor Specific Information: ID=0003 Rev=1 Len=00a <?>
        Capabilities: [250] Secondary PCI Express
        Capabilities: [280] Vendor Specific Information: ID=0005 Rev=3 Len=018 <?>
        Capabilities: [298] Vendor Specific Information: ID=0007 Rev=0 Len=024 <?>
        Capabilities: [300] Vendor Specific Information: ID=0008 Rev=0 Len=038 <?>
        Kernel driver in use: pcieport


b2:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 06)
        Flags: fast devsel, NUMA node 0, IOMMU group 72
        Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00

b2:05.4 PIC: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers (rev 06) (prog-if 20 [IO(X)-APIC])
        Subsystem: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers
        Flags: bus master, fast devsel, latency 0, NUMA node 0, IOMMU group 73
        Memory at fbf00000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [44] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [e0] Power Management version 3





b3:00.0 PCI bridge: Intel Corporation Device 37c0 (rev 09) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 33, NUMA node 0, IOMMU group 86
        Memory at fbe00000 (64-bit, non-prefetchable) [size=128K]
        Bus: primary=b3, secondary=b4, subordinate=b5, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: fbd00000-fbdfffff [size=1M]
        Prefetchable memory behind bridge: 000038c000000000-000038c003ffffff [size=64M]
        Expansion ROM at fbc00000 [disabled] [size=1M]
        Capabilities: [40] Express Upstream Port, MSI 00
        Capabilities: [80] Power Management version 3
        Capabilities: [88] Subsystem: Intel Corporation Device 0000
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [200] Secondary PCI Express
        Kernel driver in use: pcieport

b4:03.0 PCI bridge: Intel Corporation Device 37c5 (rev 09) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 33, NUMA node 0, IOMMU group 87
        Bus: primary=b4, secondary=b5, subordinate=b5, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: fbd00000-fbdfffff [size=1M]
        Prefetchable memory behind bridge: 000038c000000000-000038c003ffffff [size=64M]
        Capabilities: [40] Express Downstream Port (Slot-), MSI 00
        Capabilities: [80] Power Management version 3
        Capabilities: [88] Subsystem: Intel Corporation Device 0000
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [138] Access Control Services
        Kernel driver in use: pcieport

b5:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 09)
        DeviceName: Intel LAN X722 #1
        Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GBASE-T
        Flags: bus master, fast devsel, latency 0, IRQ 37, NUMA node 0, IOMMU group 88
        Memory at 38c000000000 (64-bit, prefetchable) [size=16M]
        Memory at 38c002800000 (64-bit, prefetchable) [size=32K]
        Expansion ROM at fbd00000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number e6-bb-ba-ff-ff-6b-1f-ac
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [1a0] Transaction Processing Hints
        Capabilities: [1b0] Access Control Services
        Kernel driver in use: i40e
        Kernel modules: i40e

b5:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 09)
        DeviceName: Intel LAN X722 #2
        Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GBASE-T
        Flags: bus master, fast devsel, latency 0, IRQ 37, NUMA node 0, IOMMU group 89
        Memory at 38c001000000 (64-bit, prefetchable) [size=16M]
        Memory at 38c002808000 (64-bit, prefetchable) [size=32K]
        Expansion ROM at fbd80000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number e6-bb-ba-ff-ff-6b-1f-ac
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [1a0] Transaction Processing Hints
        Capabilities: [1b0] Access Control Services
        Kernel driver in use: i40e
        Kernel modules: i40e

b6:00.0 PCI bridge: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 34, NUMA node 0, IOMMU group 90
        Memory at fbb00000 (32-bit, non-prefetchable) [size=256K]
        Bus: primary=b6, secondary=b7, subordinate=b9, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: fb900000-fbafffff [size=2M]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
        Capabilities: [68] Express Upstream Port, MSI 00
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
        Capabilities: [100] Device Serial Number aa-87-00-10-b5-df-0e-00
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [10c] Secondary PCI Express
        Capabilities: [148] Virtual Channel
        Capabilities: [e00] Multicast
        Capabilities: [b00] Latency Tolerance Reporting
        Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel driver in use: pcieport

b7:01.0 PCI bridge: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 36, NUMA node 0, IOMMU group 91
        Bus: primary=b7, secondary=b8, subordinate=b8, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: fba00000-fbafffff [size=1M]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
        Capabilities: [68] Express Downstream Port (Slot+), MSI 00
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
        Capabilities: [100] Device Serial Number aa-87-00-10-b5-df-0e-00
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [10c] Secondary PCI Express
        Capabilities: [148] Virtual Channel
        Capabilities: [e00] Multicast
        Capabilities: [f24] Access Control Services
        Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel driver in use: pcieport

b7:02.0 PCI bridge: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0, IRQ 38, NUMA node 0, IOMMU group 92
        Bus: primary=b7, secondary=b9, subordinate=b9, sec-latency=0
        I/O behind bridge: [disabled]
        Memory behind bridge: fb900000-fb9fffff [size=1M]
        Prefetchable memory behind bridge: [disabled]
        Capabilities: [40] Power Management version 3
        Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
        Capabilities: [68] Express Downstream Port (Slot+), MSI 00
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
        Capabilities: [100] Device Serial Number aa-87-00-10-b5-df-0e-00
        Capabilities: [fb4] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [10c] Secondary PCI Express
        Capabilities: [148] Virtual Channel
        Capabilities: [e00] Multicast
        Capabilities: [f24] Access Control Services
        Capabilities: [b70] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
        Kernel driver in use: pcieport

b8:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller Cx6 (rev 01) (prog-if 02 [NVM Express])
        Subsystem: KIOXIA Corporation NVMe SSD Controller Cx6
        Flags: bus master, fast devsel, latency 0, IRQ 35, NUMA node 0, IOMMU group 93
        Memory at fba10000 (64-bit, non-prefetchable) [size=32K]
        Expansion ROM at fba00000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=129 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Device Serial Number 00-ed-f0-02-e2-8e-e3-8c
        Capabilities: [158] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [168] Secondary PCI Express
        Capabilities: [188] Physical Layer 16.0 GT/s <?>
        Capabilities: [1b0] Lane Margining at the Receiver <?>
        Kernel driver in use: nvme
        Kernel modules: nvme

b9:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller Cx6 (rev 01) (prog-if 02 [NVM Express])
        Subsystem: KIOXIA Corporation NVMe SSD Controller Cx6
        Flags: bus master, fast devsel, latency 0, IRQ 37, NUMA node 0, IOMMU group 94
        Memory at fb910000 (64-bit, non-prefetchable) [size=32K]
        Expansion ROM at fb900000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable+ Count=129 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [148] Device Serial Number 00-eb-f0-02-e2-8e-e3-8c
        Capabilities: [158] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [168] Secondary PCI Express
        Capabilities: [188] Physical Layer 16.0 GT/s <?>
        Capabilities: [1b0] Lane Margining at the Receiver <?>
        Kernel driver in use: nvme
        Kernel modules: nvme

Sasquatch · Feb 28, 2023

My bluefin upgraded from angelfish has incomplete drivers. "Nvidia-drm" cannot be located. Update to 22.12.1 didn't help either.
passing GPU(p400) to VM works fine.

I wonder if people reporting successful nvidia-smi queries did install bluefin from scratch.

HoneyBadger · Feb 28, 2023

Sasquatch said:
I wonder if people reporting successful nvidia-smi queries did install bluefin from scratch.

Odd. I just did an Angelfish -> Bluefin -> Bluefin U1 update cycle, and nvidia-smi was able to query the GPU at each step. Uncertain why it wouldn't properly load the driver in your instance.

Sparx · Feb 28, 2023

I came from Angelfish to U1 directly. But I guess that should make any difference.
Also just removed the isolated GPU but still no nvidia-driver.

HoneyBadger · Feb 28, 2023

Sparx said:
I came from Angelfish to U1 directly. But I guess that should make any difference.

What do you get from the lsmod | grep nvidia and find commands in the codeblock below?

Code:

root@angelfish01[~]# lsmod | grep nvidia
nvidia_drm             73728  0
nvidia_modeset       1150976  1 nvidia_drm
nvidia              40853504  1 nvidia_modeset
drm_kms_helper        315392  2 vmwgfx,nvidia_drm
drm                   643072  6 vmwgfx,drm_kms_helper,nvidia,nvidia_drm,ttm

root@angelfish01[~]# find /lib/modules/5.15.79+truenas/ -type f -name '*.ko' | grep nvidia
/lib/modules/5.15.79+truenas/kernel/drivers/net/ethernet/nvidia/forcedeth.ko
/lib/modules/5.15.79+truenas/kernel/drivers/usb/typec/altmodes/typec_nvidia.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current-peermem.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current-uvm.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current-modeset.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current-drm.ko

Sparx · Feb 28, 2023

So first row results in a line feed. Nothing.

Code:

root@freenas:~# lsmod | grep nvidia
root@freenas:~# find /lib/modules/5.15.79+truenas/ -type f -name '*.ko' | grep nvidia
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current-uvm.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current-peermem.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current-drm.ko
/lib/modules/5.15.79+truenas/updates/dkms/nvidia-current-modeset.ko
/lib/modules/5.15.79+truenas/kernel/drivers/usb/typec/altmodes/typec_nvidia.ko
/lib/modules/5.15.79+truenas/kernel/drivers/net/ethernet/nvidia/forcedeth.ko

HoneyBadger · Feb 28, 2023

You could attempt to force-load the module via modprobe nvidia-current-drm - however, this might cause a failure if the module is corrupt somehow.

Sparx · Feb 28, 2023

It didnt want to do that.

Code:

root@freenas:~# modprobe nvidia-current-drm
modprobe: ERROR: could not insert 'nvidia_current_drm': No such device
root@freenas:~#

dmesg last rows:
[23342.617012] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
[23342.625726] NVRM: The NVIDIA probe routine was not called for 1 device(s).
[23342.635887] NVRM: This can occur when a driver such as:
               NVRM: nouveau, rivafb, nvidiafb or rivatv
               NVRM: was loaded and obtained ownership of the NVIDIA device(s).
[23342.656688] NVRM: Try unloading the conflicting kernel module (and/or
               NVRM: reconfigure your kernel without the conflicting
               NVRM: driver(s)), then try loading the NVIDIA kernel module
               NVRM: again.
[23342.682990] NVRM: No NVIDIA devices probed.
[23342.688819] nvidia-nvlink: Unregistered Nvlink Core, major device number 511
root@freenas:~# ~

Sparx · Feb 28, 2023

There is another case that seems to have similarities but its locked and i cant tell the outcome. Why do you hide trouble reports? They could really help to find out the root cause.

Multiple Tesla M40 GPU passthrough on one

I have two Tesla M40's installed in my system running TrueNAS Scale 22.02.3, and if I isolate one of the cards, they both disappear to the operating system. Any thoughts on what's going on? root@truenas[~]# nvidia-smi NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA...

www.truenas.com

https://ixsystems.atlassian.net/browse/NAS-117915

Ukjent1 · Feb 28, 2023

I tested the same, and got the same results as Sparx.

Important Announcement for the TrueNAS Community.

Nvidia GPU not appearing for use with SCALE

Explorer

Explorer

Captain Morgan

Contributor

Cadet

Contributor

Cadet

actually does care

Contributor

actually does care

Contributor

Explorer

actually does care

Contributor

actually does care

Contributor

actually does care

Contributor

Contributor

Cadet

Similar threads