Issues installing NVIDIA legacy drivers

[BoomShaka]

Long story short:
I tried to get drivers working on for an older GPU, Geforce 560M, and ended up bricking my system after countless methods failed.
While bricking the system was very annoying, it isn't a huge issue. However, I would appreciate some help setting up the drivers once I reinstall SCALE.

The hardware specs are a bit unconventional, but they were working fine enough.
ASUS G74SX laptop:
-Intel i7-2630QM
-Intel® HM65 Express chipset
-12GB DDR3-SDRAM 1333MHz
-NVIDIA® GeForce® GTX 560M 3GB GDDR5
-Crucial MX500 500GB 2.5" SATA SSD

I will later add:
-Seagate Ironwolf 4TB 3.5" SATA (x2)
This will be the main storage pool.

The old install, as well as the new install will use the latest version of SCALE.
The application is for home server use, not production.
The end goal of getting the GPU to work is to fold with the TrueCharts Folding@Home app and eventually some Plex decoding etc.

From what I can tell, SCALE already had Nvidia drivers installed. They were simply called nvidia-driver with no number attached to the end of the name. The version in the description was 406 or 408, seemed to be the latest.

What I tried:

1. First i tried assuming the driver was already working and adding the environment variable NVIDIA_VISIBLE_DEVICES=all as per this documentation, since the TrueCharts F@H app is based off that. The GPU did not show up. I tried adding the package nvidia-docker2 from CLI as per the documentation, and this also didn't help.

2. Tried running nvidia-smi to get info on the driver and it gave NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

3. I installed nvidia-detect to check what driver i needed. It recommended nvidia-legacy-390xx-driver. I removed nvidia-driver with apt and installed the new driver also with apt. No improvemt from F@H and nvidia-smi was still failing.

4. I got frustrated individually did apt remove every package that started with nvidia-. The next step was going to be to "clean" install the nvidia-legacy-390xx-driver. However out of habit, i did apt autoremove since there were a lot of "useless" packages. A bunch of these were system packages since i guess they installed at the same time as the default driver. System officially bricked.

During step 3 and 4, nvidia-smi was giving me a different error: Failed to initialize NVML: Driver/library version mismatch. I tried unloading the current drivers loosely following this post. This changed the error back to the original error.


Sorry if i can't give more detailed diagnostic since the system is bricked, but any help with the future install would be helpful!

 

[Ixian]

As you've discovered, you can't really use apt commands with SCALE, it's based on Debian, not a general-purpose Debian variant.

There have been tons of these posts with users confused thinking that because it's based on Debian they can treat it like a normal install. It isn't. The apt commands are only there for testers to use in specific scenarios; I believe Ix System will remove them in a future release.

I don't think you, as an end user, can roll back to an older Nvidia driver. There are all manner of dependencies which won't be met and you won't easily be able to work around because, again, Ix didn't just take a standard Debian distro and slap their GUI front end on it, they've made significant behind the scenes changes to get everything working together smoothly to provide an "appliance" type distribution.

 

[BoomShaka]

As you've discovered, you can't really use apt commands with SCALE, it's based on Debian, not a general-purpose Debian variant.

There have been tons of these posts with users confused thinking that because it's based on Debian they can treat it like a normal install. It isn't. The apt commands are only there for testers to use in specific scenarios; I believe Ix System will remove them in a future release.

I don't think you, as an end user, can roll back to an older Nvidia driver. There are all manner of dependencies which won't be met and you won't easily be able to work around because, again, Ix didn't just take a standard Debian distro and slap their GUI front end on it, they've made significant behind the scenes changes to get everything working together smoothly to provide an "appliance" type distribution.

Yes, I understand the appliance nature of Truenas. Right now though, it conveniently falls in a unicorn zone due to the limited ability to mess with the underworkings when necessary.

What would you suggest is the move going forward? Submit a feature request? Either way I would like the gpu this work, so I'm just going to end up borking my install until I figure it out.

Appliance or not, right now SCALE is contending to be a good all in one NAS and virtualisation platform with the convenience of Debian. With things like this though, their options are to either integrate it as a baked in feature or make it less impossible to do via CLI. I can't really comment on which would be harder to implement but I wouldn't mind either.

 

[Ixian]

Yes, I understand the appliance nature of Truenas. Right now though, it conveniently falls in a unicorn zone due to the limited ability to mess with the underworkings when necessary.

What would you suggest is the move going forward? Submit a feature request? Either way I would like the gpu this work, so I'm just going to end up borking my install until I figure it out.

Appliance or not, right now SCALE is contending to be a good all in one NAS and virtualisation platform with the convenience of Debian. With things like this though, their options are to either integrate it as a baked in feature or make it less impossible to do via CLI. I can't really comment on which would be harder to implement but I wouldn't mind either.

There's no harm in submitting a request for it, though the use-case is pretty narrow so hard to say if it would be implemented.

We all face the same choice: Understand, accept, and perhaps try to influence the direction SCALE is heading, or look for something else. Right now I run "vanilla" docker (compose) on my SCALE system, it works quite well, with a couple minor adjustments, but given the clear direction SCALE has set out for how apps will be managed I'm likely going to move away from it in the future. I really like the NAS aspects of SCALE (and Core, for that matter) so perhaps I'll run it on lower-powered hardware as a dedicated storage appliance and host apps/VMs on a platform/OS more flexible for that use case with my higher-powered hardware.

 

[BoomShaka]

I'll submit a request for now and continue trying to find a workaround. My best guess is that I might be able to stop the installer from setting up Nvidia stuff from the get go to avoid weird dependency associations / deletions later down the line.

In the mean time if anyone else has ways of getting around this please share.

 

[BoomShaka]

So after doing some digging I've decided not to submit a request. Here's why:

Installing, configuring and and getting apps to recognize the Nvidia legacy drivers & legacy opencl drivers is actually a piece of cake. The only thing to keep in mind is to not use apt to delete the pre installed drivers. Unfortunately though, even with all this "working", I slowly discovered that basically no software supports Fermi architecture anymore. What I mean by this is that the CUDA and opencl version included in the last working driver for my card is not supported by most software. This means that with this setup, it's theoretically possible to get some software working, but unlikely.

For anyone wondering about the process:
- apt install nvidia-detect
- nvidia-detect, this told me nvidia-legacy-390xx-driver. This is what the rest of the instructions are based off of.
- apt install nvidia-legacy-390xx-driver
- update-alternatives --config nvidia and select the legacy driver
- apt install nvidia-legacy-390xx-opencl-icd
- apt install clinfo
Everything should be set up on bare metal now, running clinfo and nvidia-smi should give something like:

So now to get the containers to see this:
- apt install nvidia-docker2
- systemctl restart docker
- Create an app and add 2 environment variables, NVIDIA_VISIBLE_DEVICES=all, NVIDIA_DRIVER_CAPABILITIES=compute,utility.
The app should now see the drivers, i can't remember if i also did this:
mkdir -p /etc/OpenCL/vendors && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
Since getting to this point i have killed the app, but it showed the proper info for both cuda and opencl, but did not recognize and usable gpus. The cuda version was 2.1 and opencl was 1.1, which from online resources, are pretty unfortunate versions.

I guess my main takeaway is that SCALE is surprisingly nice to you if you use it's Debian underbelly as long as you don't try to remove the packages that came with the initial install. Since they all installed at once, apt gets confused and thinks they are related.

Maybe i'll get some video transcode working in the future, but for now i'll leave it here. Hopefully someone builds off this or at least finds it useful and saves some time.

 

[truecharts]

To be clear: The backend for GPU support in docker, is Kubernetes not docker itself.
So you should not be installing drivers and mounts for the normal "docker way" either...

Either they might break things, not work or both.

 

[Ixian]

I get that used hardware and video cards, even secondhand, are a bit difficult to be choosy over today due to chip shortages and crypto-backed botters but it still seems to me it would be a whole lot easier to simply use better suited hardware i.e. not a laptop with an ancient GPU or if that is not in the cards, then a regular Ubuntu/etc. install than trying to reverse-engineer SCALE to suit this purpose.

 

[BoomShaka]

Yup, there's definitely paths of less resistance than the one I've chosen. This is more of an experiment to see how far I can stretch hardware that I have laying around the house.

In terms of bare Ubuntu servers, I've already experimented with that on this machine. It worked great actually, never got around to testing the gpu though. The really tight NAS features of Truenas were really appealing though, so I figured I'd try it out.

 

[BoomShaka]

To clarify a bit, I'll be using this machine as the NAS and hosting apps and containers that only use cpu going forward. In that regard it's actually performing great. For comparison, I'm upgrading from a Pi4.

Also thanks for clarifying that all the apps and docker are built on a kubernetes back-end, that actually clears up a lot of confusion.

 

[Ukhupacha]

To clarify a bit, I'll be using this machine as the NAS and hosting apps and containers that only use cpu going forward. In that regard it's actually performing great. For comparison, I'm upgrading from a Pi4.

Also thanks for clarifying that all the apps and docker are built on a kubernetes back-end, that actually clears up a lot of confusion.

Hey man, I have a GTX 550 Ti that is about 10 years old or more. I was wondering if the process for you worked before.

Kindly,

Ukhu

 

[BoomShaka]

Hey man, I have a GTX 550 Ti that is about 10 years old or more. I was wondering if the process for you worked before.

Kindly,

Ukhu

Sadly no. After the removal of apt, I haven't been able to get the old GPU to work at all.

 

[BoomShaka]

One sort of off topic work around I have implemented though, is to offload work that would normally require a gpu to my gaming rig on the same network.

For example, for my media server stuff, I transcode all the files to a format that is compatible with all my devices before hand so that the server can direct stream 99% of the time. This transcoding uses the gaming rig as a remote node (using Unmanic).

 

[aramis0221]

Same issue here. Running Truenas Scale 22.12.1 with an Nvidia Quadro 6000

 

[12f23eddde]

So after doing some digging I've decided not to submit a request. Here's why:

Installing, configuring and and getting apps to recognize the Nvidia legacy drivers & legacy opencl drivers is actually a piece of cake. The only thing to keep in mind is to not use apt to delete the pre installed drivers. Unfortunately though, even with all this "working", I slowly discovered that basically no software supports Fermi architecture anymore. What I mean by this is that the CUDA and opencl version included in the last working driver for my card is not supported by most software. This means that with this setup, it's theoretically possible to get some software working, but unlikely.

For anyone wondering about the process:
- apt install nvidia-detect
- nvidia-detect, this told me nvidia-legacy-390xx-driver. This is what the rest of the instructions are based off of.
- apt install nvidia-legacy-390xx-driver
- update-alternatives --config nvidia and select the legacy driver
- apt install nvidia-legacy-390xx-opencl-icd
- apt install clinfo
Everything should be set up on bare metal now, running clinfo and nvidia-smi should give something like:
View attachment 53311
View attachment 53313
So now to get the containers to see this:
- apt install nvidia-docker2
- systemctl restart docker
- Create an app and add 2 environment variables, NVIDIA_VISIBLE_DEVICES=all, NVIDIA_DRIVER_CAPABILITIES=compute,utility.
The app should now see the drivers, i can't remember if i also did this:
mkdir -p /etc/OpenCL/vendors && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
Since getting to this point i have killed the app, but it showed the proper info for both cuda and opencl, but did not recognize and usable gpus. The cuda version was 2.1 and opencl was 1.1, which from online resources, are pretty unfortunate versions.

I guess my main takeaway is that SCALE is surprisingly nice to you if you use it's Debian underbelly as long as you don't try to remove the packages that came with the initial install. Since they all installed at once, apt gets confused and thinks they are related.

Maybe i'll get some video transcode working in the future, but for now i'll leave it here. Hopefully someone builds off this or at least finds it useful and saves some time.

Huge thank you for your instructions! I'm trying get my quadro k2000 to work on 22.12 (based on debian bookworm, linux kernel 6.1). nvidia-detect complains on 22.12:
Uh oh. Failed to identify your Debian suite.
Luckily, the good old legacy 470 driver is still in the repository:
apt install nvidia-tesla-470-driver
Then: update-alternatives --config nvidia and select the legacy driver
With a reboot, K2000 pops up in nvidia-smi.
Feels like you can play around in TS like a bare-metal debian if you would take the risk of breaking and fixing things.

 

[Papa G]

So after doing some digging I've decided not to submit a request. Here's why:

Installing, configuring and and getting apps to recognize the Nvidia legacy drivers & legacy opencl drivers is actually a piece of cake. The only thing to keep in mind is to not use apt to delete the pre installed drivers. Unfortunately though, even with all this "working", I slowly discovered that basically no software supports Fermi architecture anymore. What I mean by this is that the CUDA and opencl version included in the last working driver for my card is not supported by most software. This means that with this setup, it's theoretically possible to get some software working, but unlikely.

For anyone wondering about the process:
- apt install nvidia-detect
- nvidia-detect, this told me nvidia-legacy-390xx-driver. This is what the rest of the instructions are based off of.
- apt install nvidia-legacy-390xx-driver
- update-alternatives --config nvidia and select the legacy driver
- apt install nvidia-legacy-390xx-opencl-icd
- apt install clinfo
Everything should be set up on bare metal now, running clinfo and nvidia-smi should give something like:
View attachment 53311
View attachment 53313
So now to get the containers to see this:
- apt install nvidia-docker2
- systemctl restart docker
- Create an app and add 2 environment variables, NVIDIA_VISIBLE_DEVICES=all, NVIDIA_DRIVER_CAPABILITIES=compute,utility.
The app should now see the drivers, i can't remember if i also did this:
mkdir -p /etc/OpenCL/vendors && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
Since getting to this point i have killed the app, but it showed the proper info for both cuda and opencl, but did not recognize and usable gpus. The cuda version was 2.1 and opencl was 1.1, which from online resources, are pretty unfortunate versions.

I guess my main takeaway is that SCALE is surprisingly nice to you if you use it's Debian underbelly as long as you don't try to remove the packages that came with the initial install. Since they all installed at once, apt gets confused and thinks they are related.

Maybe i'll get some video transcode working in the future, but for now i'll leave it here. Hopefully someone builds off this or at least finds it useful and saves some time.

I had to chmod +x /usr/bin/apt* /usr/bin/dpkg to follow this, along with all commands in sudo.

But you are a legend for providing this! I couldn't get the jellyfin container to deploy, was just stuck in that phase and mentioned no GPU.

I wasn't able to restart docker using that command, but I went back to the apps and jellyfin was running! I didn't change any of the env vars.