Modified drivers for Nvidia consumer cards

[jolness1]

TrueNAS Version: SCALE-22.12-RC.1

I will soon be grabbing a new GPU for my windows box and was thinking of repurposing my trusty 1080 as a hardware transcoding device for Plex.
However, the consumer cards are capped at 2 concurrent streams (1 per encoder block on the GPU) through drivers, I don't need the number of streams it could handle (have seen numbers in the 50s) but sometimes I have up to 5 or 6 concurrent devices and if a few of them need transcoding, I would like to avoid hitting the CPU as it sucks down a ton of power and my 2660v3 isn't exactly a powerhouse, I could add a second one but GPU seems like the best option, even though it is more costly. However, there are modified drivers for linux available here: https://github.com/keylase/nvidia-patch.

If I was running an off the shelf linux box, I would be able to set it up confidently. With TrueNAS being more of an appliance I avoid installing packages or other software that isn't blessed by the devs there since I don't know how things will play together and an update could break something which I don't want to deal with. For that reason, I am seeing if anyone has any experience running modified drivers, I can't see any reason why it would be an issue but as this box is something I rely on quite heavily for backups, media storage, VMs etc, I try to be cautious. Even if folks don't have experience with it but have some sort of insight as to why this would be okay or why it would be a bad idea, that would be appreciated too.

Ultimately, I can sell my 1080 and get a p2000 or similar but I know my card has been cared for, I have repasted and done new thermal pads in the last 18 months or so, heatsink is clean and it's had a very easy life. I know the quadro p2000 cards are low power, clocked lower and likely weren't beat on but would rather keep my card if at all possible.

Thanks in advance!

 

[jgreco]

If I was running an off the shelf linux box, I would be able to set it up confidently. With TrueNAS being more of an appliance I avoid installing packages or other software that isn't blessed by the devs there since I don't know how things will play together and an update could break something which I don't want to deal with.

Well, on one hand, that's the right understanding to have about the nature of the appliance OS. On the other hand, I will point out that as it is an appliance, downloading your configuration to safe storage (maybe on a PC?) makes it relatively safe to experiment on the NAS; if the driver update works, that's great. If not, you can reinstall the NAS and upload your configuration to restore things to just the way they were.

With that in mind, I will warn you that the tool you link to looks like a bash shell script of some sort. I haven't looked at it in depth, but be aware that the NAS may lack some of the tools needed by a script to manipulate drivers. Expect that there's a good chance of problems if you wish to do this. I heartily encourage someone with an appropriate GPU and desire to make this work to figure out a TrueNAS compatible solution though. The artificial GPU capability limit thing sucks.

 

[jolness1]

I really spent a lot of time considering how I wanted to set up the server. Considered linux and "rolling my own" NAS/VM box but the ease of FreeNAS out of the box seemed worth a little bit of flexibility lost, I think of it like an off the shelf NAS except much better and hardware flexibility. I guess that's true, I can just back it up, my data isn't going to get torched unless I really screw up the driver modifications. I think I will give it a go and document what I do if it works (or even if it doesn't) for folks wondering the same thing.

I need to clone the repo and take a closer look, see what dependencies are required etc. but figured I would just put feelers out first to see if anyone had done it and had horrible issues. It is unfortunate Nvidia does this, I get that they want to prevent people from using these for professional work but it still sucks.

Appreciate the response!

 

[kiler129]

This will most likely work, as the script is essentially doing a binary patch on the driver. It shouldn't nuke your TNS box ;) However, remember: you have the power of ZFS with you. You can always create a snapshot of the boot volume before you start playing with the script and roll back if something explodes. Worst case scenario it will break your NV driver, which TNS doesn't need to boot.

 

[jgreco]

I guess that's true, I can just back it up, my data isn't going to get torched unless I really screw up the driver modifications.

I can't imagine a realistic scenario where this would happen, just from trying to install a video driver. ZFS is fragile to certain types of corruption, because it is reliant on accurately posting transaction groups out to your pool. You may have seen the scary "No RAID controllers!" and "Use ECC!" posts and stuff like that. ZFS lacks the ability to do an fsck or chkdsk; once your pool is corrupt, game over man, game over! But ZFS is also very resilient when you have something like a general system panic unrelated to the disk subsystem. At worst I can see it trying to boot, failing to import the pool, and making you unwind a transaction group to regain consistency. Which may be a slightly scary manual process, but not a data-torching event.

Not a license to do anything stupid and also not a guarantee, just an opinion from someone who's been using ZFS a really long time.

 

[shadofall]

just search to forum for keylase you arnt the first to ask of discuss it..

 

[jolness1]

@shadofall I guess searching by the name of it rather than "modified nvidia drivers" might have been a good idea.
Gonna give this a go in a few days when my 4090 shows up at my local Best Buy (wish they weren't the only source, wish they would just ship direct but worth the wait).

 

[jolness1]

I can't imagine a realistic scenario where this would happen, just from trying to install a video driver. ZFS is fragile to certain types of corruption, because it is reliant on accurately posting transaction groups out to your pool. You may have seen the scary "No RAID controllers!" and "Use ECC!" posts and stuff like that. ZFS lacks the ability to do an fsck or chkdsk; once your pool is corrupt, game over man, game over! But ZFS is also very resilient when you have something like a general system panic unrelated to the disk subsystem. At worst I can see it trying to boot, failing to import the pool, and making you unwind a transaction group to regain consistency. Which may be a slightly scary manual process, but not a data-torching event.

Not a license to do anything stupid and also not a guarantee, just an opinion from someone who's been using ZFS a really long time.

I have read more than a few comments from you over the last several years about it being an appliance OS and not mucking about with other packages and stuff so if you think it is something that is unlikely to be an issue, I trust you judgement.

It also makes perfect sense, it's not like I will be doing anything that could grenade the data.

I am so excited for this, being able to transcode my 4K content to 1080p when I am on the road via a GPU should cut down on my power usage a lot, the E5 2660v3s I have are solid but they can suck down a lot of power transcoding a few streams.

Appreciate your help!

 

[jgreco]

I have read more than a few comments from you over the last several years about it being an appliance OS and not mucking about with other packages and stuff so if you think it is something that is unlikely to be an issue, I trust you judgement.

Well, be careful to understand that I am not saying it will be problem free. I am just saying it is unlikely to drown puppies or anything like that. Your data should be fine at the end of the day, but you may muck with the ability of the system to update, etc.

As long as you pick up on the idea that the goal of most of us on the forum is protecting your data, there is definitely a third option between "the right way" and "the wrong way." I am fine with people doing potentially dangerous things as long as they have been informed of the hazards, or even stuff that's just ill advised as long as they understand the implications.