Sorry I was a little in a hurry earlier.
You can find out what is happening exactly by going to Plex web.
In the upper right hand corner there is a link to jump to the dashboard:
View attachment 59218
When a stream is running you can see it on top.
If you pick more details and you can see what Plex is doing:
View attachment 59219
Here is one example. I started a stream from Plex for iOS, the device is determined to be local. The video source is 1080p HEVC and audio track is DTS-HD MA. In both cases he says it is 'Direct Play', i.e. no transcoding required.
Here is case number two:
View attachment 59220
Here the client is the same. The source is 4K HDR h264 which needs to be transcoded to 4k h264 (so HDR is an issue here) and the same goes for audio as TrueHD 7.1 is transomed to AAC.
Maybe you can run a test with one of your samples and take a screen shot.
The reason I am asking is two fold.
A. I believe that most of the transcoding can actually be avoided by pairing source material and end device compatibility intelligently. If transcoding can be avoided it saves you energy, hardware and a lot of grey hairs.
B. from what I see in the previous analysis your Geforce card seemed to work correctly. The secret with hardware transcoding is that it has very specific requirements (see the compatibility matrix
here). Maybe you are throwing a codec at your card that it simply cannot handle.