I solved my problem with re-sitting GPU from first PCI-E slot to second. Now it works as it should and everything's stable.
TrueNAS SCALE is doing the passthrough into it's own VM. I had a lot of crashing too, until I manually changed the board to run the slot at PCIe 3.0. even though the board only supports PCIe 3.0, it seems the card was still trying to negotiate 4.0. so I forced it. I've had this issue with other 30-series cards.