Truenas Scale PCIe Passthru issue

[Sparx]

Hi!

I just upgraded to Truenas Scale from a working Core setup. The issue I have is that I want to Passthru an LSI 9380-8e (Connected to a JBOD full of ext3 devices with chia mining) to a virtual machine. I couldnt get the ext3 drives mounted as drives so I just passed them in to a VM instead.
But no PCIe device shows up to pass on in scale. IOMMU is enabled. I do believe the virtualization settings in BIOS are set correctly also since it was working before. But im checking it at the moment. Since there are a lot of drives connected to this system the bios setup is slow to get to. =)

Hardware Supermicro X11SPM-TF with Gold 6248 and 192GB RAM. LSI 9380-8e device at adress 17.00.0.

root@freenas:/mnt# dmesg | grep -e DMAR -e IOMMU
[ 0.019417] ACPI: DMAR 0x000000005B784CE8 000148 (v01 SUPERM SMCI--MB 00000001 INTL 20091013)
[ 0.019469] ACPI: Reserving DMAR table memory at [mem 0x5b784ce8-0x5b784e2f]
[ 0.030549] DMAR: IOMMU enabled
[ 1.817783] DMAR: Host address width 46

 

[Kris Moore]

Sounds like it should show up as an option to passthrough then. Can you open a ticket on https://jira.ixsystems.com and attach a debug file? (System -> Advanced -> Download debug). We'll want to take a look at that and see if there's something we need to fix to make sure it shows up as a passthrough device option.

 

[Sparx]

My thought was that since its a raid card (running in JBOD mode) it might be something that is a bit dangerous to allow in to a VM without thinking about it. Since it could be hosting other pools or even your dataset pool or similar. But its not the case for me right now. Ill post a ticket.

 

[jgreco]

My thought was that since its a raid card (running in JBOD mode) it might be something that is a bit dangerous to allow in to a VM without thinking about it. Since it could be hosting other pools or even your dataset pool or similar. But its not the case for me right now. Ill post a ticket.

Completely safe to pass in to a VM. RAID cards are dangerous to use with ZFS, and should never be used to build pools on your ZFS filer, therefore, by definition, you are actually safer handing it off to a VM.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[Sparx]

Yes thats partially true. But since the raid-card is running in JBOD-personality its actually running as an HBA.

 

[jgreco]

But since the raid-card is running in JBOD-personality its actually running as an HBA.

The linked article I provided explains why this is simply not true. You can paint a semi truck to look like a pickup truck, but it doesn't actually make it into a pickup truck.

 

[Sparx]

Its a pretty common use case in many deployments with Nexenta. Also running ZFS. Any ways its not the question of this topic.

 

[jgreco]

Its a pretty common use case in many deployments with Nexenta. Also running ZFS. Any ways its not the question of this topic.

It's a pretty common use case in many deployments with Windows too. Still, not safe under TrueNAS. And your followup seemed to suggest that perhaps it was because "it might be something that is a bit dangerous to allow in to a VM". My point is that this isn't true, and that there aren't any safety belts in TrueNAS that would implement such protection.

 

[Sparx]

Alright. That's somewhat sad. Mostly because then there would probably be a simple solution to fix and not a bug :)

 

[Sparx]

I never made it to BIOS ( Too many drives ) and gave up and rebooted to Core again.
Is there an option that one of the VT-d settings would be wrong and IOMMU would still show as enabled? I was thinking since it worked in Core it should work in Scale as well.
I can rip out the jbod cable and reboot to bios. Should be faster without the drives. Ideally i would remove the UEFI and BIOS parts of the SAS card to remove them as bootable options as I would never boot externally.

 

[jgreco]

That's somewhat sad.

Yeah, it's sad. It's really unfortunate that lots of hardware is poorly supported, or designed for other use cases, or that manufacturers stop giving a damn even before the cash register bell even rings. It's pathetic that only a few vendors, such as Intel and LSI, spend significant amounts of time on making primo drivers for non-Windows operating systems.

Even your issue, itself, may be due to deficiencies in the overall PC architecture, which of course has grown from the original IBM PC to this monstrosity with all sorts of complex subsystems with difficult interplay. PCIe passthru is sorta magic, and you always have to remember that when you're virtualizing, you're placing a complex house of cards on top inside another complex house of cards. I'm actually amazed it works as well as it does.

 

[Sparx]

Oh yes indeed. The reason I jumped to try (or sure, this isnt really jumping. Its RC after all) was that I wanted to have NVIDIA driver support for the OS. And the reason for that is going back to a Dell workstation that plainly refuses to enable a Nvidia Tesla card. I guess it doesnt have support for PCIe High BAR addressing. When going in to the BIOS of the Dell there are lots of references to laptop similar bios. On a workstation. Cant really see how thats connected.
Will be fun trying out if I can do some mining on Scale.

 

[Sparx]

I have activated Core on this computer. When I reboot and choose Scale from the grub loader it fails and cant find the freenas-boot pool. It gives an option to manually import. It seemed harsh and I booted back to core. I had hoped its possible to activate next boot as its shown in the boot options as 22.02-RC.2. But when i try to activate it, it says to reinstall Scale. How should i proceed?