Passthrough for SATA drives

[Jethro]

I have 3 older SATA drives in my TrueNAS box that are not suitable to be used by ZFS. However, I am wondering whether it is possible to make them directly available to a VM.

I've tried the PCI Pass-through device with some items in the list here. I am able to get the USB controller to give me access to a couple of USB drives. But nothing seems to work for the SATA drives. In fact, when I tried one of the SATA controllers, it made TrueNAS unbootable, and I had to reinstall it. (Not a problem, because I am still in the initial stages of setting up this system.)

Are my SATA disk drives just not addressable in this manner?

I am using TrueNAS SCALE on an ASRock motherboard.

Any insight would be helpful. Thanks.

 

[Samuel Tai]

Are my SATA disk drives just not addressable in this manner?

No, at least not without additional hardware. You have 2 SATA controllers on your motherboard:

• 00:17.0 - This is the SATA controller from your motherboard chipset, which is used to access the boot drives. Passing this through, as you've discovered, makes your host non-bootable.
• 03:00.0 - This is the SATA controller attached to your IPMI, which is used to mount ISOs over the network for remote console OS installs

You'll need to purchase another controller, preferrably a SAS HBA in IT mode (which can be had off eBay for $20-30), to pass that through to your VM.

 

[Jethro]

No, at least not without additional hardware. You have 2 SATA controllers on your motherboard:

• 00:17.0 - This is the SATA controller from your motherboard chipset, which is used to access the boot drives. Passing this through, as you've discovered, makes your host non-bootable.
• 03:00.0 - This is the SATA controller attached to your IPMI, which is used to mount ISOs over the network for remote console OS installs

You'll need to purchase another controller, preferrably a SAS HBA in IT mode (which can be had off eBay for $20-30), to pass that through to your VM.

Thanks for the expert reply, Samuel.

When ordering the parts for this system, I accidentally ordered a SATA card. I had no use for it. But I didn't return it because I thought that I might have a use for it in the future. Lo and behold! I'm going to go find it and plug it in asap.

 

[Jethro]

No, at least not without additional hardware. You have 2 SATA controllers on your motherboard:

• 00:17.0 - This is the SATA controller from your motherboard chipset, which is used to access the boot drives. Passing this through, as you've discovered, makes your host non-bootable.
• 03:00.0 - This is the SATA controller attached to your IPMI, which is used to mount ISOs over the network for remote console OS installs

You'll need to purchase another controller, preferrably a SAS HBA in IT mode (which can be had off eBay for $20-30), to pass that through to your VM.

I installed that PCIe-to-SATA adapter, and it works great with the PCI pass-through.

I tried PCI pass-through with the USB controller. As expected all the USB ports now show up in the VM and none inTrueNAS. I have one more PCIe slot that I can use for a USB card. Onward and upward! Thanks again.

 

[Jethro]

A word of caution about this PCI pass-through business. I removed a different PCIe device from my system. The data from the drives connected to the PCIe/SATA card completely disappeared. The drives showed up in the guest (Linux). But they were empty, and I couldn't write to them. Turns out the PCI address for the card changed. I changed the VM PCI device to the new address, and the data reappeared.

 

[Scharbag]

If you are serious about PCI Passthrough, you really should get a proper HBA card. The LSI 9211-8i (SAS 2008 family) is excellent for this application.

Passing through USB drives may not be the best option for stability.

Cheers,

 

[Jethro]

If you are serious about PCI Passthrough, you really should get a proper HBA card. The LSI 9211-8i (SAS 2008 family) is excellent for this application.

Passing through USB drives may not be the best option for stability.

Cheers,

No, I'm not serious.

 

[homer27081990]

Try passing the pcie port (not the entire pcie controller) where the SATA controller is.

 

[Samuel Tai]

Try passing the pcie port (not the entire pcie controller) where the SATA controller is.

That won't work.

 

[homer27081990]

That won't work.

You are right, and I must take greater care to fully read all posts. I skipped the part with the PCIe SATA controller.

 

[Scharbag]

This is sorta like ZFS - while it is "possible" to make a pool that includes a Z1, Z2, Z3 and mirrored VDEV, it is not wise.

Virtualizing TrueNAS is the same. While it may be "possible" to passthrough odd controllers, it is equally not wise. If it is for fun and testing, then fine - the risk should be minimal. But if you are serious about running TrueNAS on top of a Tier 1 Hypervisor, then there are some simple best practices that should be followed. Chief among these is to passthrough a PROPER HBA card, such as an LSI SAS 2008 chipset card (I sound like a broken record). Even though ESXi 7 will warn that the SAS 2008 chipset is no longer supported in ESXi, it will passthrough fine to TrueNAS.

The worst thing you can do is try to fool around with cheap, untested hardware when using a Tier 1 hypervisor and TrueNAS.

Do it properly or make sure your backups are up do date.

Cheers,

 

[Alex_K]

What is the PCIe-to-SATA you accidentally ordered, can you tell Jethro ? Its not impossible that its LSI based and could be crossflashed.

 

[jyavenard]

On a similar note:
Could using the host block device to the sata drive be doable?
right now, bhyve is started with a ahci device that links to something like `/dev/zvol/pool/vms/zvol_name`

truenas doesn't appear to provide much of what bhybe allows.
giving it as argument something like `-s 3,ahci-hd,/dev/da0,sectorsize=512` should do the trick.

But unless I'm mistaken, there doesn't appear to be a similar option available in truenas

 

[Scharbag]

On a similar note:
Could using the host block device to the sata drive be doable?
right now, bhyve is started with a ahci device that links to something like `/dev/zvol/pool/vms/zvol_name`

truenas doesn't appear to provide much of what bhybe allows.
giving it as argument something like `-s 3,ahci-hd,/dev/da0,sectorsize=512` should do the trick.

But unless I'm mistaken, there doesn't appear to be a similar option available in truenas

This sounds like a terrible idea.

 

[Davvo]

On a similar note:
Could using the host block device to the sata drive be doable?
right now, bhyve is started with a ahci device that links to something like `/dev/zvol/pool/vms/zvol_name`

truenas doesn't appear to provide much of what bhybe allows.
giving it as argument something like `-s 3,ahci-hd,/dev/da0,sectorsize=512` should do the trick.

But unless I'm mistaken, there doesn't appear to be a similar option available in truenas

You want to read the following resource (and the ones linked to it).

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

 

[jyavenard]

This sounds like a terrible idea.

And why is that?
when it's a perfectly workable solution with all systems, and in fact bhyve supports it to start.

 

[jyavenard]

You want to read the following resource (and the ones linked to it).

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

If I was to follow what is stated there, you wouldn't use TrueNAS virtualisation system to start. bhyve isn't a type 1 supervisor.

 

[Davvo]

If I was to follow what is stated there, you wouldn't use TrueNAS virtualisation system to start. bhyve isn't a type 1 supervisor.

For instance, KVM and bhyve are kernel modules that effectively convert the host operating system to a type-1 hypervisor.

Hypervisor - Wikipedia

en.m.wikipedia.org

Besides, I don't see the correlation: it's one thing to virtualize TN, another to virtualize in TN.

You were told where to look, whether you follow or not it's up to you.
You should also start your own thread.