Using CLI to passthrough HDD to a VM

[Raitei Kijin]

Hi guys, I am running TrueNAS Scale 22.01-RC2 on an Epyc 3201 board with an Adaptec 71605 controller configured as a HBA Adapter and I need to pass several disks to a VM hosted by the integrated KVM. I know TrueNAS does not provide support through Web GUI, but I am pretty sure QEMU can do the trick.

By any chance, has anyone tried to do this before?

Also, I looked into the Internet, but it seems every tool I can use to communicate with the hypervisor fails because the hypervisor itself doesn't use standard (expected) settings (e.g.: virsh looks for a socket that apparently is not in the "right" place).
Can anyone suggest me what tool I can use to interact with QEMU and what parameters I need to allow it to connect to the hypervisor?



Many thanks to everyone

 

[jgreco]

The 71605 is a RAID controller and cannot truly be configured as an "HBA adapter".

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

You should avoid using the 71605 for your ZFS pool. However, it is fine to use PCIe passthru to pass the 71605 itself to the virtual machine of your choice.

 

[Raitei Kijin]

Thanks jgreco, I am fully aware that it is better to have ZFS manage the disks and forget about the RAID, but I need a lot of SATA ports. Therefore, I configured the adapter to not install its BIOS and behave just like an HBA: this way TrueNAS can have direct access to all disks, you can trust me on this

Unfortunately, I am not able to passthrough the whole adapter because all my disks are attached to it. I have 13 disks: 4 are managed by TrueNAS as a RAID-Z, and the others comes from a Windows-based Storage Pool. Not a good scenario...
I tried to build a filly virtual infrastructure using VMware ESXi as hypervisor, but since ESXi has no working drivers for Adaptec 71605 it has to peacefully die in a corner. Therefore I have to use TrueNAS as a Hypervisor.

 

[jgreco]

configured the adapter to not install its BIOS and behave just like an HBA: this way TrueNAS can have direct access to all disks, you can trust me on this

I do not trust you on this. I know you THINK what you are doing is okay. However, there was a reason I wrote the article that I linked to above, and it is unwise to rely on an Adaptec controller, regardless of the number of ports you need. High port counts are available on LSI HBA's or via SAS expanders, both known to work swimmingly well.

but since ESXi has no working drivers for Adaptec 71605

Because it's a crappy outdated controller. Actually I'm pretty sure 6.7 has legacy Linux drivers for it, but legacy drivers got dropped in 7.0, and Adaptec isn't going to go write a native driver for a controller they no longer sell. This is sort of related to why you shouldn't use it for TrueNAS either.

If you wanted to make a converged platform, you could consider using ESXi 7 along with a reasonable LSI 3108-based RAID controller to hold your virtual machines. If you only need a small number of SATA ports for NAS, you can then pass thru the mainboard controller to a TrueNAS VM. We commonly do that here for hypervisors, using a 3108 and some SSD's for ESXi datastores, then SATA ports or an LSI 3008 HBA passed through to TrueNAS for fileservice. That way, the ESXi gets the high performance hardware RAID you want for that, and TrueNAS VM gets a well-supported HBA that is excellent for that application.

 

[Raitei Kijin]

Ok, I understand your point, bot I do not agree completely with you, especially because I did my homework before doing this. BUT, this is going out of topic, and we should not want to continue further.

Going back to the main question, I did some research and I found out a way to do it.
Maybe it is not the right one, and surely it is not supported, but for the ones who want to try it I can follow these steps:

Prerequisites:
Ensure you have SSH access as root to your TrueNAS; also, locate (or create) the VM you want to attach the disks and power it on (the attach is not possible if the VM is not running). It is a good thing to have the Guest OS already set up and running.

Note that this procedure uses the utility virsh, available by default on TrueNAS Scale 22.02-RC2.

Procedure:
1. Login to TrueNAS using SSH with root user
2. Locate the disk(s) you want to pass to the VM. The right thing is to not use the standard name (like /dev/sda etc...), but use the ID provided by /dev/disk/by-id. To locate them, this command can be really useful:
find /dev/disk/by-id/ -type l|xargs -I{} ls -l {}|grep -v -E '[0-9]$' |sort -k11|cut -d' ' -f9,10,11,12

Here an example outout:

Code:

root@datacenter[~]# find /dev/disk/by-id/ -type l|xargs -I{} ls -l {}|grep -v -E '[0-9]$' |sort -k11|cut -d' ' -f9,10,11,12 | sort
/dev/disk/by-id/ata-KINGSTON_SA400M8120G_50026B7685357785 -> ../../sdn
/dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2KB41G -> ../../sdb
/dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2KG36H -> ../../sdd
/dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2KG8BD -> ../../sda
/dev/disk/by-id/ata-ST16000NM001G-2KK103_ZL2KGSRT -> ../../sdc
/dev/disk/by-id/ata-Samsung_SSD_850_EVO_500GB_S21JNSAG158509B -> ../../sdj
/dev/disk/by-id/ata-Samsung_SSD_850_EVO_500GB_S2RBNX0H937583L -> ../../sdk
/dev/disk/by-id/ata-WDC_WD10JFCX-68N6GN0_WD-WX11A15ET12E -> ../../sdm
/dev/disk/by-id/ata-WDC_WD10JFCX-68N6GN0_WD-WX11A15ETRZ9 -> ../../sdl
/dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1DFDE54 -> ../../sdi
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0135118 -> ../../sde
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0185240 -> ../../sdh
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0243702 -> ../../sdg
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4ENSDRE27 -> ../../sdf
/dev/disk/by-id/wwn-0x5000c500dbd815d2 -> ../../sdb
/dev/disk/by-id/wwn-0x5000c500dbe1b5b7 -> ../../sdd
/dev/disk/by-id/wwn-0x5000c500dbe40981 -> ../../sdc
/dev/disk/by-id/wwn-0x5000c500dbe46e89 -> ../../sda
/dev/disk/by-id/wwn-0x50014ee209112b98 -> ../../sde
/dev/disk/by-id/wwn-0x50014ee20b01a0fc -> ../../sdf
/dev/disk/by-id/wwn-0x50014ee210a8b4bc -> ../../sdi
/dev/disk/by-id/wwn-0x50014ee25e74959c -> ../../sdg
/dev/disk/by-id/wwn-0x50014ee2b3bbed86 -> ../../sdh
/dev/disk/by-id/wwn-0x50014ee6b01bee55 -> ../../sdl
/dev/disk/by-id/wwn-0x50014ee6b01bfbaa -> ../../sdm
/dev/disk/by-id/wwn-0x5002538d414b3774 -> ../../sdk
/dev/disk/by-id/wwn-0x5002538da015f254 -> ../../sdj
/dev/disk/by-id/wwn-0x50026b7685357785 -> ../../sdn
root@datacenter[~]#

In this output, it is possible to see each available disk with its ID and its name, therefore identify them is easy.

3. For each disk you want to passthrough, decide for a Target Name. This is only a name used by QEMU to refer to the disk. The name must be in Linux standard, therefore an easy way is to replace the first letter of the existing name with v.
E.g.: sdj becomes vdj

4. Make virsh to connect to the QEMU environment. Connection is made through a socket, which on TrueNAS Scale seems to be located at /run/truenas_libvirt/libvirt-sock=. The most obvious way is to create an alias:
alias virsh='virsh -c "qemu+unix:///system?socket=/run/truenas_libvirt/libvirt-sock" $1'

4. Find the name of your VM. This is required because the VM Name is slightly different from the one showed by TrueNAS Web Gui:
virsh list --all

Here an example output (note that on TrueNAS this VM is called mover).

Code:

root@datacenter[~]# virsh list --all
 Id   Name      State
-------------------------
 7    1_mover   running

root@datacenter[~]#

5. Attach the disk(s) using the info gathered until now:
virsh attach-disk <VM Name> <Disk ID> <Target Name>

Here an example:
virsh attach-disk 1_mover /dev/disk/by-id/ata-Samsung_SSD_850_EVO_500GB_S21JNSAG158509B vdj


Remember the attach is not persistent: if you perform a reboot inside the Guest OS, everything is fine, but if you Power Off or Restart the VM from TrueNAS, the attached disks will disappear. Anyway, reattaching can be done following this procedure again.


One last notice: disks will be passed through using virtIo emulation. Going away from the technical details, it only means you MIGHT require additional drivers on your Guest OS to correctly identify and use the attached device. For Windows, you can find virtIo drivers here: https://github.com/virtio-win/virtio-win-pkg-scripts/blob/master/README.md



Hope this will help in the future.