XCP-ng 7.5.0 PCI pass through

melloa

Wizard
Joined
May 22, 2016
Messages
1,749
If someone is thinking about trying XCP-ng as a hypervisor and run FreeNAS or FreeBSD, here's some quick PCI pass through commands I've tested and worked:

Assuming XCP-ng is installed and the VM is created, the first step is to find out the addresses for your PCI interfaces. The way to do that is with lspci from the hypervisor shell.
Code:
[root@xcp-lab ~]# lspci
00:00.0 Host bridge: Intel Corporation 5520/5500/X58 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22)

...

Step 1 - Find the PCI interface address

You can be more specific using grep and the manufacture for your device. For instance, if you want to get the information for your LSI HBA:

lspci | grep LSI
Output:
Code:
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02

Make a note of the interface address, 04:00.0 in the case above.

Step 2 - Exclude the interface from dom0

With that interface address handy, you can change XCP configuration using the following command: /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=<PCI Device Address>"

In this example:

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(04:00.0)"


Step 3 - Reboot is needed to ensure that device is no longer in use by XCP.


Step 4 - Obtain the VM UUID

From the shell command: xe vm-list

Output:
Code:
uuid ( RO)		 : 827ba682-920e-40b2-9e7c-2860b68aaa76
		name-label ( RW): Control domain on host: xcp-lab
		power-state ( RO): running

uuid ( RO)		 : bfc9ea79-431c-ebb2-6e90-051910d333b4
		 name-label ( RW): FreeNAS11
		 power-state ( RO): running

Make a note of the VM UID, bfc9ea79-431c-ebb2-6e90-051910d333b4, in this example.

Step 5 - Add the passed through PCI interface to the VM

Use the following command:

Code:
xe vm-param-set other-config:pci=0/0000:<B:D.f> uuid=<vm uuid>

Where:
<B:D.f> is the PCI Device ID (04:00.0 in this example)


xe vm-param-set other-config:pci=0/0000:04:00.0 uuid=bfc9ea79-431c-ebb2-6e90-051910d333b4


After those five steps you will be able to start your VM and use ZFS to manager your pool.

In case you have more than one HBA you'd like to assign to a VM, you will get the information for both HBAs and the command for step 5 will look like:

xe vm-param-set other-config:pci=0/0000:01:00.0,0/0000:04:00.0 uuid=88c901ac-b98b-e576-9999-2671907dc298

In the case above two PCI interfaces with addressed 01:00.0 and 04:00.0 are been assigned to the VM. That was tested with one HBA and one Chelsio T320:
Code:
[root@xcp-lab ~]# lspci | grep Che
01:00.0 Ethernet controller: Chelsio Communications Inc T320 10GbE Dual Port Adapter

 

Barlog951

Dabbler
Joined
Nov 29, 2016
Messages
20
Hi @melloa thanks for the article

Can I do similar procedure for onboard SATA controler ?
What if I don't install VM Freenas yet, should I do step 4 and 5 after I install Freenas?

Thanks for your answer :)
 

acp

Explorer
Joined
Mar 25, 2013
Messages
71
You will find doing pci pass through on motherboard devices more trouble than on devices plugged into a pcie slot. Sometimes it simply doesn't work for reasons I never been able to explain. I never tried sata but I tried USB and never could get it will work. Bought a cheap card and it worked like a champ.

As to the steps, you can create the VM but don't start it. That way you can get the uuid to assign the hardware to. Once assigned, start and do the install. Monitor the messages log in /var/log (if I remember correctly) in dom0. If the pci assigned is successful or not it will be logged.

Reminder all of this is command line, xcp-ng command center can't do this. I think you can do this in XO, but never tried there.

Good luck.
 

Barlog951

Dabbler
Joined
Nov 29, 2016
Messages
20
I need min 9 pass through SATA ports and 1 for xcp-ng , so this will be probably very hard :/
 

silverback

Contributor
Joined
Jun 26, 2016
Messages
134
If someone is thinking about trying XCP-ng as a hypervisor and run FreeNAS or FreeBSD, here's some quick PCI pass through commands I've tested and work.

Thanks, as I am looking for alternatives to ESXi free.
The 8 core limit is understandable, but frustrating.
 

His.Dudeness

Dabbler
Joined
May 14, 2019
Messages
15
Hi,
my mainboard got two SATA controllers. One 2-Port-SATA by ASMedia and one 6-Port-SATA by AMD. I would like to passthrough the 6-Port-SATA controller, but it seems to consist of 2 different PCI devices.

# lspci | grep -i sata
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
25:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
2c:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

Any Idea which of the two devices I should try to passthrough? Do I need both? Why are there two of them? o_O

cheers
Michael
 

amp88

Explorer
Joined
May 23, 2019
Messages
56
Any Idea which of the two devices I should try to passthrough? Do I need both? Why are there two of them? o_O
I'm not 100% sure, but I believe the reason there are two AMD SATA controllers listed is because 1 of them (the FCH one) will be used if you put a SATA-based M.2 drive in the motherboard. It's pretty common for boards to have one M.2 slot which supports either PCIe NVME or SATA M.2 SSDs. I don't know how you can easily test this in your case, but my belief is that the "03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)" entry will be for the 6-port controller and the "2c:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)" will be for the M.2 slot.
 
Top