Truenas on XCP-NG SATA controller passthrough

[impovich]

Hi all, trying to install Truenas Core 12 on xcp-ng 8.2.
Passed through SATA controller to a VM, installer detected hard drives, installed Truenas and after reboot, I'm getting "no bootable device".
What I did:

Code:

[10:56 chicago ~]# lspci
00:00.0 Host bridge: Intel Corporation Device 3e33 (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.4 Communication controller: Intel Corporation Device a364 (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #22 (rev f0)
00:1b.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #23 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
00:1f.0 ISA bridge: Intel Corporation Device a30a (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

Code:

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(00:17.0)"

Code:

[11:08 chicago ~]# xe vm-list
uuid ( RO)           : 7d16b32f-fe80-4271-8543-68c5a642220e
     name-label ( RW): Control domain on host: chicago
    power-state ( RO): running


uuid ( RO)           : adfa344c-d136-885f-8396-4f332ad40770
     name-label ( RW): Truenas
    power-state ( RO): halted

Code:

[11:25 chicago ~]# xl pci-assignable-list
0000:00:17.0

Code:

 xe vm-param-set other-config:pci=0/0000:00:17.0 uuid=adfa344c-d136-885f-8396-4f332ad40770

Code:

[11:27 chicago ~]# xe host-dmesg | grep VT-d
(XEN) [    2.053436] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB
(XEN) [    2.066660] Intel VT-d Snoop Control enabled.
(XEN) [    2.079753] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    2.092849] Intel VT-d Queued Invalidation enabled.
(XEN) [    2.105873] Intel VT-d Interrupt Remapping enabled.
(XEN) [    2.118799] Intel VT-d Posted Interrupt not enabled.
(XEN) [    2.131762] Intel VT-d Shared EPT tables enabled.

Also, I tried to install Truenas on virtual HDD which is M.2 SSD, and passthrough the SATA controller, ended up having two ada0 devices(ada0 - virtual HDD and ada0 HDD connected to SATA controller which is passed through)

What am I doing wrong?

 

[sretalla]

I tried to install Truenas on virtual HDD which is M.2 SSD, and passthrough the SATA controller, ended up having two ada0 devices(ada0 - virtual HDD and ada0 HDD connected to SATA controller which is passed through)

You're going in the right direction there, but you need to resolve that duplication of naming in order to succeed.

I'm not familiar enough with XCP to tell you how to get that to be different, but you might look into options to assign the virtual disk to a SCSI controller instead of SATA...

 

[impovich]

You're going in the right direction there, but you need to resolve that duplication of naming in order to succeed.

I'm not familiar enough with XCP to tell you how to get that to be different, but you might look into options to assign the virtual disk to a SCSI controller instead of SATA...

I want to build on top of https://www.supermicro.com/en/products/system/1U/5019/SYS-5019C-FL.cfm, so there is no choice. PCIe already occupied with 10Gbe NIC.

Currently, I decided to pass through separate HDD drives, and it works, will test it out.

 

[impovich]

Found the solution:

SOLVED - multiple "ada0" partitions on Xen PV w/passthrough

Hi, I've been using FreeNAS for several months now, in a few different configurations - some production, some educational. As disclosed in the title, this particular issue was encountered on a Xen PV platform - nevertheless I believe this may be a storage issue rather than being a consequence...

www.truenas.com

set via GUI tunables interface:
hint.ada.0.at="scbus100"