TrueNAS-12.0-U1.1 installation stuck at mps0: <Avago Technologies (LSI) SAS2008> at device 0.0 at pci1

[itreta]

Hi,

I started with installing TrueNAS-12.0-U1.1 in a HyperV VM and that went fine. Next I was looking to attach a set of disks to create a ZFS vdev/pool. I assigned the HBA (LSI 2008, flashed in IT mode with P20 firmware) to the VM, and started a new installation and the boot gets stuck at the below line:
mps0: <Avago Technologies (LSI) SAS2008> at device 0.0 at pci1
While stuck on this, the HyperV console shows that VM is using 9% of CPU (stuck in some loop) ?

Here are the commands I use for DDA assignment to the VM

Code:

Set-VM -VMName TrueNAS -LowMemoryMappedIoSpace 3Gb -HighMemoryMappedIoSpace 33280Mb -GuestControlledCacheTypes $true -AutomaticStopAction TurnOff
Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(C0)#PCI(0101)#PCI(0000)" -Force
Add-VMAssignableDevice -LocationPath "PCIROOT(C0)#PCI(0101)#PCI(0000)" -VmName "TrueNAS"

I have tried with both a Gen1 and Gen2 VM and it gets stuck at this same place. There is currently one HDD (WD SATA 14TB) attached to the controller.

I tried installing ubuntu-20.04.1-desktop-amd64 in another VM (and assigned the the HBA to ubuntu VM) and HBA is detected correctly (both in setup and after installation), the filesystem on the disk can be mounted and accessed.

I also tried using FreeNAS-11.3-U5.iso, with the same results.

LSI card BIOS has been set with defaults. I've also tried with the P14 /IR mode (which the device came in before i flashed it) and it showed the same behavior.

Code:

E:\>sas2flash.exe -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2008(B2)

        Controller Number              : 0
        Controller                     : SAS2008(B2)
        PCI Address                    : 00:c1:00:00
        SAS Address                    : 500605b-0-079d-e560
        NVDATA Version (Default)       : 14.01.00.06
        NVDATA Version (Persistent)    : 14.01.00.06
        Firmware Product ID            : 0x2213 (IT)
        Firmware Version               : 20.00.07.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9210-8i
        BIOS Version                   : 07.39.02.00
        UEFI BSD Version               : 07.27.01.01
        FCODE Version                  : N/A
        Board Name                     : SAS9210-8i
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

The hardware specs are: AMD EPYC 3202, ASRock Rack ROMED8-2T, 256 GB ECC RAM, LSI SAS2008 HBA, Samsung 980 Pro SSD (m2)
The VM specs are: Gen 2 VM, Secure Boot Disabled, 8GB RAM, 4 Virtual Processor, 128 GB VHDX for OS install, Virtual NIC, All Integration Services offered, No Checkpoints

Any suggestions on troubleshooting why adding HBA to the VM halts the installation ?

<disclaimer: this is my first look at FreeNAS/TrueNAS so pardon if i've overlooked anything>

 

[itreta]

Update:
Changed the DDA command to not include LowMemoryMappedIoSpace / HighMemoryMappedIoSpace / GuestControlledCacheTypes parameters

Tested (all Gen2 VMs)
Windows 10 Version 2004 - card was detected in the guest post install (didn't try to provide drivers during install)
FreeBSD-12.2-RELEASE-amd64-dvd1.iso - same issue with setup stuck on mps0 message
FreeBSD-13.0-ALPHA3-amd64-20210129-40cb0344eb2-256214-disc1 - same issue with setup stuck on mps0 message

 

[itreta]

Further Testing

TrueNAS-SCALE-20.12-ALPHA - card was detected during install and HDD was available for pool creation, which was successful as well.
FreeNAS-11.0-RELEASE.iso - install boot was able to proceed to menu as mps was not loaded ? the card was not visible during install
FreeNAS-11.3-U3 to U5 - same issue with setup stuck on mps0 message

Enabled verbose message for boot option during install of TrueNAS-12.0-U1.1 and that gives some additional spew

 

[itreta]

and more testing

Got my hands on a LSI SAS 9300-8i. Updated to latest FW (SAS3_FW_Phase16.0-16.00.12.00_Firmware), BIOS and is in IT mode.

ESXi 6.7, Patch Release ESXi670-201912001 as hypervisor. Was able to boot and successfully passthrough the card to the guest.
ESXi 7.0, Patch Release ESXi 7.0b as hypervisor. Was able to boot and successfully passthrough the card to the guest.
HyperV on Server 2022 Insider Build 21301. TrueNAS / FreeBSD gets stuck during install (when HBA is present via DDA)

verbose boot spew

 

[itreta]

Postponing the project to set up TrueNAS with ZFS till my second server (which has an Intel CPU) becomes operational.

For this server, I decided to go ahead with Hyper-V and Storage Spaces. Dropped the HBA card as well, since the onboard ports are enough and no pass-through is needed.

7 * 14 TB (WD) with dual parity for a 89.4 TB RAW (and 58.9 TB usable) disk space. GPT Disk, simple volume on the virtual disk, formatted as NTFS (64K).

In my testing, getting a transfer speed between 350 to 500 MB/s while copying 170GB of virtual machine disks and ISOs

Code:

$PhysicalDisks = (Get-PhysicalDisk | ? {$_.CanPool -eq "True"})
New-StoragePool -FriendlyName <PoolName> -StorageSubsystemFriendlyName "Windows Storage*" -PhysicalDisks $PhysicalDisks
New-VirtualDisk -StoragePoolFriendlyName <PoolName> -FriendlyName <VirtualDiskName> -UseMaximumSize -ResiliencySettingName Parity -PhysicalDiskRedundancy 2 -WriteCacheSize 200GB

Executing the below two commands before creating the virtual disk (last step above) did not make much difference to transfer speed in my testing

Code:

Set-StoragePool -FriendlyName SSPOOL -IsPowerProtected $true
Set-PhysicalDisk -FriendlyName "Samsung SSD 850 PRO 256G" -Usage Journal