SOLVED TrueNAS-SCALE-21.08-BETA.1 hang on install with nvme disks

[Diff]

Trying to do a clean install of TrueNAS-SCALE-21.08-BETA.1 on PowerEdge Dell R7515, spec:

Code:

Dell R7515
- AMD EPYC 7302P 16-Core Processor
- 32GB ECC DDR4
- PERC H730P in HBA mode
  - 3 x WD HC550 18GB SATA
- BOSS-S1 2 x M.2 SATA
  - 2 x Micron 1300 M.2 SATA
- Supermicro AOC-SLG3-2M2 2 x M.2 NVMe
  - XPG SX8200 512GB M.2 NVMe
  - Sabrent Rocket 512GB M.2 NVMe
- 10Gtek NV95NF HBA card, PCIe x16 4 x M2 NVMe
  - 1 x XPG SX8200 M.2 1TB NVMe
  - 3 x Sabrent Rocket 1TB M.2 NVMe
Boot: UEFI

Install just hangs right after I select Install TrueNAS option with black screen.

In same time I am able to install:
- Windows 10
- Fedora server 34
both of these can see all disks fine.

For example, here is output from Fedora 34

Code:

[root@fedora ~]# lsblk -a -o KNAME,TYPE,SIZE,MODEL,ALIGNMENT,HCTL,WWN,VENDOR |grep -iE "disk|type"
KNAME     TYPE   SIZE MODEL               ALIGNMENT HCTL       WWN                                                                        VENDOR
sda       disk 238.5G MTFDDAV256TDL               0 0:0:0:0    0x500a075124f20e9e                                                         ATA
sdb       disk 238.5G MTFDDAV256TDL               0 1:0:0:0    0x500a075124f809cc                                                         ATA
sdc       disk  16.4T WDC_WUH721818ALE6L4         0 12:0:0:0   0x5000cca2c2c88971                                                         ATA
sdd       disk  16.4T WDC_WUH721818ALE6L4         0 12:0:1:0   0x5000cca2afe15ed9                                                         ATA
sde       disk  16.4T WDC_WUH721818ALE6L4         0 12:0:2:0   0x5000cca284d80d5e                                                         ATA
nvme0n1   disk 476.9G Sabrent                     0            eui.6479a72fb1902fbd                                                 
nvme3n1   disk 953.9G Sabrent                     0            eui.6479a72ef0011581                                                 
nvme4n1   disk 953.9G Sabrent                     0            eui.6479a749205005fd                                                 
nvme5n1   disk 953.9G Sabrent                     0            eui.6479a7492050066b                                                 
nvme2n1   disk 953.9G ADATA SX8200PNP             0            nvme.1cc1-324a35303230303238343032-414441544120535838323030504e50-00000001
nvme1n1   disk 476.9G ADATA SX8200PNP             0            nvme.1cc1-324b30373230303733393734-414441544120535838323030504e50-00000001 

FreeBSD based TrueNAS-12.0-U5.1 seems like NOT being able to see Micron 1300 M.2 SSDs (on Dell BOSS-S1 PICe adapter).

So seems like I am stuck on not being able to use TrueNAS Core, because it cannot see drives on BOSS-S1, which I was planning to use for BOOT mirror, but TrueNAS Scale hang completely (I think after me installing 10Gtek NV95NF HBA card, PCIe x16 4 x M2 NVMe with 4 x 1TB NVMe drives).

Reported as Jira:

https://jira.ixsystems.com/browse/NAS-112585

UPDATE1:
- Verified all BIOS and firmware versions, for Dell, BOSS-S1, all NVMe drives, everything up to date.

UPDATE2:
- downgrade Dell BOSS-S1 firmware to A03 2.5.13.3011 (https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=rpd7y), seems to resolve the problem for TrueNAS Scale (Debian based) on TrueNAS Core (FreeBSD based) had really hard time with drivers for Dell BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller.

Any advice from more experienced TrueNAS users?