Large Pool preventing reboot

[sulfursnow]

We have a very odd problem: -

AIC - SB405-VL server barebone
2x PM883 240Gb drives in mirror for system on the mainboard SATA controller
1x Broadcom - HBA-9500-8i 05-50134-01
256Gb of RAM
2 x 4216 Xeon Silver Scalable
97 x Western Digital Ultrastar DC HC560 0F38785 20 TB Hard Drive -
3 x PM1643A 3.8TB SSDs

11 x 8 RAIDz vdevs
1 x SSD in cahce
2 x SS in Log.

Whenever we provision the pool to be larger than 1Pib, on reboot the UEFI drive scan process hangs if the 2x 240Gb drives are present. The system SSDs not being present boot completes fine, and drives are visible on the HBA.

If we blank the data drive tables, the unit boots fine and the original truenas install is present with no issues except missing data.

If we add the SSD drives to the backplane the unit boots.

We're currently assuming that the HBA is at fault? But we've never seen no heard of anything like this before?

Does anyone have any ideas?

 

[HoneyBadger]

Hello @sulfursnow

I assume this is your system/server here described as the "AIC SB405-VL"?

PRODUCTS-OEM, ODM and COTS Server, Storage and Chassis Solutions - AIC

SB405-VL, a 4U 102-Bay Ultra density storage server, the built-in system MB supports dual Intel®

www.aicipc.com

I also assume you are running the latest release of TrueNAS CORE (13.0-U5.1) since you've posted this in the CORE subforum.

From the description, is it hanging up at the UEFI process prior to TrueNAS being loaded, or does it at least get to the bootloader before stalling somewhere during the FreeBSD boot? Eg: do you see /boot/loader.efi on the console output?

 

[sulfursnow]

Thats correct and yes I'm on the latest 13.0-U5.1

Its hanging before the truenas bootloader being loaded, The EFI settings are also locked out.

 

[NickF]

One thing to maybe try is to disable the BIOS setting for the "Option Rom" on the PCI-E slot that the 9500 is plugged into. This will prevent you form booting from any of the drives in the backplane all together, but that shouldn't be an issue for you.