FreeNAS on supermicro board with 2 HBA's

UserSN

Dabbler
Joined
Jul 23, 2020
Messages
41
Hello Community,

I'm testing a new system prior installing in production. During testing i'm running into some unexpected behavior and am trying to diagnose what could be the cause.

I've installed FreeNAS onto 2 SSD's in a mirror, in bios mode from a USB drive. After the installation completes, i remove my USB drive and the machine boots into FreeNAS fine. To test the mirroring of the boot drives, i then physically remove 1 of the 2 boot drives from the machine and turn it on. Once the machine posts it displays the message: "This is a FreeNAS data disk and can not boot system. System halted".

Next i reinstall the missing boot drive into the machine, restart and now it always says that same error message. My expectation was that FreeNAS would boot even if i remove (simulating a failed drive) 1 of the 2 boot drives.

Note* During the installation process i noticed 4 line items for my 2 drives. I suspect this is because i have 2 HBA cards connected to the same backplane for redundancy. Taking this a step further in the Avago HBA configuration utility i've ensured that both my cards have the correct primary and alternate boot drives specified. I did find it odd that during the install it displayed 4 drives instead of 2 but again i'm thinking the installer read my 2 drives, once from each HBA card separately (my best guess).

As a second test I tried to install FreeNAS selecting all 4 drives show on the install prompt (a error i believe) that displayed that failed.

What could be wrong? Post installation i was expecting to remove 1 drive and have FreeNAS boot up fine, and then swapping drives to ensure the reverse also works. In the FreeNAS web GUI i saw the mirror was healthy in the Boot Status.

Here is the hardware i'm using:
Supermicro 4U 24 Bay X10QBi 4x E7-4820 V2 2Ghz 32-Cores 128GB
-- Chassis 848XTS-R3240BP
-- Backplane: BPN-SAS3-846EL1 (https://www.supermicro.com/support/manuals/?mlg=0)
-- Motherboard: X10QBI (https://www.supermicro.com/en/products/motherboard/X10QBi)
2x HBA Cards (LSI SAS 9300-8i)
2x PNY SSD's (Boot Drives)
8x WD RED SSD's (SSD Storage Pool for speed)
6x WD RED HDD's (HD Storage Pool for capacity)

Any Ideas?
 

UserSN

Dabbler
Joined
Jul 23, 2020
Messages
41
Update:

I've since found that 1 of my HBA cards has a wrong configuration for BOOT Only instead of BOOT AND OS i've rectified that.
My next tests were the following:

1st: Reinstall FreeNAS on my 2 boot drives (SSDs) without any storage drives in the machine

2nd: Load into FreeNAS & verify my mirror boot installation, it was healthy.

3rd: Shut down the system & remove my primary boot drive then power-up: The machine booted fine and showed degraded Mirror pool for the boot drives (this is what i was expecting)

4th: Shut down the system & reinstalled primary boot drive, removed secondary boot drive and then power-up: The machine booted fine and showed degraded Mirror pool for the boot drives with primary back and secondary down (this is what i was expecting)

5th: Shut down the system & reinstalled both boot drives, powered-up: Machine boots fine shows both drives, boot pool is healthy.

6th: Reinstalled all my storage drives into the machine, 8SSD's for 1 pool, 6HDD's for a 2nd pool. The pools show up healthy everything is good.

7th: Shut down the system& removed 1st boot drive, powered on the machine: Machine posts, then displays the infamous"This is a FreeNAS data disk
and can not boot system. System halted"

8th: Shut down the system & reinstalled 1st boot drive, powered on the machine: Machine posts, then displays the same error message.

9th: Shut down the system, removed all storage drives: Machine posts, loads FreeNAS fine, everything in FreeNAS is okay except all storage pools are degraded (this is normal because the drives are not plugged in)

My expected outcome would be that if i were to remove the boot drive (either of them) from the machine with all my storage drives installed. The system would boot up fine, only if BOTH my boot drives were removed (simulating a failed drive) would i get this error message?

Not sure whats going on.

Possible Issue?
I recall on my 1st run through of the installation, i did install on one of the SSD's that is used as part of the storage pool. Since the SSD is 2TB i bought 2 new ones the PNY's to use for boot pool instead. Then wiped the SSD's in my storage pool when creating it through the FreeNAS UI. Could it be that there is still some part in that drive thats throwing off the system thinking it's a boot drive when it's not? I though the wiping of the drives when creating a pool completely erases everything?
 
Last edited:
Top