TrueNAS-13.0-U6.1 fails to boot from a cold start with a message about being Out of chain frames

[dak180]

Specifically mpr0: Out of chain frames, consider increasing hw.mpr.max_chains. it then gets into a loop reseting the controller. Breaking it out of this requires pulling the cables from the HBA to the expander (this disconnects all the sata drives except the boot pool), reseting the server, increasing hw.mpr.max_chains (now at 16384), reconnecting the expander and rebooting. After all of this it will boot up normally. Does anyone have any ideas as to what might be happening or even what to test for? (@jgreco I thought I saw a thread you were in with something similar.)

dev.mpr.0.io_cmds_highwater: 66
dev.mpr.0.prp_page_alloc_fail: 0

 

[Ericloewe]

What's your SAS topology, exactly? This one is best illustrated with a diagram... And is the controller a Dell H330 crossflashed to HBA330 firmware?

 

[dak180]

And is the controller a Dell H330 crossflashed to HBA330 firmware?

Yes.

What's your SAS topology, exactly?

Actually, very simple: both ports from the HBA are connected to the Adaptec expander which then has three breakout lines attached to it, one of which goes to the 4 sata ssds in the jail pool, the other two go to the 7 8tb drives in the main pool (the 12tb one is plugged into a motherboard sata port, the hba does not like it and I have not had time to figure that one out).

 

[Ericloewe]

Adaptec expander? Don't see many of those. A couple of things to try out:

• Single cable instead of two
• Are there any dedicated uplink ports on Adaptec expanders?

 

[dak180]

Single cable instead of two

Already tried; did not change anything.

Are there any dedicated uplink ports on Adaptec expanders?

Yes, 3: 1 internal and the 2 external (presently unused), and that internal port (and the one next to it) are the ones used to link to the HBA.

 

[Ericloewe]

Latest firmware on the HBA and the expander?

 

[dak180]

Latest firmware on the HBA and the expander?

Code:

    Adapter Selected is a Avago SAS: SAS3008(C0)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS3008(C0)  16.00.11.00    0e.01.00.39    08.37.02.00     00:08:00:00

Not sure how to get the expander's info on freebsd.

 

[Ericloewe]

Yeah, not sure an application is even available, unfortunately. Even the LSI expanders are a bit iffy.

The HBA firmware is not quite the latest, though, 16.00.12 is available from iX. Can't hurt to try.

 

[dak180]

The HBA firmware is not quite the latest, though, 16.00.12 is available from iX. Can't hurt to try.

Code:

# sas3flash -o -f ./SAS9300_8i_IT.bin
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    Advanced Mode Set

    Adapter Selected is a Avago SAS: SAS3008(C0)

    Executing Operation: Flash Firmware Image

        Firmware Image has a Valid Checksum.
        Firmware Version 16.00.12.00
        Firmware Image compatible with Controller.

        Valid NVDATA Image found.
        NVDATA Major Version 0e.01
        Checking for a compatible NVData image...

        NVDATA Device ID and Chip Revision match verified.
        NVDATA Versions Compatible.
        Valid Initialization Image verified.
        Valid BootLoader Image verified.

        Beginning Firmware Download...
        Firmware Download Failed!

    Firmware Returned Exception. IOCStatus=0x4, IOCLogInfo=0x30010408
    Due to error remaining commands will not be executed.
    Unable to Process Commands.
    Exiting SAS3Flash.

 

[Ericloewe]

Are you trying to do that in TrueNAS? That's not a good idea with a running pool, it's best to do it from the UEFI shell.

 

[dak180]

Are you trying to do that in TrueNAS? That's not a good idea with a running pool, it's best to do it from the UEFI shell.

That was me trying to do that in TrueNAS with only the boot pool plugged into the motherboard (no other pools mounted) and the HBA unplugged from the expander entirely.

 

[Ericloewe]

OK, so that resolves the danger part. Still, the error is weird and atypical.

 

[dak180]

Still, the error is weird and atypical.

This is the story of my whole life. Forget the zebras, I get unicorns and pegasi. Any suggestions?

 

[Ericloewe]

Short of replacing the expander, the long-shot option is still to try the firmware update, but from the UEFI shell.