Stuck on "Welcome to GRUB!"

[Ashkaan]

Ever since the first, completely fresh install of TrueNAS SCALE 22.12-RC.1, I get this screen for about 20 minutes. Then, it boots just fine. Any ideas?

It's driving me nuts because I'm trying to troubleshoot another issue, and rebooting is a pain.

Thanks,

 

[WN1X]

We cannot help you without knowing what you are running (both hardware and software):

Forum Rules

 

[Ashkaan]

So, sorry! I thought my signature showed up on my posts:

Chassis: 4U Supermicro CSE-847BE1C-R1K28LPB
Motherboard: Supermicro X10DRi-T4+
Processor: 2x Intel Xeon E5-2683 V3 - 28 cores
Memory: 128 GB ECC memory
Boot: 1x WD Blue SSD SA510 500GB
HBA: 1x AOC-S3008L-L8e HBA 12Gb/s
Storage: 36x HGST SAS3 12Gbps 6TB
VDEVs: 4x RAIDZ2

 

[WN1X]

Is that HBA flashed for IT mode? Have you tried booting without the HBA to see if you get the delay?

 

[Ashkaan]

Is that HBA flashed for IT mode? Have you tried booting without the HBA to see if you get the delay?

Ooo, I'm afraid to since all my data is on there. I haven't tried that.

 

[WN1X]

You did not answer my first question....are you sure your HBA card is flashed for IT mode?

 

[Ashkaan]

are you sure your HBA card is flashed for IT mode?

Sorry, I don't know. I'm not sure how to check for that. Can I go into the HBA menu or do I need to flash it with new firmware that has that bit to be sure?

 

[Daisuke]

My cards, run command from terminal:

Code:

# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2308_2(D1)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2308_2(D1)   20.00.07.00    14.01.00.06    07.39.02.00     00:02:00:00
1  SAS2308_2(D1)   20.00.07.00    14.01.00.06    07.39.02.00     00:44:00:00

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

Or to show detailed information for a specific controller:

Code:

# sas2flash -list -c 0
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2308_2(D1)

    Controller Number              : 0
    Controller                     : SAS2308_2(D1)
    PCI Address                    : 00:02:00:00
    SAS Address                    : [removed]
    NVDATA Version (Default)       : 14.01.00.06
    NVDATA Version (Persistent)    : 14.01.00.06
    Firmware Product ID            : 0x2214 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9207-8i
    BIOS Version                   : 07.39.02.00
    UEFI BSD Version               : 07.27.01.01
    FCODE Version                  : N/A
    Board Name                     : SAS9207-8i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

 

[Ashkaan]

Ok, I get this:

Code:

root@TrueNAS[~]# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    No LSI SAS adapters found! Limited Command Set Available!
    ERROR: Command Not allowed without an adapter!
    ERROR: Couldn't Create Command -listall
    Exiting Program.

and:

Code:

root@TrueNAS[~]# sas2flash -list -c 0
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    No LSI SAS adapters found! Limited Command Set Available!
    ERROR: Command Not allowed without an adapter!
    ERROR: Couldn't Create Command -list
    Exiting Program.

I'm guess it's because I have a different brand than you?

 

[Daisuke]

That means your installed card is not flashed with LSI firmware in IT mode or you are not using an LSI card.

I see you use AOC-S3008L-L8e card, that is already flashed to IT mode, so that should not be a blocker. If you remove the card, can you boot into OS? Also, where did you installed the OS? Can you install it on an SSD connected to a preferably internal USB port?

 

[Ashkaan]

It's a little challenging to stop production to pull the card, but I think you're right that it's a good next troubleshooting step.

The OS boots after 20m of being stuck on that screen. My goal is to fix the delay.

 

[Daisuke]

I saw another thread exhibiting the same behaviour in Scale, stuck in boot for 30min. Can you search for it? I thought you are stuck forever on GRUB.

 

[Ashkaan]

I saw another thread exhibiting the same behaviour in Scale, stuck in boot for 30min. Can you search for it? I thought you are stuck forever on GRUB.

Thanks for the lead! After about an hour of searching, this was the only post I could find. Their solution was to disable the serial console and sadly mine was already disabled.

 

[Daisuke]

Where is your OS booting from, internal disk, USB disk? If you go to BIOS and choose to boot from a specific disk (Scale OS), do you have the same problem? Does your server has any other OS installed, beside Scale?

 

[Ashkaan]

It's an internal SSD plugged into the motherboard's Intel ACHI controller. It's the only storage device plugged into the motherboard directly.

Yes, if I override boot the OS (EFI), it exhibits the exact same problem. In fact, I went so far as to delete from the list every other possible boot location just in case.

 

[Daisuke]

Can you make me a big favour and plug an SSD into a (preferably internal) USB port and see if you can boot fine? I know it does not makes sense, but we try to isolate the problem. You should not experience this issue, since you were running fine on Scale 22.02.4, is worth opening a ticket with IX and provide them the debug file. I run Scale on a Samsung SSD connected to an internal USB port, in my Dell R720xd.

 

[Ashkaan]

Can you make me a big favour

LOL.. you're doing me the favor by giving me troubleshooting advice!

plug an SSD into a (preferably internal) USB port

Ok, let me figure out when I can take this guy down (so many things dependent on it).

 

[Daisuke]

The part which does not makes any sense is: you don't have this issue in Angelfish. How is this possible, you are at GRUB, a program for loading and managing the boot process and for some reason, it does not see any distribution available, for 30min. Please open a support ticket and provide your debug file. Link that ticket number here, I'm curious to see the resolution. On Dec 15, Scale Bluefin will be released, there is a discussion thread for upgrade issues, I will post that ticket number into thread, if you have the same issues with final release.

 

[Ashkaan]

Oh no, I never mentioned Angelfish. I've never used TrueNAS before this experience. This is a fresh, and my first, install.

However, the USB installer didn't have any issues booting.

 

[Daisuke]

Oh no, I never mentioned Angelfish.

My bad, then we should definitely try to boot from a different disk, can you try an USB stick for a quick test? This should allow you to skip playing with the internal components. From BIOS you should be able to pick the USB stick.

I use every single available server slot for disks, I can only afford to boot from internal USB SSD, installed into a PCIe slot bracket. If it breaks, I can replace it in 2 seconds and restore all settings from backup. But from what you say, this looks like is a company production appliance, so you might not be inclined to follow the same logic.