Stuck at Welcome to Grub!

[Chad Brown]

So I have recently built myself a VMWare ESXi Server (ESXi v6.5) and it has been working flawlessly for a few weeks, but the day before yesterday I upgraded from FreeNAS 9.10 to FreeNAS 11-RC3

For some reason it refuses to boot past the Welcome to Grub message but only on the second boot.

The sequence of events is as follows:

First - Upgrade.
1.) From within the UI I changed my train to 11-RC3
2.) The upgrade procedure ran, and the system rebooted.
4.) The next day the VM server went through it's weekly maintenance cycle shutting down each VM and backing up the images to an external source.
5.) The server booted back up, and I found it stuck on the "Welcome to Grub!" message after that.


I have tried a few suggestions for updating the VMs BIOS (Turning off floppy drives, changing boot settings to make sure the hard drive is first.)

Second - Manual Upgrade.
1.) I downloaded the FreeNAS 11-RC3 ISO image.
2.) I booted from the ISO image and did a manual upgrade.
3.) Rebooting the server it booted up just fine, and was running without issue.
4.) Reboot the server after uploading the backup of my configuration.
5.) The server stopped at the "Welcome to Grub!" message.

It doesn't have any errors or anything I can find, and previous to v11-RC3 it would boot just fine. If I leave it sitting on the "Welcome to Grub!" message for a few minutes (about 5) it will go into a boot loop where it reboots over and over and over again.

Third - Clean install
1.) Deleting the VM I went for a clean install of the FreeNAS 11-RC3 on clean hard drives without any prior setup.
2.) Installed the FreeNAS system.
3.) Reboot and the system boots up just fine.
4.) Reboot and it does the same thing. Stopping on the Welcome to Grub! message and going into a boot loop.

System Configuration
VMWare ESXi v6.5 (Build 5310538)

Hardware
- Motherboard: SuperMicro - H8SGL
- CPU: AMD Operton(tm) Processor 6376
- RAM: 128 GB (DDR3 Registered ECC)
- Network: 2 Port Intel 82571EB NIC
- HD: SanDisk 120GB SSD

VM Configuration
- RAM: 64 GB
- CPU: 6 Cores
- HDs: 2 x 8GB VM Drives

I've researched as much as I can, but none of the solutions that are out there have proved to be successful.

Thanks in advance!

 

[joeschmuck]

I have no issue running FreeNAS 11-RC3 on ESXi 6.5, this is a test setup for me so I have no pass-through devices on this setup. I'm running VM Version 13, 8GB RAM, 8GB Boot Device VM drive, 20GB Data Storage VM drive, E1000 NIC, and everything else default. Again, this is just a testing setup. I won't upgrade one of my other systems until FreeNAS 11 Release is out.

I suspect it has something to do with your VMX file setup. Why don't you provide some details on how you setup your VM and did you use the BIOS setting during the FreeNAS install or UEFI? I used BIOS. Maybe you could drop a copy of your VMX file here in code tags.

If I don't get back to you right away, I'm headed into town soon to spend time with the family. Someone else may also jump in and offer advice.

Good Luck

 

[awe9]

Yesterday, I upgraded using UI from FreeNAS 11.0 RC2 to RC3 and encountered the same issue. I'm running FreeNAS on bare hardware--no virtualization or VMs.

It rebooted after upgrade and got stuck in a boot loop. "Welcome to Grub" shows for a second, and then system immediately reboots. Restored an RC2 boot environment for now.

Upgrade path: Clean install of FreeNAS 11.0 RC1 -> Upgrade to RC2 -> Upgrade to RC3 (fail).

Diffing /boot/grub/grub.cfg entries of RC2 and RC3 showed only a minor difference: a few integer values to kernel options were unquoted in RC3. E.g.,

- set kFreeBSD.kern.cam.ctl.ha_id="0"
+ set kFreeBSD.kern.cam.ctl.ha_id=0

I didn't manually edit grub.cfg to test it, of course, but I doubt quotes make much of a difference. It reboots immediately, so, can't be anything in the user space.

Hardware:
- Motherboard: SuperMicro - X10SL7-F
- CPU: Intel Core i3 4150
- RAM: 32GB ECC UDIMM

 

[Chad Brown]

I used BIOS, and I have gone back to 9.10 and it still boots just fine.

Um lets see.
So the setup process was normal.

I mean I'm not sure exactly which details you are looking for here, and to provide the VMX I would have to reinstall the v11-RC3 to get you the VMX file as I had to get it back into working order so I restored my 9.10 setup. I can do the reinstall tomorrow.

 

[Stux]

Sounds like one of you may need to make a bug report.

http://bugs.freenas.org

 

[joeschmuck]

@Chad Brown
The VMX file should be the same unless you created a new VM. If you are unaware what the VMX file is then you need to read up on ESXi.

Sounds like one of you may need to make a bug report.

http://bugs.freenas.org

I agree.

EDIT: In the meantime you two could try the Nightlies Train to see if the problem has been fixed. Every day there is a new build that incorporates fixes.

 

[deed02392]

I am having exactly the same issue.

Clean install of FreeNAS 11 RELEASE.

After getting stuck in a boot loop I tried installing FreeNAS 11 over the top of the previous install. This didn't work, and after rebooting at the end of the installation, I got "free magic failed" which appears to be a grub error message.

Just installed FreeNAS RC2 fresh, as a user above here didn't have issues with it. But for me I get a GRUB bootloop too.

Now I've installed 9.3.1 and that is working and rebooting fine. FYI.

VMX
----
Boot via BIOS
3x VMXNET 3 adapters
CD-ROM
PCI Passthrough
24 GiB RAM (reserve all)
Eagerly zeroed VMDKs LSI Parallel

ESXi 6.5
X10SRH-CLN4F
64 GiB SK Hynix ECC RDIMM
2x 8 GiB VMDK (zfs mirror)
LSI 3008 IT (w/ 3x 3 Tb disks) pass-thru

 

[norbs]

Same issue here... BIOS VM using PCI Passthrough.

 

[norbs]

Weird issue...

So I ran an upgrade from the ISO with all my PCIE device disconencted. After the reboot it seemed to have worked. I added PCIE devices back and upgraded pool.

I reboot. Same issue...

Ran the upgrader from the ISO again, this time with PCIE devices remaining connected and it seemed to have worked. At least for now... I'm honestly scared to reboot the VM.

 

[ovearj]

I have the same problem. Any progess here?

 

[norbs]

So to at least recover from this you need to boot from the ISO and install over the same install disks. You can just do an update mode and it will backup the database so you don't lose your settings. I've literally had to do this maybe 8 times now. I've changed little things here and there to try and figure out the source of the issue.

My latest change was to move from dual boot disk to only a single disk. It seems to be working but I can't say 100% since this probably is kind of intermittent. Sometimes a reboot causes it other time just a system update.

Would be interesting to hear the specs of your system to see what we have in common so that we can determine the root cause.

 

[Mirfster]

FWIW, just saw this when testing out v11. Issue was noticed when using a VM Version 11 (I am currently running ESXi 6 U2 I think). However, my existing VMs are running VM Version 8. So, I deleted the freshly created test VM and remade it as VM Version 8 and everything is running fine.


Out of curiosity what VM Version are you all using?

 

[norbs]

FWIW, just saw this when testing out v11. Issue was noticed when using a VM Version 11 (I am currently running ESXi 6 U2 I think). However, my existing VMs are running VM Version 8. So, I deleted the freshly created test VM and remade it as VM Version 8 and everything is running fine.


Out of curiosity what VM Version are you all using?

Looks like I'm VM Version 11. For me what seems to be working right now is simply having only one boot disk. Before running into this issue I had 2 on different datastores.

 

[Mirfster]

Looks like I'm VM Version 11. For me what seems to be working right now is simply having only one boot disk. Before running into this issue I had 2 on different datastores.

Thanks for the feedback. Have not tested with running just one disk, but if that works for you (and you have regular Config backups) you should be fine. I like your idea of two disks on different datastores but doubt I will get around to testing that.

For me I don't have much need of VM Version 11 and have been very content with 8 for most of my machines. If I ever get to the point of needing more than 1011 GB of RAM for a VM I'll think about it. ;)

May mess around more later and see if there is another possible culprit; but we'll see how much time I have for that.

 

[Ryan Winkler]

I am not running it in a VM and had the same issue.

I installed 11-stable then used the web interface to upgrade to U1 and then U2 without any issues.
Ran 11-U2 for some time with many many reboots and had no issues.

Used the web again to upgrade to 11-U3 and it didn't upgrade successfully.
When the system rebooted I didn't have any network.
I rebooted and selected the 11-U2 and the system rebooted into 11-U2 with network and ran again without issue.
Using the Web I deleted the first attempt 11-U3 upgrade then tried the upgrade again this time with success.
The system rebooted and came up running 11-U3 and appeared to be working fine.
The system seamed to be working fine running 11-U3 until it rebooted again and now it is stuck doing the "Starting Grub" and immediately reboots the computer, I don't have the option to boot into the 11-U2 this time.

Guess I should have taken the upgrade issue as a sign not to upgrade to 11-U3.

At this point I decided that I would boot from CD and try an "Upgrade" from CD.
I downloaded the ISO, burned it to Disc and booted from the 11-U3 install/upgrade CD.
I selected the "Upgrade" option and selected the "Install into it's own boot area" (or something like that, it only had the two options and the other option was to format the USB so..... I didn't want to format the USB).
The system completed the upgrade with some interesting messages around "DeprecationWarning: invalid escape sequence" and messages after the Grub message it stating something about being unable to decrypt ....
The message went by so fast I was not able to read it.

So far I have only rebooted the system a few times but it does boot up now.

Now when I look at the boot options it shows that I have another "default" but has todays date.
The full list now shows:
default
Initial-Install
11.0-U1
11.0-U2
11.0-U3
default-20170913-124714

My system seams to be working again.
I only have two Jails setup and they both appear to be working as well.

Only time will tell if another reboot will again cause the system to go into the Grub boot loop.

 

[Mirfster]

Yeah, I am starting to think that there is an issue with 11.0-U3. I have done a little more testing and am avoiding updating directly to 11.0-U3 from within the OS (Web Interface). Seems to me like it may default to wiping the boot drives and any existing boot options.

However, if I mount the ISO (in my VM) and direct it to NOT wipe the other boot options (the "Install into it's own boot area") things seem to work fine to get up to 11.0-U3. *** Note: This is pertaining to an Upgrade Only.

Luckily on my existing Main FN Server I only have upgraded from 9.10.2-U5 to 11.0-U1 and will not go higher at this time.

Lastly, even installing 11.0-U3 on clean drives (Using either 1 or 2 drives in a VM) is kind of "Hit or Miss" and IMHO not trustworthy. May seem all fine and dandy, then out of nowhere the "Grub Loop" occurs.

 

[Inxsible]

I already updated from 11.0-U2 to 11.0-U3 and haven't had any problems yet. IIRC, I did reboot it at least once.

But just to be safe, I will backup my config in case this Grub Loop comes back to bite my ass !!

 

[Mirfster]

I already updated from 11.0-U2 to 11.0-U3 and haven't had any problems yet. IIRC, I did reboot it at least once.

But just to be safe, I will backup my config in case this Grub Loop comes back to bite my ass !!

Out of curiosity, what Boot Environments do you have listed under [System] - [Boot]?

 

[Inxsible]

I am not at home, but if I remember correctly, I have 3. default, Initial Install and the 11.0-U2.

I can put up a screenshot later tonight.

 

[Mirfster]

I am not at home, but if I remember correctly, I have 3. default, Initial Install and the 11.0-U2.

I can put up a screenshot later tonight.

Thx, maybe if the existing installation is 11.0-Ux then it defaults to NOT wiping prior Boot Environment (when 11.0-U3) is installing... May test some later, but perhaps a Dev can shed some light on the matter...