SOLVED Some debian VMs do not start after upgrade from 11.2u8 to 11.3u4.1: symbol grub_calloc not found

[tslw]

I recently upgraded from 11.2u8 to 11.3u4.1. After the upgrade some of my Debian VMs do not start anymore (2 out of 6).
In VNC I see the following error message for the failing VMs:

Code:

error: symbol 'grub_calloc' not found.
Entering rescue mode.
grub_rescue> _

The VM has the following properties:

Code:

Virtual CPUs: 2
Memory Size: 4.00 GiB
Boot Loader Type: UEFI
System Clock: LOCAL
VNC Port: 5918
Com Port: /dev/nmdm4B

The VM has three devices: disk, nic and vnc

I have read through the forum and tried to change the disk mode (VirtIO/ AHCI) but this does not help. I saw some posts on the internet suggesting to repair it with an USB repair disk (e.g. https://askubuntu.com/questions/1263125/how-to-fix-a-grub-boot-error-symbol-grub-calloc-not-found) but I don't know how to do this in freenas.

Does anybody have any suggestions to solve this?

 

[Samuel Tai]

If you have a Debian live ISO image, you can add a CDROM device pointing to the live ISO file to the misbehaving VMs, with a sequence order in front of the other devices. Then after the VM boots from the live ISO, you can reinstall grub to the disk zvol. Afterwards, shut down the VM, and remove the CDROM device.

 

[tslw]

Thanks for the tip! It helps me a step further but it does not work yet.

I booted into the live CD and executed the commands (below note that your disk and partitions may differ).
It executes without errors but when I restart the VM I get the same grub error as before :-(

Commands (edited):

Code:

sudo mount /dev/vda2 /mnt
sudo mount /dev/vda1 /mnt/boot/efi
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt$i; done
sudo chroot /mnt
sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi /dev/vda
update-grub

Edit:
See post #7 that you might have to copy grubx64.efi to another location:
sudo cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/BOOT/bootx64.efi

from: https://askubuntu.com/questions/831216/how-can-i-reinstall-grub-to-the-efi-partition

 

[Samuel Tai]

Are you sure you're reinstalling the EFI GRUB? From your grub-install command, you could be installing the BIOS GRUB. Try sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi /dev/vda to force the EFI GRUB.

 

[tslw]

I am pretty sure that the second time I tried I used the parameter --target=x86_64-efi (and did not update the commands above). I checked that the files in /boot/efi were updated. Tonight I will try again to be sure.

 

[KrisBee]

@tslw Looks like you've been hit by the grub bug that was widely report in linux land by people updating their systems at the end of July & beginning of August. Once you've correctly chrooted into your non-booting VM you'll need to apt update then reinstall the grub package and/or apt upgrade. No point in reinstalling a faulty package. Make sure you have a network connection within your VM after the chroot.

 

[tslw]

I retried the suggestion from @Samuel Tai but it did not help.

I just remembered that when I created the VM I had to set an extra step to get it running again.
See https://www.jongibbins.com/solving-uefi-boot-problems-on-bhyve-freenas-vm/

After copying grubx64.efi to the right location everything is running fine again.
sudo cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/BOOT/bootx64.efi

Thanks for your suggestions!