TrueNAS 12.0-U1.1 break centos7 vm

[ertank]

Hello,
I just updated from 12.0-RELEASE to 12.1-U1.1

After that I see below line repeating thousands of times in one Vm log

Code:

atkbd data buffer full

I have two vms. Other one is debian 10 and it just works ok. Only CentOS Vm has this problem. It also consumes a lot of cpu time (seemingly completely hogs one core out of 4 total). I always see kqread state for this bhyve process.

Vm works fine after complete power recycle for a couple of seconds (ping requests answered) after this I cannot reach this Vm over SSH or using browser. Ping requests are not answered, too.

This CentOS is running NethServer and is my main Vm to reach to the internet. I am posting from my mobile phone as my internet is no more. No updates are applied to the Vm OS for about one month. It should not be the Vm OS.
I wonder if anybody else has such problem. I also wonder if there would be a solution.

Any help is appreciated.

Thanks & regards,
Ertan

 

[ertank]

I could connect with VNC on browser. Below is crash screen. Probably that is changing because I could sometimes get 1-2 ping reply.

Any help is appreciated.

Thanks & Regards,
Ertan

 

[Samuel Tai]

How is your CentOS VM configured? Are you using emulated devices (AHCI disk/e1000 NIC), or have you tried switching both to VirtIO?

 

[ertank]

It is installed using 1GB ram. Afterwards I reduced RAM to 512MB. It was running like that for months.
AHCI disk, one VirtIO nic and one e1000 NIC is what I use. Two NICs assigned to this specific VM.
I did not change any configuration so far. I do not have much experience with bhyve. These systems, I setup them and they were running just fine until today's update.

 

[ertank]

Weird, I changed both NICs to use VirtIO and now system seems to boot.
I need to do some more tests before I can say problem is solved.

 

[Samuel Tai]

Weird, I changed both NICs to use VirtIO and now system seems to boot.
I need to do some more tests before I can say problem is solved.

I also recommend you switch the AHCI disk to VirtIO. This will change your mounts in the VM from /dev/sda* to /dev/vda*.

 

[ertank]

I am not a CentOS guy either. My experience with Linux is also limited. I see my fstab file is as following.

Code:

#
# /etc/fstab
# Created by anaconda on Fri Oct 30 16:56:41 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       xfs     defaults        0 0
UUID=12ee0b0b-3d4a-44a2-87fb-33a266f6eae8 /boot                   xfs     defaults        0 0
UUID=B988-BC01          /boot/efi               vfat    umask=0077,shortname=winnt 0 0
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0

I could not see any /dev/sda1 etc above. So I do not know where and how to modify before I switch to VirtIO for VM disk. Because doing nothing and making the switch leads to boot failure.

 

[Samuel Tai]

Look in /boot/efi/EFI/ for device.map and grub.cfg. These may be in sub-directories. Replace any reference to sda with vda.

 

[ertank]

I still could not figure it out. Below is a list of files in my /boot

Code:

[root@neth boot]# ls -lR /boot
/boot:
total 194140
-rw-r--r--  1 root root   153567 Aug 25 20:27 config-3.10.0-1127.19.1.el7.x86_64
-rw-r--r--. 1 root root   153562 Apr  1  2020 config-3.10.0-1127.el7.x86_64
-rw-r--r--  1 root root   153596 Dec 18 19:38 config-3.10.0-1160.11.1.el7.x86_64
-rw-r--r--  1 root root   153595 Nov 17 17:03 config-3.10.0-1160.6.1.el7.x86_64
drwx------  3 root root    16384 Jan  1  1970 efi
drwxr-xr-x. 2 root root       27 Oct 30 16:58 grub
drwx------. 2 root root       21 Oct 31 20:50 grub2
-rw-------. 1 root root 64988346 Oct 30 17:03 initramfs-0-rescue-7ef2c711d2ac4483afb5d02f678c092e.img
-rw-------  1 root root 20927855 Dec 19 16:18 initramfs-3.10.0-1127.19.1.el7.x86_64.img
-rw-------  1 root root 20772423 Oct 31 20:52 initramfs-3.10.0-1127.el7.x86_64.img
-rw-------  1 root root 20927582 Jan 12 00:45 initramfs-3.10.0-1160.11.1.el7.x86_64.img
-rw-------  1 root root 20927194 Dec 19 16:17 initramfs-3.10.0-1160.6.1.el7.x86_64.img
-rw-r--r--  1 root root   320536 Aug 25 20:27 symvers-3.10.0-1127.19.1.el7.x86_64.gz
-rw-r--r--. 1 root root   320512 Apr  1  2020 symvers-3.10.0-1127.el7.x86_64.gz
-rw-r--r--  1 root root   320678 Dec 18 19:39 symvers-3.10.0-1160.11.1.el7.x86_64.gz
-rw-r--r--  1 root root   320667 Nov 17 17:03 symvers-3.10.0-1160.6.1.el7.x86_64.gz
-rw-------  1 root root  3612420 Aug 25 20:27 System.map-3.10.0-1127.19.1.el7.x86_64
-rw-------. 1 root root  3611662 Apr  1  2020 System.map-3.10.0-1127.el7.x86_64
-rw-------  1 root root  3617106 Dec 18 19:38 System.map-3.10.0-1160.11.1.el7.x86_64
-rw-------  1 root root  3617083 Nov 17 17:03 System.map-3.10.0-1160.6.1.el7.x86_64
-rwxr-xr-x. 1 root root  6762688 Oct 30 17:03 vmlinuz-0-rescue-7ef2c711d2ac4483afb5d02f678c092e
-rwxr-xr-x  1 root root  6765160 Aug 25 20:27 vmlinuz-3.10.0-1127.19.1.el7.x86_64
-rwxr-xr-x. 1 root root  6762688 Apr  1  2020 vmlinuz-3.10.0-1127.el7.x86_64
-rwxr-xr-x  1 root root  6769256 Dec 18 19:38 vmlinuz-3.10.0-1160.11.1.el7.x86_64
-rwxr-xr-x  1 root root  6769256 Nov 17 17:03 vmlinuz-3.10.0-1160.6.1.el7.x86_64

/boot/efi:
total 2
drwx------ 4 root root 2048 Oct 30 17:01 EFI

/boot/efi/EFI:
total 4
drwx------ 2 root root 2048 Oct 31 20:50 BOOT
drwx------ 3 root root 2048 Jan 28 03:34 centos

/boot/efi/EFI/BOOT:
total 1924
-rwx------ 1 root root 1243864 Jul 31 23:41 BOOTX64.EFI
-rwx------ 1 root root  362232 Jul 31 23:41 fallback.efi
-rwx------ 1 root root  362232 Jul 31 23:41 fbx64.efi

/boot/efi/EFI/centos:
total 6986
-rwx------ 1 root root     134 Jul 31 23:41 BOOT.CSV
-rwx------ 1 root root     134 Jul 31 23:41 BOOTX64.CSV
drwx------ 2 root root    2048 Oct 31 20:50 fonts
-rwx------ 1 root root    6995 Jan 12 00:45 grub.cfg
-rwx------ 1 root root    1024 Jan 12 00:45 grubenv
-rwx------ 1 root root 1097544 Jul 28  2020 grubx64.efi
-rwx------ 1 root root 1154640 Jul 31 23:41 mmx64.efi
-rwx------ 1 root root 1154640 Jul 31 23:41 MokManager.efi
-rwx------ 1 root root 1243864 Jul 31 23:41 shim.efi
-rwx------ 1 root root 1237824 Jul 31 23:41 shimx64-centos.efi
-rwx------ 1 root root 1243864 Jul 31 23:41 shimx64.efi

/boot/efi/EFI/centos/fonts:
total 2502
-rwx------ 1 root root 2560080 Jul 28  2020 unicode.pf2

/boot/grub:
total 4
-rw-r--r--. 1 root root 1350 Nov 15  2011 splash.xpm.gz

/boot/grub2:
total 0
lrwxrwxrwx 1 root root 25 Oct 31 20:50 grubenv -> ../efi/EFI/centos/grubenv
[root@neth boot]# 

There is no device.map file I could find. System.map* files do not contain any "sda" reference. My grub.cfg file contains:

Code:

[root@neth centos]# cat grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
set tuned_initrd=""
### END /etc/grub.d/00_tuned ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'NethServer (3.10.0-1160.11.1.el7.x86_64) 7.9.2009' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1127.el7.x86_64-advanced-b5dbf1af-5107-4ec8-9f06-58d31c85e5e1' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod xfs
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        else
          search --no-floppy --fs-uuid --set=root 12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        fi
        linuxefi /vmlinuz-3.10.0-1160.11.1.el7.x86_64 root=/dev/mapper/VolGroup-lv_root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=VolGroup/lv_root rd.lvm.lv=VolGroup/lv_swap nodmraid rhgb quiet LANG=en_US.UTF-8
        initrdefi /initramfs-3.10.0-1160.11.1.el7.x86_64.img
}
menuentry 'NethServer (3.10.0-1160.6.1.el7.x86_64) 7.8.2003' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1127.el7.x86_64-advanced-b5dbf1af-5107-4ec8-9f06-58d31c85e5e1' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod xfs
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        else
          search --no-floppy --fs-uuid --set=root 12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        fi
        linuxefi /vmlinuz-3.10.0-1160.6.1.el7.x86_64 root=/dev/mapper/VolGroup-lv_root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=VolGroup/lv_root rd.lvm.lv=VolGroup/lv_swap nodmraid rhgb quiet LANG=en_US.UTF-8
        initrdefi /initramfs-3.10.0-1160.6.1.el7.x86_64.img
}
menuentry 'NethServer (3.10.0-1127.19.1.el7.x86_64) 7.8.2003' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1127.el7.x86_64-advanced-b5dbf1af-5107-4ec8-9f06-58d31c85e5e1' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod xfs
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        else
          search --no-floppy --fs-uuid --set=root 12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        fi
        linuxefi /vmlinuz-3.10.0-1127.19.1.el7.x86_64 root=/dev/mapper/VolGroup-lv_root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=VolGroup/lv_root rd.lvm.lv=VolGroup/lv_swap nodmraid rhgb quiet LANG=en_US.UTF-8
        initrdefi /initramfs-3.10.0-1127.19.1.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1127.el7.x86_64-advanced-b5dbf1af-5107-4ec8-9f06-58d31c85e5e1' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod xfs
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        else
          search --no-floppy --fs-uuid --set=root 12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        fi
        linuxefi /vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/mapper/VolGroup-lv_root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=VolGroup/lv_root rd.lvm.lv=VolGroup/lv_swap nodmraid rhgb quiet LANG=en_US.UTF-8
        initrdefi /initramfs-3.10.0-1127.el7.x86_64.img
}
menuentry 'CentOS Linux (0-rescue-7ef2c711d2ac4483afb5d02f678c092e) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0-rescue-7ef2c711d2ac4483afb5d02f678c092e-advanced-b5dbf1af-5107-4ec8-9f06-58d31c85e5e1' {
        load_video
        insmod gzio
        insmod part_gpt
        insmod xfs
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        else
          search --no-floppy --fs-uuid --set=root 12ee0b0b-3d4a-44a2-87fb-33a266f6eae8
        fi
        linuxefi /vmlinuz-0-rescue-7ef2c711d2ac4483afb5d02f678c092e root=/dev/mapper/VolGroup-lv_root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=VolGroup/lv_root rd.lvm.lv=VolGroup/lv_swap nodmraid rhgb quiet
        initrdefi /initramfs-0-rescue-7ef2c711d2ac4483afb5d02f678c092e.img
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
[root@neth centos]# 

I am kind of stuck converting my disk controller to VirtIO. I really do not want to re-install complete system now as it started to work again.

Thanks & Regards,
Ertan

 

[Samuel Tai]

Look in /boot/efi/EFI/centos/ or /boot/efi/EFI/. There should be a device.map file there.

 

[ertank]

Actually, there is no device.map file. I shared all files under /boot directory for that in my earlier post.

This is CentOS 7.9.2009 version. Maybe missing device.map file is related with that. I really don't know.

Thanks & Regards,
Ertan

 

[ertank]

I could convert my disk controller to VirtIO with help from NethServer community here.
Basically, I needed to install generic initramfs package, rebuild initramfs before controller change.

Code:

yum -y install dracut-config-generic
kernelrelease=$(uname -r)
dracut -f /boot/initramfs-$kernelrelease.img $kernelrelease

After that, shutdown VM, change disk controller type to VirtIO. Boot VM. It boots just fine.

It is advised to remove generic initramfs package and rebuild initramfs to shrink it after successfully converting disk controller to VirtIO. I did that using below commands:

Code:

yum -y erase dracut-config-generic
kernelrelease=$(uname -r)
dracut -f /boot/initramfs-$kernelrelease.img $kernelrelease

These commands helped me convert my VM from AHCI to VirtIO disk controller.

Thanks & Regards,
Ertan

 

[Samuel Tai]

Ok, thanks for the update.