LONG Startup-Time (Hours) after Upgrade

[streppel]

Hey all,

I upgraded truenas to bluefin today. Sadly, after the upgrade, the boot times have increased immensely.
The kernel log up until the point "Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist" is fine.

After that, it takes almost exactly 30 seconds for each other message to appear.

I already tried a fresh install of truenas bluefin and imported the backup there -> first boot was fine.
After another reboot the same issue occured.

CPU: Ryzen 5600X
Board: MSI MAG Motar B550m
128GB RAM (non-ECC)

Pools:
SSD: 4x Crucial MX500 4TB
HDD: 2x5TB WD Red
Boot-Pool: Single 64GB SSD

Other Hardware:
NVidia Quadro P400

Example Log:

For the moment I'll switch back to the previous release.
If you need any further information, let me know.

Thanks for your help!

 

[cloudswei]

Hey all,

I upgraded truenas to bluefin today. Sadly, after the upgrade, the boot times have increased immensely.
The kernel log up until the point "Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist" is fine.

After that, it takes almost exactly 30 seconds for each other message to appear.

I already tried a fresh install of truenas bluefin and imported the backup there -> first boot was fine.
After another reboot the same issue occured.

CPU: Ryzen 5600X
Board: MSI MAG Motar B550m
128GB RAM (non-ECC)

Pools:
SSD: 4x Crucial MX500 4TB
HDD: 2x5TB WD Red
Boot-Pool: Single 64GB SSD

Other Hardware:
NVidia Quadro P400

Example Log:

For the moment I'll switch back to the previous release.
If you need any further information, let me know.

Thanks for your help!

I have exactly the same issue. Upgraded to Bluefin from 22.02.4 and reboot, it took systemd exactly 30 seconds for printing each other message.
Tried a fresh install using 22.12.0 iso. First boot seemed to be working fine. After reboot, the problem happened again and it took forever to boot up.

 

[cloudswei]

My setup:

CPU: i3 10100T
Motherboard: MSI B560i
RAM: 64G DDR4 3200MHz

Pool: 2 NVME SSDs for cache, 5 HDDs for data

 

[usaleem-ix]

Hey all,

I upgraded truenas to bluefin today. Sadly, after the upgrade, the boot times have increased immensely.
The kernel log up until the point "Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist" is fine.

After that, it takes almost exactly 30 seconds for each other message to appear.

I already tried a fresh install of truenas bluefin and imported the backup there -> first boot was fine.
After another reboot the same issue occured.

CPU: Ryzen 5600X
Board: MSI MAG Motar B550m
128GB RAM (non-ECC)

Pools:
SSD: 4x Crucial MX500 4TB
HDD: 2x5TB WD Red
Boot-Pool: Single 64GB SSD

Other Hardware:
NVidia Quadro P400

Example Log:

For the moment I'll switch back to the previous release.
If you need any further information, let me know.

Thanks for your help!

I believe this is happening because of serial console and the following should fix your issue:

It look like you are not using the serial console. If there is nothing connected to the serial port, and/or you are not using the serial console, can you please try the following?

While on GRUB menu, press 'e' while boot entry for 22.12 is selected, and you should see a similar window

View attachment 61319
You should see console=ttyx console=ttySx,115200 . Please remove the console=ttySx,115200 and add loglevel=8 instead. Now you can press Ctrl+x or F10 to boot.

If you are able to boot after serial console parameter is removed, please disable the serial console from WebUI by going into System Settings > Advanced > Console > Configure > uncheck the Enable Serial port and save.

 

[cloudswei]

Thanks. Tried to disable console from 22.02.04 and upgrade to 22.12.0. It worked! After console is disabled, the system boot time was reduced to ~1 min from 4 mins (It is because the default console baud-rate in 22.02.04 is 9600).

 

[majorgear]

I believe this is happening because of serial console and the following should fix your issue:

It look like you are not using the serial console. If there is nothing connected to the serial port, and/or you are not using the serial console, can you please try the following?

While on GRUB menu, press 'e' while boot entry for 22.12 is selected, and you should see a similar window

View attachment 61319
You should see console=ttyx console=ttySx,115200 . Please remove the console=ttySx,115200 and add loglevel=8 instead. Now you can press Ctrl+x or F10 to boot.

If you are able to boot after serial console parameter is removed, please disable the serial console from WebUI by going into System Settings > Advanced > Console > Configure > uncheck the Enable Serial port and save.

Almost a year later, this post saved my life. I upgraded the CPU/Mobo/RAM on my TN and also removed some excess SSD's from the box today. I got to "starting journal" and just sat there for hours. I rebooted it, commented out ttySx speed as per instructs, and my server came right up!

Much thanks for this post. I had the installer USB drive inserted, but decided to try editing grub before wiping the system disk.

 

[artemis]

I will also get on the band wagon and praise this post. It just saved me as well. thanks again!

 

[artemis]

As a follow up, this does fix the issue, but because removing the 'console=ttys0 115200' line doesn't get saved to the Grub boot menu, the problem just reoccurs and the 8hr boot up time is back. Since this is a headless unit, I have to hook up a monitor in order to change the line to avoid the failed boot. Is there a way to permanently edit the grub menu to remove that line?

 

[artemis]

As a follow up, this does fix the issue, but because removing the 'console=ttys0 115200' line doesn't get saved to the Grub boot menu, the problem just reoccurs and the 8hr boot up time is back. Since this is a headless unit, I have to hook up a monitor in order to change the line to avoid the failed boot. Is there a way to permanently edit the grub menu to remove that line?

Never mind, this requires me to read :) Do this to remove the serial console. System Settings > Advanced > Console > Configure > uncheck the Enable Serial port and save.