TrueNAS will not boot after upgrade to 13

[speedtriple]

Upgraded from 12U8.1.
Everything worked to begin with.
But now TrueNAS 13 has shut itself down and will not boot.
Hanging on "Doorbell handshake failed".

 

[mistermanko]

Assuming you made a config backup beforehand, I'd try to install from iso to an new boot environment and restore the config through the gui.

 

[speedtriple]

Assuming you made a config backup beforehand, I'd try to install from iso to an new boot environment and restore the config through the gui.

Thanks, have config, so that will work. Or reinstall.

Hardware running TrueNAS as a VM might have some problems, missing drivers for 13.0 maybe LSI-card, will check.

 

[speedtriple]

TrueNAS 13 boots up fine after rebooting ESXi host. But then TrueNAS 13.0 shuts down after som minutes..for some reason.

Was able to revert to 12U8.1 in the boot menu. No problems with that edition..(!)

So it seems like a problem with LSI-card and TrueNAS 13.0 for my ESXi installation. See screenshot.
Driver problem in TrueNAS 13?

When reverting to previous 12U8.1 version in the boot menu, how to make it permanent?
ie how to delete the 13.0 version so that it does not boot up 13.0 in case of power trouble or any other reason for a reboot of the system?

 

[wnholt01]

Same problem here. Updated to 13.0U1 from 12.0U8.1 on ESXI 6.7 with LSI 2008 pass through and TrueNAS would shutdown after a few minutes and then "Doorbell handshake failed" error when trying to restart TrueNAS. Rebooting the ESXI host temporarily resolved the issue, but only for a few minutes. Restoring 12U8.1 fixed the issue.

 

[speedtriple]

In the 13U1-prerelease-thread I described that I did a second attempt.
Exactly the same outcome, except now I was totally unable to restore back to 12U8.1.

I figured that I would have to do a clean install of CORE 13U1 this time.
Instead I did a clean install of SCALE 22.02.2, and then applied the CORE 12U8.1-config file. Everything worked again. (except Syncthing manual jail).

But it was very very snappy to install the Syncthing-App in SCALE which also was up to date to latest version. Now I do not have to bother with the manual Syncthing-jail-update procedure in CORE 12U8.1 to have it up to date.

So far I have not seen any negatives with running SCALE instead of CORE, and the linux-base should be more robust, also for Syncthing and VM's, than CORE, and maybe make ESXi obsolete for me in the future.

 

[wnholt01]

In the 13U1-prerelease-thread I described that I did a second attempt.
Exactly the same outcome, except now I was totally unable to restore back to 12U8.1.

I figured that I would have to do a clean install of CORE 13U1 this time.
Instead I did a clean install of SCALE 22.02.2, and then applied the CORE 12U8.1-config file. Everything worked again. (except Syncthing manual jail).

But it was very very snappy to install the Syncthing-App in SCALE which also was up to date to latest version. Now I do not have to bother with the manual Syncthing-jail-update procedure in CORE 12U8.1 to have it up to date.

So far I have not seen any negatives with running SCALE instead of CORE, and the linux-base should be more robust, also for Syncthing and VM's, than CORE, and maybe make ESXi obsolete for me in the future.

 

[wnholt01]

Thanks for the update. I was hoping scale would work due to the different OS. I will probably just skip 13 and update to scale once the performance problems have been resolved.

 

[speedtriple]

Good strategy.

I just did the SCALE upgrade because I had to do a new clean VM/install anyway, and did not want to waste more time if 13U1 still would stall on ESXi.

Lawrence perfomance testing is from May on SCALE 22.02.1.
The SCALE performance might already be on par with 22.02.2, or will go past CORE in August with 22.02.3.

For my use I can not see any difference as of today.

Truenas Scale vs Core

Hello all, I recently configured a NAS server for media sharing purposes, then tried both scale and core on it. Hardware configuration. Mobo : Asrock X570 Taichi CPU : AMD 5600x HDD : 8 X 12 TB Exos RAM : Corsair Vengeance 128 GB RAM NIC : Intel X540 - T2 SATA : Using Mobo's...

www.truenas.com

 

[mav@]

Did anybody of reporters create a ticket about it so that it could be fixed one day?

 

[speedtriple]

Did anybody of reporters create a ticket about it so that it could be fixed one day?

Not so far. As far as I know iXsystems do not recommend or give any support for virtualized instances of any TrueNAS.
Maybe it works if clean install of an ESXi VM with TrueNAS 13. I have not tested.

 

[mav@]

Indeed we are not recommending VM installs, but trying to do what we can, especially if the issue is serious and/or there are multiple reporters.

It would be good to see what vmware logs say about "TrueNAS 13 has shut itself down". May be there are some hints towards the cause, unless it was some panic from TrueNAS side we could debug instead. We'd need some input, otherwise we'd have to spend time reproducing it.

 

[speedtriple]

Understand.
I am not an expert with ESXi. Looked trough the ESXi logs, but can not see anything I can give as more info.
And I do not want to start the old 13 VM again since it is setup with the same LSI-card/hdd's that I am using in the now working SCALE-install - I do not want any corrupt data. I believe the LSI-card was the problem since the boot halts after the card is identified by the CORE boot process, so no point in removing the LSI-card from the 13U1 VM.

The only info I have are the screenshot in the start of this thread, and some more similar screenshots of the boot process before it halts.

 

[speedtriple]

Indeed we are not recommending VM installs, but trying to do what we can, especially if the issue is serious and/or there are multiple reporters.

It would be good to see what vmware logs say about "TrueNAS 13 has shut itself down". May be there are some hints towards the cause, unless it was some panic from TrueNAS side we could debug instead. We'd need some input, otherwise we'd have to spend time reproducing it.

Ticket created.

 

[mav@]

Just to be sure, does it affect both 13.0-RELEASE and 13.0-U1?

 

[speedtriple]

Yes. June 8 was the regular upgrade on the train from 12U8.1 to 13.0-release. Was able to recover back to 12U8.1.
July 3 was a manual upgrade from 12U8.1 to the 13.0U1-prerelease, was unable to recover this time.

"Doorbell handshake failed" in constant loop when booting after ident of the LSI-card.

 

[mav@]

I am interested in this part: "Rebooting the ESXI host temporarily resolved the issue, but only for a few minutes." -- How does it look there? Any more relevant messages? "Restoring 12U8.1 fixed the issue." -- does it fix it without ESXi host reboot?

 

[speedtriple]

I am interested in this part: "Rebooting the ESXI host temporarily resolved the issue, but only for a few minutes." -- How does it look there? Any more relevant messages? "Restoring 12U8.1 fixed the issue." -- does it fix it without ESXi host reboot?

No relevant messages after host reboot, the 13-release booted up fine, but crashed after some minutes. And any attempt to reboot it thereafter got stuck at the "doorbell-message".
Restore to 12U8.1 did not require host reboot, and fixed the issue in June.

In July, 12U8.1 would not boot after my second upgrade attempt.

 

[ddion]

Exact the same issue as described by speedtriple.
@speedtriple do you have the ticketnumber so we can see the progression or are these tickets private?

 

[speedtriple]

According to iXsystems it is due to latest FreeBSD, so there is nothing iXsystems can do about this case and they suggest stick to SCALE as it is working for me.

Have to wait for FreeBSD to update with fixes.