SOLVED VM locking with "bug soft lockup", not even rebooting

Fire-Dragon-DoL

Contributor
Joined
Dec 22, 2018
Messages
103
Hi,
For some reason my "brand new" VM with Debian latest locks up randomly with a soft lockup issue.
I'm not sure what's causing it, I do have docker containers running in it.
Based on my research, it seems to be a Bhyve problem so I'm not sure there is an actual solution (threads I found with solutions are quite old).

Any recommendation?
I like jails, but various apps I use are delivered as compose files and I don't want to deal with creating a bunch of new services every time the compose file changes, hence the VM.

I have a decent server at home: 96GB of RAM, Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
The usage is very low (only my family), so I doubt the issue is the hardware.

Any suggestion?

I would be satisfied with rebooting the VM every 24 hours, but I already have that measure in place and the lockup still happened.
 

Attachments

  • softlock-vm.png
    softlock-vm.png
    6.2 KB · Views: 139

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
I upgraded to scale to resolve this.

And it did.

My Ubuntu/debian VMs came up quite easily.

And finally, I no longer needed a VM anyway to run my compose files using native sandboxes/jails with jailmaker.
 

Fire-Dragon-DoL

Contributor
Joined
Dec 22, 2018
Messages
103
I upgraded to scale to resolve this.

And it did.

My Ubuntu/debian VMs came up quite easily.

And finally, I no longer needed a VM anyway to run my compose files using native sandboxes/jails with jailmaker.
I'm afraid that's the answer
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
iirc @Patrick M. Hausen wrote about something similar a good while ago (at least 6 to 9 months ago?) somewhere.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
It’s a bhyve bug. I think it’s fixed in 14.
 

Fire-Dragon-DoL

Contributor
Joined
Dec 22, 2018
Messages
103
It’s a bhyve bug. I think it’s fixed in 14.
You mean FreeBSD 14? But from my understanding, TrueNAS CORE will never move to FreeBSD 14, which is a huge disappointment, since this problem will never be fixed.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
You mean FreeBSD 14? But from my understanding, TrueNAS CORE will never move to FreeBSD 14, which is a huge disappointment, since this problem will never be fixed.

Yes. Which is why I upgraded to Scale... now that scale has jail-like sandboxes, and that resolved the issue for me, and in fact, it also fixed some other bhyve issues I had... for example, not being able to reliably shutdown/reboot a bhyve vm from inside the vm.
 

Fire-Dragon-DoL

Contributor
Joined
Dec 22, 2018
Messages
103
Yes. Which is why I upgraded to Scale... now that scale has jail-like sandboxes, and that resolved the issue for me, and in fact, it also fixed some other bhyve issues I had... for example, not being able to reliably shutdown/reboot a bhyve vm from inside the vm.
I'm terrified by the upgrade (something can go wrong). But I'm also concerned there were various people saying SCALE is "beta software" because X (I can't remember the various reasons).
I'm surviving with 2 cronjobs rebooting the VM per day for now, but I wish there was an actual fix.

I'm planning to move to SCALE, since being able to use docker and compose is a dealbreaker even if you don't want to use it.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
I'm terrified by the upgrade (something can go wrong). But I'm also concerned there were various people saying SCALE is "beta software" because X (I can't remember the various reasons).
I'm surviving with 2 cronjobs rebooting the VM per day for now, but I wish there was an actual fix.

I'm planning to move to SCALE, since being able to use docker and compose is a dealbreaker even if you don't want to use it.

Yes. The scale upgrade is scary. Its also fairly amazing how well the migration works.

Backup your config. And try it, if something goes horribly wrong, then re-install TrueNAS core and upload your config.

BUT it went pretty well for me the few times I did it

 

Fire-Dragon-DoL

Contributor
Joined
Dec 22, 2018
Messages
103
Yes. The scale upgrade is scary. Its also fairly amazing how well the migration works.

Backup your config. And try it, if something goes horribly wrong, then re-install TrueNAS core and upload your config.

BUT it went pretty well for me the few times I did it

That sounds scaringly easy.
I'm assuming I will need to rebuilt all my jails from scratch though.

The scary part is losing data, the annoying part is rebuilding my jails (with whatever is the equivalent)
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
I migrated from jails to docker compose inside a vm using NFS mounts some time back.

The VM ports very easily.

Then I easily migrated the docker compose files to sandboxes when I had a few hours to tinker, switched to host mounts from nfs mounts, and was able to shutdown the NFS server and the VM.
 

Fire-Dragon-DoL

Contributor
Joined
Dec 22, 2018
Messages
103
I migrated from jails to docker compose inside a vm using NFS mounts some time back.

The VM ports very easily.

Then I easily migrated the docker compose files to sandboxes when I had a few hours to tinker, switched to host mounts from nfs mounts, and was able to shutdown the NFS server and the VM.
That warning at the top doesn't inspire using the sandboxes though, lol

Thank you by the way, really appreciate your input. Let me mark this as "solved"
 
Top