SOLVED VM locking with "bug soft lockup", not even rebooting

[Fire-Dragon-DoL]

Hi,
For some reason my "brand new" VM with Debian latest locks up randomly with a soft lockup issue.
I'm not sure what's causing it, I do have docker containers running in it.
Based on my research, it seems to be a Bhyve problem so I'm not sure there is an actual solution (threads I found with solutions are quite old).

Any recommendation?
I like jails, but various apps I use are delivered as compose files and I don't want to deal with creating a bunch of new services every time the compose file changes, hence the VM.

I have a decent server at home: 96GB of RAM, Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
The usage is very low (only my family), so I doubt the issue is the hardware.

Any suggestion?

I would be satisfied with rebooting the VM every 24 hours, but I already have that measure in place and the lockup still happened.

 

[Stux]

I upgraded to scale to resolve this.

And it did.

My Ubuntu/debian VMs came up quite easily.

And finally, I no longer needed a VM anyway to run my compose files using native sandboxes/jails with jailmaker.

 

[Fire-Dragon-DoL]

I upgraded to scale to resolve this.

And it did.

My Ubuntu/debian VMs came up quite easily.

And finally, I no longer needed a VM anyway to run my compose files using native sandboxes/jails with jailmaker.

I'm afraid that's the answer

 

[Davvo]

iirc @Patrick M. Hausen wrote about something similar a good while ago (at least 6 to 9 months ago?) somewhere.

 

[Stux]

It’s a bhyve bug. I think it’s fixed in 14.

 

[Fire-Dragon-DoL]

It’s a bhyve bug. I think it’s fixed in 14.

You mean FreeBSD 14? But from my understanding, TrueNAS CORE will never move to FreeBSD 14, which is a huge disappointment, since this problem will never be fixed.

 

[Stux]

You mean FreeBSD 14? But from my understanding, TrueNAS CORE will never move to FreeBSD 14, which is a huge disappointment, since this problem will never be fixed.

Yes. Which is why I upgraded to Scale... now that scale has jail-like sandboxes, and that resolved the issue for me, and in fact, it also fixed some other bhyve issues I had... for example, not being able to reliably shutdown/reboot a bhyve vm from inside the vm.

 

[Fire-Dragon-DoL]

Yes. Which is why I upgraded to Scale... now that scale has jail-like sandboxes, and that resolved the issue for me, and in fact, it also fixed some other bhyve issues I had... for example, not being able to reliably shutdown/reboot a bhyve vm from inside the vm.

I'm terrified by the upgrade (something can go wrong). But I'm also concerned there were various people saying SCALE is "beta software" because X (I can't remember the various reasons).
I'm surviving with 2 cronjobs rebooting the VM per day for now, but I wish there was an actual fix.

I'm planning to move to SCALE, since being able to use docker and compose is a dealbreaker even if you don't want to use it.

 

[Stux]

I'm terrified by the upgrade (something can go wrong). But I'm also concerned there were various people saying SCALE is "beta software" because X (I can't remember the various reasons).
I'm surviving with 2 cronjobs rebooting the VM per day for now, but I wish there was an actual fix.

I'm planning to move to SCALE, since being able to use docker and compose is a dealbreaker even if you don't want to use it.

Yes. The scale upgrade is scary. Its also fairly amazing how well the migration works.

Backup your config. And try it, if something goes horribly wrong, then re-install TrueNAS core and upload your config.

BUT it went pretty well for me the few times I did it

TrueNAS CORE 13-U6.1 -> SCALE 23.10.2 Migration Feedback

Just performed a migration to SCALE 23.10.2 from one of my CORE servers. I ran two VMs on CORE, one for pfSense with NIC PCIe passthrough, and the other an Ubuntu docker machine for running docker compose apps. I moved away from Jails years ago now, and am looking forward to jailmaker in...

www.truenas.com

 

[Fire-Dragon-DoL]

Yes. The scale upgrade is scary. Its also fairly amazing how well the migration works.

Backup your config. And try it, if something goes horribly wrong, then re-install TrueNAS core and upload your config.

BUT it went pretty well for me the few times I did it

TrueNAS CORE 13-U6.1 -> SCALE 23.10.2 Migration Feedback

Just performed a migration to SCALE 23.10.2 from one of my CORE servers. I ran two VMs on CORE, one for pfSense with NIC PCIe passthrough, and the other an Ubuntu docker machine for running docker compose apps. I moved away from Jails years ago now, and am looking forward to jailmaker in...

www.truenas.com

That sounds scaringly easy.
I'm assuming I will need to rebuilt all my jails from scratch though.

The scary part is losing data, the annoying part is rebuilding my jails (with whatever is the equivalent)

 

[Stux]

I migrated from jails to docker compose inside a vm using NFS mounts some time back.

The VM ports very easily.

Then I easily migrated the docker compose files to sandboxes when I had a few hours to tinker, switched to host mounts from nfs mounts, and was able to shutdown the NFS server and the VM.

 

[Fire-Dragon-DoL]

I migrated from jails to docker compose inside a vm using NFS mounts some time back.

The VM ports very easily.

Then I easily migrated the docker compose files to sandboxes when I had a few hours to tinker, switched to host mounts from nfs mounts, and was able to shutdown the NFS server and the VM.

That warning at the top doesn't inspire using the sandboxes though, lol

Thank you by the way, really appreciate your input. Let me mark this as "solved"