Diagnosing a hung VM, Bhyve? What to do.

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
phk and I were both presenters at SANE 2000 and I had the pleasure of chatting with him for a bit at the time.
I miss meeting him and chatting with him dearly. Even pre COVID he stopped doing a lot of traveling and attending e.g. EuroBSDCon
That is, rather than populating a jail with all the FreeBSD release files and going over to /usr/ports, instead I start with an empty jail, and then to install something like PostgreSQL, for example, use something like
[...]
I know about that option - there were examples in the docs for e.g. named. Unfortunately (or not) we are providing hosting services so my main focus shifted to full-blown jails with everything and the kitchen sink in them, orchestration, private virtual networking between jails in addition to the Internet facing interface etc.

As for Docker - I thought that was the point? Containerising a single binary, not a full blown OS environment. That's how Docker proponents always explained it to me.

Kind regards
Patrick
 

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
That really does sound like the hard-way to do a jail/container in 2022, so few know how to even run ./configure these days :)

That said, on the Docker / container side, you can always start with a super minimal image, or even ship just a single binary if needed. Seen lot of really tiny containers that are running some super minimal thing with some GO binary inside them. A shell is nice to have inside the container for testing / debug work of course, but would probably something I'd strip out in production.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
docker-works-on-my-machine.jpg
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,377
The last year or so? I've been finding useful and interesting nuggets in his posts for years. He and I both seem to do similar kinds of work in slightly different ways. There's lots of room there for picking up useful information, and it would be foolish not to do so.
This may be true but I am infrequent on the forums. He's behaving like a support hound paid to do it at this point :)
Some smart people here, dunno how you all do it.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Some smart people here, dunno how you all do it.
Thanks to all for the compliments. Why I do it: not being a 1337 kernel hacker this is my way of giving back to FreeBSD and related projects. I run my entire hosting department on jails and ZFS and I am very fond of appliances built on top of FreeBSD, so I do support work for TrueNAS and OPNsense. And starting last year I even got into coding for OPNsense.
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,377
FYI - This is a primary reason we ripped all the bhyve out of our internal iX infrastructure in the past few years. Lots of these random lockups / freezes that are related to the general maturity of bhyve in general. We're using KVM/Xen now and it's been smooth sailing ever since.

Hey Kris,

Didn't realise that, thanks.
I would (blindly) assume that the BSD folks would be aware of these kinds of issues and work towards nailing them. It has such a stable reputation.
I will say my 2 other VMs are rock solid. Maybe it's a load thing.


Definitely a good reason to consider scale.
Thanks for info!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
As for Docker - I thought that was the point? Containerising a single binary, not a full blown OS environment. That's how Docker proponents always explained it to me.

I thought that was the point too. However, I just did "docker pull postgres", "docker run -e POSTGRES_HOST_AUTH_METHOD=trust postgres", "docker run postgress ls -l bin/sh" and it shows a link to dash (a stripped-down bash).

To the best of my understanding, this isn't someone's private copy of Postgres, it's the version that most of the Docker world uses. It downloaded a crapton of stuff from the Internet in doing this. I'm too Docker-naive to have done anything complicated or clever installing the thing.

Now I am happy to admit where my strengths are NOT, and Docker is certainly one of the not-strengths. But when I poke around inside the thing it looks like some sort of reasonably full UNIX environment, more complete than Busybox. And I can see that it uses some sort of overlay mechanism; I actually had this figured out at one point, which was where I lost faith in containers. Heh.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I would (blindly) assume that the BSD folks would be aware of these kinds of issues and work towards nailing them.

So many facets of hypervisors involve arcane issues that may not be frequently exercised, it's gotta be really hard. Even VMware isn't perfect. I am actually relatively happy with bhyve as part of automated build testing here. It's pretty surreal to see it just sitting there grinding out the image builds I have the VM's doing. To save on wear and tear on disk, I have it building system images on a virtual disk that is actually a RAMdisk in the host system's memory. The bhyve host itself is actually just a whopping big ESXi VM (32GB/6 core). It has come a huge way in such a short time.
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,377
Well, I've caught her in the act with an issue, though slightly different to normal.


My docker container web frontends not available.
Not responding to ping
However, the VNC panel works for a change.
This is the console.
TBdeDNF.png



I can just reboot it to bring the network back up in this instance, not a full hang.
Any tips what I should look for regardless?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Why are you using AHCI virtual disks? You should run disk and network as VirtIO devices when running Linux or FreeBSD guests.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
I thought that was the point too. However, I just did "docker pull postgres", "docker run -e POSTGRES_HOST_AUTH_METHOD=trust postgres", "docker run postgress ls -l bin/sh" and it shows a link to dash (a stripped-down bash).

To the best of my understanding, this isn't someone's private copy of Postgres, it's the version that most of the Docker world uses. It downloaded a crapton of stuff from the Internet in doing this. I'm too Docker-naive to have done anything complicated or clever installing the thing.
As far as I have seen, developing and administering a bunch of various things, the somewhat-dirty secret is that few things are as stripped-down as they could be. This is good and bad.
It's good because Docker's tooling is byzantine when things work well and infuriatingly user-hostile when they don't. The main problem, from my perspective - keeping in mind that I need to peek into the containers for stuff like debugging, configuration, etc. - is that the container's filesystem is an opaque image. To access anything inside the container, there's no reasonable alternative to executing a shell process inside the container and attaching a TTY to it (docker exec -it ${CONTAINERNAME} bash, slightly less horrible with docker-compose: docker-compose exec ${CONTAINERNAME} bash).
It's bad because the attack surface is larger and it adds bloat.

To the best of my understanding, this isn't someone's private copy of Postgres, it's the version that most of the Docker world uses. It downloaded a crapton of stuff from the Internet in doing this. I'm too Docker-naive to have done anything complicated or clever installing the thing.
What happens in practice, behind the buzzwords, is the following:
  1. Take a "base image", which is the base userland. Think bsdinstall or debootstrap, but as a proprietary image.
  2. Add/change your application's dependencies on top of that, using the standard tooling. Hopefully the author cleaned up the cruft!
  3. Add the actual application bits on top of 2.
3. is pretty straightforward. 2. suffers from all the same things that have always plagued admins.
1. is "pick a distro" territory, with all that entails. Postgres, for instance, offers two sets of options: Debian or Alpine. The former is something like the minimum install of Debian. Alpine is weird and less-than-compatible, by virtue of not using glibc in favor of something smaller, which means that things break sometimes.
The crapton of stuff are the various layers, basically a stack of CoW overlays. These can be merged together, but common practice is to have several layers to allow for reuse. E.g. one layer for the base image, one for major dependencies that don't change much, one for dependencies that do change frequently, and one for the actual application.
Although it's not quite "Bob's copy of Postgres", it's not too far off. It's effectively "Postgres installed in a semi-custom environment that Bob setup up", shipped in binary format. For Open-Source stuff, the expectation is that the Dockerfiles defining the build are also OSS.

As for Docker - I thought that was the point? Containerising a single binary, not a full blown OS environment. That's how Docker proponents always explained it to me.
That's mostly a fantasy along the lines of "the OS is the kernel and the rest is like, your opinion, man". It's a line of reasoning that serves to dump problems and design flaws on top of others, but adds nothing to any real-world scenario. Conceptually, you could compile a single binary that does everything, but there are good reasons why nobody does that outside of embedded systems.

So, what's good about Docker? I think there are two things that made it popular: Dockerfiles and Dockerhub. A Dockerfile doesn't do anything you can't do in a shell script, but it makes a lot of boilerplate stuff easy, allowing for nice, repeatable (not necessarily reproducible, in practice) builds without the frustration of a typical shell script. And Dockerhub makes it easy to get started by just downloading an image and then letting you grab the Dockerfile and work from there.

Some of the networking options are neat and automate some boring stuff like DNS and are good enough for containers that should not communicate with the outside world, only with other containers. I say "some" because the options somewhat bloated, as I understand it to support Docker Swarm (which I will venture to say is used by nobody of consequence).

I think that's about where my praise ends, at least relative to jails. Sure, it saves my ass to be able to run an aggressively closed-source app developed for a very specific CentOS 6 environment without having to run CentOS 6 on the host machine, but jails can do that, too.

Most of Docker's problems are probably caused by unfocused development efforts, throwing random features at the wall to see what sticks. There seems to be exactly zero planning involved in the development of Docker. Let's take a look at docker-compose: Compose files are where the problems begin, and I invite those fortunate enough to not yet have been scarred to have a look at the documentation. Here are some nuggets:
  • You'd think 2.x would supersede 1.x and the same for 3.x and 2.x. But that's not true, they evolved in parallel over multiple versions of Docker.
  • Big deal, just update Docker and use 3.x, you say? Nope, 2.x has features not in 3.x.
  • What really changed? I'm not sure I can explain it. The changes seem more like a Wikipedia edit war over the capitalization of an article's title than any serious change to the API.
  • Want to know more about a given entry? Chances are, it'll just point to the main Docker docs, which tend to be more bizarre than helpful.
  • Wait, what's this "Compose Specification"? Excellent question, I think they gave up on versioning and now it's a fscking free-for-all. Oh, and the docs are worse now, they're just a crummy markdown file on Github.
Docker also suffers a bit from having to reinvent some wheels, being a Linux thing:
  • Image handling. For systems without clean water ZFS, they need to be able to support the image stuff. If they could rely on ZFS, it could trivially be replaced with ZFS snapshots and send/recv. There is a ZFS driver, but I'm not sure it's using any ZFS features beyond creating tons of datasets. Still, it's better than partitioning off some physical disk space and handing it over to Docker, which was the recommended approach and maybe still is.
  • A million distros mean a million base images and it becomes hard to standardize tooling.
  • Linux doesn't have containers, so Docker has to cobble together the various bits and pieces to create that abstraction.
 
Last edited:

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,377
My UbuntuServer has been up and behaving for 11 days straight.
I have a 3 days old SSH session open and start teaching a buddy about TrueNAS and docker. He's impressed

So I then open the TrueNAS console, show him the front end and go to VMs
The exact second I connect via VNC to show him the 'real' console.

"Error" no connection

fXuUUEY.png


The SSH session then dropped.
There's no doubt in my mind at this point that somehow, connecting to a VNC session can CAUSE the VM to hang.

Just FYI, if no one cares due to scale, that's ok. I am building a proxmox machine for my VMs and I'll be keeping TrueNAS to just handle storage soon sadly.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
I never use VNC - possibly that's why I don't notice problems.
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,377
To be fair I /almost/ never use it either but I do from time to time and that was plain as day, definitely hung the system, no question
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Did you file an issue in JIRA for that already? If it's reproducable, it can probably be fixed. By "never" I mean I actually remove the VNC device from Linux VMs.
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,377
Did you file an issue in JIRA for that already? If it's reproducable, it can probably be fixed. By "never" I mean I actually remove the VNC device from Linux VMs.

I think I will, but I'm unsure what to provide them with, to establish it's occurring, I guess I could download a debug log next time and upload it.
I get the impression from the forums, I'm the only one even running linux under bhyve.
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,377
"I am sorry but there is simply not much we can do here.
libvirt and bhyve issues are outside of our control and we have no resources to investigate ourselves.
For a better VM experience we suggest you to try TrueNAS SCALE."


Ok so that's that, I would suggest the VM functionality be removed from CORE or a disclaimer is added stipulating it's entirely unsupported by IX
I get that they can't do it, but it's pointless to keep it in there without a disclaimer.
I'll try SCALE one day but now that I have a Proxmox machine it's going to be a while before I need to move to SCALE.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Could you spare some time to join one of the next bhyve production user calls? I at least am still interested in what happens in the first place. Because I do run dozens of VMs, Windows and Linux in production without problems. The Windows VMs don't crash whe I use VNC once every few months. The Linux VMs don't have VNC devices as I said. So there must be something different, and I would like to know that something, because currently I have no idea why my bhyve behaves acceptably but would very much like to keep it that way in the future.

This is going to be a bit of work but I'm all for improving FreeBSD. We would need to hack the middleware a bit to make bhyve log more debug information etc. - that's why I would start with joining the call with the devs.

DM me for details if you are interested.
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,377
Potentially. Though I'm in Australia east time zone. I'd certainly find it interesting.

I'm also willing to trust a small fraction of people here to remote in and check if necessary yourself included.

I can check the vnc console with more regularity if we add some kind of extensive logging.

I wouldn't be suprised if this is somehow something I've done, however I figure it really shouldn't do what it's doing regardless.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@Patrick M. Hausen Out of interest, what distro base do you use for your Linux VMs? I've used alpine for a VM acting as a docker host and not seen problems. True it's ligthly loaded and not left running for weeks at a time, but avoiding systemd, graphical interfaces, LVM, etc. seemed a sensible choice when using bhyve.
 
Top