Diagnosing a hung VM, Bhyve? What to do.

[diskdiddler]

VM no longer responds to ping
VM is turned 'on' in the TrueNAS UI.

Can I talk to the com port?
/dev/nmdm19B

Unfortunately VNC comes back with
"Failed to connect to server"


Perhaps a bhyve CLI command to pull status or something would be nice? See if ANY processes or memory use is moving / changing?
I'd love to know why this occurs.

 

[sretalla]

Can I talk to the com port?
/dev/nmdm19B

That's possibly your best bet...
cu -l /dev/nmdm19B

 

[diskdiddler]

That's possibly your best bet...
cu -l /dev/nmdm19B

Thank you.

I've rebooted it, but I wanna learn for next time.
I know in a WORKING state this is what happens.


root@truenas[~]# cu -l /dev/nmdm19B
Connected



I can't seem to do more. (I do not know, how to configure my linux VM to "accept" commands over serial, which is surely not disabled as default?)
None the less "connected" in green is a start.

 

[sretalla]

Pressing enter should get you started...

 

[diskdiddler]

Enter does nothing.

Machine is up.
It's UbuntuServer.

 

[Patrick M. Hausen]

Configuring a Linux VM for serial console takes some steps:

1. Edit /etc/default/grub

Code:

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,9600n8"
 
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

2. Run update-grub

3. Create /etc/init/ttyS0.conf

Code:

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
 
 
start on stopped rc RUNLEVEL=[12345]
stop on runlevel [!12345]
 
respawn
exec /sbin/getty -L 9600 ttyS0 vt102

4. Reboot

This works even in modern systemd based installations. Systemd will pick up the old-fashioned init file and create what ever it needs to create to start the login service.

HTH,
Patrick

 

[sretalla]

Not sure if this is different...

Attempting to use serial console to connect to Ubuntu VM hangs?

Hi, I have an Ubuntu VM installed with FreeNAS 11.2. I can use VNC to connect to the jail and ssh once configured. The serial option to connect to the jail never seems to work. The serial port is listed as /dev/nmdm1B and the console terminal attempt to connect using cu -l /dev/nmdm1B however...

www.truenas.com

 

[Patrick M. Hausen]

That thread is essentially the same problem. Linux by default does not use a serial console and needs to be explicitly told to do so. Once for GRUB so you can control the boot process, and then you need to create a getty service so you can login when the system is up and running.

 

[diskdiddler]

Configuring a Linux VM for serial console takes some steps:

1. Edit /etc/default/grub

Code:

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,9600n8"
 
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

2. Run update-grub

3. Create /etc/init/ttyS0.conf

Code:

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
 
 
start on stopped rc RUNLEVEL=[12345]
stop on runlevel [!12345]
 
respawn
exec /sbin/getty -L 9600 ttyS0 vt102

4. Reboot

This works even in modern systemd based installations. Systemd will pick up the old-fashioned init file and create what ever it needs to create to start the login service.

HTH,
Patrick

Patrick, FYI you've been consistently helpful the last year or so - this continues again today. Worked perfectly.
I'm logged in - console commands are coming up now which is interesting - might be useful to see what happens next time this baby freezes.


I mean ultimately, I may switch to SCALE and use KVM (or use docker native...) one day but for the mean time I want to see how my little UbuntuDockerVM goes on BHyve.
She freezes once every 6 to 18 weeks which isn't that often, it's otherwise been an amazing addition to CORE.


Let's see how we go.

I nearly tried to commit this post to disk with a CTRL-O .....

 

[jgreco]

Patrick, FYI you've been consistently helpful the last year or so

The last year or so? I've been finding useful and interesting nuggets in his posts for years. He and I both seem to do similar kinds of work in slightly different ways. There's lots of room there for picking up useful information, and it would be foolish not to do so.

 

[Kris Moore]

FYI - This is a primary reason we ripped all the bhyve out of our internal iX infrastructure in the past few years. Lots of these random lockups / freezes that are related to the general maturity of bhyve in general. We're using KVM/Xen now and it's been smooth sailing ever since.

 

[Patrick M. Hausen]

This is a primary reason we ripped all the bhyve out of our internal iX infrastructure in the past few years.

Something something dogfood ...

 

[Kris Moore]

Sure, we're dogfooding on KVM / SCALE now :P Bhyve was an interesting project, but just not worth the investment for iX to fix all these issues when there are lots of perfectly viable alternative OSS hypervisors out there. Why re-invent the wheel :)

 

[jgreco]

Why re-invent the wheel :)

To advance the state of the art and make more viable options to VMware (Cadillac of hypervisors, even today), Hyper-V, and other paid products.

Quite frankly, I want to be able to run using FreeBSD as the host platform, which reduces options somewhat. Our image builder here is based on bhyve because it scripts trivially, but it has the nasty habit of locking up every now and then.

I am in no way blaming you for switching, I understand that some of these things are really heavy lifts for a small team to be having to develop. The iSCSI kernel support was an amazing contribution to the state of FreeBSD. There are always tradeoffs and limits to how much development can be done with limited resources.

 

[Patrick M. Hausen]

@Kris Moore How about one of your devs joins the Bhyve Production Users Call held monthly by @michaeldexter? The name is intentional, there's a second developers call at a different date. So even if you don't have much capacity to aid in actual development of bhyve you can at least funnel feedback and your particular use cases and problems back at the FreeBSD devs who also join the users call.

Your user base is possibly one of the largest groups of users of bhyve and jails in the FreeBSD world.

 

[Kris Moore]

To advance the state of the art and make more viable options to VMware (Cadillac of hypervisors, even today), Hyper-V, and other paid products.

Quite frankly, I want to be able to run using FreeBSD as the host platform, which reduces options somewhat. Our image builder here is based on bhyve because it scripts trivially, but it has the nasty habit of locking up every now and then.

I am in no way blaming you for switching, I understand that some of these things are really heavy lifts for a small team to be having to develop. The iSCSI kernel support was an amazing contribution to the state of FreeBSD. There are always tradeoffs and limits to how much development can be done with limited resources.

Sure, we evaluated that and made the decision to support development KVM for this purpose on TrueNAS. It's far beyond bhyve in capabilities and already much more mature as a fully open-source alternative to the commercial options available.

 

[Kris Moore]

@Kris Moore How about one of your devs joins the Bhyve Production Users Call held monthly by @michaeldexter? The name is intentional, there's a second developers call at a different date. So even if you don't have much capacity to aid in actual development of bhyve you can at least funnel feedback and your particular use cases and problems back at the FreeBSD devs who also join the users call.

Your user base is possibly one of the largest groups of users of bhyve and jails in the FreeBSD world.

We had done that in the past. We've opened tickets and given feedback about the state of the world with regard to bhyve and jails.

Bottom line is that they just aren't evolving at the pace we need to see to consider them viable technologies for virtualization or containerization moving forward. And this is speaking as a guy who came out of the BSD world.

It's just sheer numbers at this point looking at the momentum of some of the BSD alt-projects vs their various Linux open-source counterparts. We have to be strategic in how we spend our development hours. It's a very poor return on investment to spend too many cycles trying to "catch up" a technology where there are already very viable open-source options available that have the features and maturity we want out of the box, and could be used as building blocks towards even greater things that end-users care about.

 

[Patrick M. Hausen]

I understand - at least in the case of bhyve. For jails I cannot understand why a container technology vastly superior to anything else for certain use cases (e.g. multi-tenancy with a full stack in each container) does not gain more traction.

 

[Kris Moore]

I understand - at least in the case of bhyve. For jails I cannot understand why a container technology vastly superior to anything else for certain use cases (e.g. multi-tenancy with a full stack in each container) does not gain more traction.

I've pondered that for years. Even worked directly on some various Jail management tools (cough cough, warden, cough cough, iocage). But the pragmatist in me has to admit now that the Linux containers have completely dominated the landscape of open source application deployment. Docker, and now K8's have completely taken it to a new level, which is a shame since the underlying jail technology has been around SO long, but never got much traction in the wider ecosystem.

 

[jgreco]

I've pondered that for years. Even worked directly on some various Jail management tools

I may momentarily have the attention of two highly qualified people to give feedback on this. Hm.

phk and I were both presenters at SANE 2000 and I had the pleasure of chatting with him for a bit at the time.

Confining the omnipotent root

This was right around the time that FreeBSD 4.0 was being released with native support for jails, IIRC. As one of the earliest adopters of phk's jails, I guess I've tended to view (and use) jails as more of a chroot-on-steroids, and not really gotten into the VNET stuff or "jail management tools" after trying one or two in the early days and finding them lacking.

In the service provider environment, a lot of the risk in UNIX systems are the classic stack smash, so, running applications in a /bin/sh-less environment is desirable.

I am generally focused on creating "apps" (jails) which run on a FreeBSD platform and are built "from the ground up". That is, rather than populating a jail with all the FreeBSD release files and going over to /usr/ports, instead I start with an empty jail, and then to install something like PostgreSQL, for example, use something like

./configure --prefix=/pgsql --sysconfdir=/pgsql/etc

when building it to force it "inside" the jail. Then it can be launched in a bare minimum /bin/sh-less environment. This is not as easy as I'm making it sound, naturally, but it's totally doable.

It is my impression from a relatively small sample set of encounters with other jail/container/etc strategies that this is relatively unusual, and most of them start with some sort of filesystem image. i.e. if you do "docker run <container> ls -l bin/sh" there's a shell there (bash, dash, whatever).

I've confirmed this several times over the years talking to developerfolks who were giving talks on Docker or K8S, and when I ask about this, I usually get a very befuddled look like "how else could it be."

Are there other efforts out there to create "safer" jails/containers/etc or am I just crazy? No hurt feelings either way. ;-)