can't ping/access ubuntu VM soon(ish) after reboot

[jaaassh]

Twice now, I've attempted to ssh into my ubuntu VM (from Freenas, and from elsewhere on the network) and found that I can't get in. The VM is up and running, and I can connect via VNC though.

It seems that rebooting freenas "fixes" this, temporarily, but I last did that 5 days ago, and now the same issue has reoccured. Restarting the VM doesn't change the situation. I don't such much in /var/log/messages - or any log tha tI've poked around in.... but I could be looking in the wrong place.

Is there anything I can try short of restarting freenas?

 

[dlavigne]

Does this resolve the issue for you: https://redmine.ixsystems.com/issues/62712#note-4?

 

[jaaassh]

I don't have that box checked. My symptoms aren't he same either. As far as I can tell the VM boots fully when freenas starts - at least by the time I connect to it with VNC it is already at the login prompt.

 

[dlavigne]

Anything in the logs on the Ubuntu side (eg losing IP address, NIC going down, SSH service stopping)?

 

[diskdiddler]

Can you clarify for me, is the VM even on (can you ping it?)

I have had similar issues.

 

[jaaassh]

Note: I've updated to 11.2 now. I destroyed the old VM and created a new one with the new UI. It was up and running after the install and setup.

But now, a few days later.. the issue is back.

I tried sshing into the instance and that didn't do much. pinging didn't do anything. I tried VNC from the freenas UI and that showed a black screen. So, I restarted the instance. I still can't get into the intstance though. It looks like it came up (booted to a prompt) but the VNC client doesn't connect normally. e.g if I connect, then type things, I don't see the keystrokes. If I reconnect the VNC client, then I can see the updates. I can't ssh into the VM either.

Fast forward an hour or so of screwing around with network configs inside ubuntu and (everything looks standard) I decided to try a new VM.

Same VNC/hanging issue in the new VM... so yea, doesn't seem like an ubuntu issue.

I'm pretty sure rebooting freenas resolves this, so what could be happening that causes vnc to not work well after a few days?

 

[diskdiddler]

This is very remniscent of my issues, I made a guacamole docker and the docker host VM simply, eventually 'died' networking wise.


Take a peek at this job, I've just tried to unmark it private, I ended up stopping running VMs (I've been very short for time, will be for a month)
https://redmine.ixsystems.com/issues/61101


Here's a similar, strange issue.
https://redmine.ixsystems.com/issues/63045
VM died, tried to open a shell, if you're not observant, you're actually in the REAL freenas shell.

 

[jaaassh]

Rebooted a few days ago.... tried to get in today. Fail.

@diskdiddler i'm not seeing huge spikes in load on freenas, so I'd guess it's not the same issue happening here? It's also not showing that 'all ports busy' message - this screenshot was taken after minutes of just sitting there with the cursor blinking. Looks like it's never going to get to the VM prompt:

 

[jaaassh]

Even if I create a second VM, this issue persists on that one too. Presumably until I reboot freenas.

 

[jaaassh]

I rebooted the ubuntu VM - and now it's in this state where I can't SSH to it, and VNC doesn't refresh correctly (see above).

Additionally, it seems that ubuntu can't correctly pull an IP:

...this just hangs for the full 5 minutes before booting beyond that without any network.

Not clear, but it seems like whatever virtual networking these VMs depend on is completely torpedoed.

 

[diskdiddler]

Rebooted a few days ago.... tried to get in today. Fail.

@diskdiddler i'm not seeing huge spikes in load on freenas, so I'd guess it's not the same issue happening here? It's also not showing that 'all ports busy' message - this screenshot was taken after minutes of just sitting there with the cursor blinking. Looks like it's never going to get to the VM prompt:

View attachment 27651

My huge spikes are in the VM, not FreeNAS, the VM is almost entirely unresponsive.

 

[diskdiddler]

Can you ping the VM, from your desktop, can you ping the VM from FreeNAS itself?

 

[jaaassh]

My huge spikes are in the VM, not FreeNAS, the VM is almost entirely unresponsive.

Understood - but if there are spikes in the VM, wouldn't they translate to some spikage in FreeNAS? Either way - I don't think I'm seeing similar spikes.

 

[jaaassh]

Negative. No pinging. And from the last screenshot, thats not entirely surprising as it seems it can't even pull an address from DHCP.

 

[diskdiddler]

Understood - but if there are spikes in the VM, wouldn't they translate to some spikage in FreeNAS? Either way - I don't think I'm seeing similar spikes.

Not based on my reporting no, the VM went haywire and the main box was mostly fine.

 

[KrisBee]

@jaaassh It might help to know more details about your VM, e.g: Ubuntu release/kernel version, static IP or DHCP, any remote mounts.
Virtio rather than e1000 for VM NIC would probably give you better performance. Does systemd give any clue as to why network is failing in VM?

Why do you expect to be able to make a serial connection to your VM? Have you modified grub/kernel params to provide connection over serial connection?

 

[jaaassh]

@jaaassh It might help to know more details about your VM, e.g: Ubuntu release/kernel version, static IP or DHCP, any remote mounts.
Virtio rather than e1000 for VM NIC would probably give you better performance.

Previously (start of this thread) it was built from ubuntu-16.04.5-server-amd64.iso - static IP (configured in /etc/network/interfaces). No remote mounts. NIC is default: intel 82454(1000). VirtIO doesn't seem to play nice on my hardware and the intel performance is acceptable.

At some point in the mix I rebuilt the VM, and used DHCP, but set a MAC address in my router.

Both of those setups resulted in the same issue.

As of 24 hours ago, I rebuilt another VM on ubuntu-18.04.1.0-live-server-amd64.iso, same DHCP config (but no mac set in router, yet).... The VM is still accessible normally as of this morning.

Does systemd give any clue as to why network is failing in VM?

Hard to say when I can't get into the VM once this issue crops up.

Why do you expect to be able to make a serial connection to your VM? Have you modified grub/kernel params to provide connection over serial connection?

I dunno, I'm not super familiar with configuring those. I kinda assumed they would just work. All things considered, that's kinda a nice to have as I wouldn't use it much unless there's a problem with ssh.

---

For my own edification, why do we still think it's something to do with the VM config when, once the issue appears, newly built VMs exhibit the same symptoms?

 

[KrisBee]

@jaaassh What's the state of the FreeNAS network when the VM appears to drop net connectivity? I've not heard of virtio NIC problems before, what is your hardware? There might be some clues in /var/log/middlewared.log . If and when your VM becomes unresponsive, can you use your install iso to boot into recovery mode and poke around?

There's no serial console by default for a typical linux distro, see here for a post-install config example: https://www.hiroom2.com/2018/04/30/ubuntu-1804-serial-console-en/

 

[jaaassh]

FreeNAS is fine... I mostly use NFS and Plex, they seem completely unaffected.

I'll check that log file when this happens again.

...I guess I could try the ISO/recovery mode. But if brand new VMs that I set up after the issue experience the same thing... what would I be expecting to find in the busted one? I guess I'm not sure where to look when it seems the issue would have(?) to be in freenas somewhere and not inside a VM.

 

[jaaassh]

Okay, happened again, and got some info out of /var/log/middlewared.log

Attached are two files:

1) with_networking_issue.txt - showing the VM booting, noodling around, and a forced shutdown.
... then I rebooted freenas (which fixes things) and started the VM:
2) without_networking_issue.txt - showing the VM booting.

The 2nd one is a lot less noisy. The first shows a bunch of repeated output, but it's unclear if any of that is an explicit error?