debian vm's under bhyve, clock?

[jaywest]

I am not sure if this is a debian bullseye issue or a truenas core (bhyve?) issue, but wanted to check here before I go down the debian path.

I have three different debian vm's. All occasionally toss this message in their respective /var/log/messages:
[ 344.714783] clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc' as unstable because the skew is too large:
[ 344.714788] clocksource: 'hpet' wd_now: 5883c9dc wd_last: 5800adce mask: ffffffff
[ 344.714790] clocksource: 'tsc' cs_now: 10a813796b3 cs_last: 10a3e0bbd07 mask: ffffffffffffffff
[ 344.714797] tsc: Marking TSC unstable due to clocksource watchdog
[ 344.714809] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
[ 344.714811] sched_clock: Marking unstable (344544344566, 170072341)<-(344732744698, -17936377)
[ 344.714797] clocksource: Checking clocksource tsc synchronization from CPU 1.
[ 344.714877] clocksource: Switched to clocksource hpet
[ 9452.922253] CE: hpet2 increased min_delta_ns to 38625 nsec
[80579.478266] CE: hpet1 increased min_delta_ns to 38625 nsec
[326897.779672] CE: hpet1 increased min_delta_ns to 57937 nsec

A) Is this likely a bhyve issue with what it is representing hardware-wise to debian? Or is it more likely strictly a debian issue? The (dual) cpus are AMD Opteron 41KX HE.
B) Is the suggestion in the message the best path to try next (which is, using tsc-unstable boot option for the vm) or is there something at the truenas level to try first?

I've googled the clocksource & CE messages above quite a bit, but still not coming away with any understanding as to which direction to look. Any advice appreciated!

Best,

J

 

[Patrick M. Hausen]

Not much to tune at the TrueNAS level if you want to stay in the line of supported and UI controlled features. I never noticed this on the variety of hardware I run, see my signature for details.

I would try to tackle this from the Linux side. You do have ntpd up and running in all your VMs, don't you?

 

[jaywest]

yep, I definitely believe in staying within the bounds of "supported and UI controlled features" for truenas. Despite being a fairly advanced and long-time freebsd user, I believe it's best to let the appliance be an appliance. VM's are where you can "go your own way" :D

I will look at it from the linux side then. I don't have ntpd running, but am using the default systemd-timesyncd to X.debian.pool.ntp.org on all the debian vm's and 'systemctl status systemd-timesyncd' shows all are synced. I dont' think it's the system time running afoul in /var/log/messages, looks like the hardware timers. Thanks for the input!

 

[alejo_true]

I have 3 Ubuntu VM's all three of them are experiencing similar messages in their respective `/var/log/messages`.

I have been looking at the linux side for a while now, but I am yet to find a solution :/

@jaywest Have you come up with anything?

Regards

 

[GeorgePatches]

I recently noticed this same error on my Debian machine that I setup recently. I did some digging as to how time keeping works and what's causing this error.

Chapter 15. Timestamping Red Hat Enterprise Linux for Real Time 7 | Red Hat Customer Portal

Access Red Hat’s knowledge, guidance, and support through your subscription.

access.redhat.com

An introduction to timekeeping in Linux VMs

Now's a good time for an introduction to timekeeping in Linux VMs.

opensource.com

So most of the big players in hypervisors (Vmware, KVM, HyperV, Xen) seem to intercept the rdtsc instruction and do some magic that ensures that the TSC stays consistent even as the VM moves from core to core on a host, or even from one host to another. Bhyve does not seem to provide this magic, or at least it is not as refined as say VMware's magic. I'm still looking into how Bhyve does timekeeping stuffs, but info is thin compared to say VMware or KVM. I'll post back if I find a solution.

Now, on the issue of your time source, and TSC vs HPET. TSC is preferred beause it returns much faster, down in the nanosecond range according to Red Hat's article above. HPET takes much longer, but like single microseconds, so still pretty damn fast. The best solution at this time is to just use HPET, it'll only be an issue if you're hosting a VM that's responding to millions of requests all the time. If you're a home gamer like me, it'll never come up. If you're hosting a business, why are running your VMs on TrueNAS Core and Bhyve? Like sure it works, but it seems like the way of pain.

*magic = Difficult and skilled work done by experienced software engineers that I can't even begin to comprehend.