• Currently using VMware at work? We want to hear from you.

    Thinking of making a switch from VMware? We'd love to hear your thoughts and feedback about which hypervisor you have been researching or already using. Click here to vote and share your thoughts! You can vote HERE!

Emulated clock in bhyve is local time?

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,429
Hi all,

I just noticed that after reboot the clocks in my bhyve VMs (FreeBSD and Ubuntu) run one hour fast. This is quickly corrected by the NTP I have set up in each VM, but just out of curiosity: is it advisable to touch /etc/wall_cmos_clock for installations inside bhyve? If yes, does anyone know the equivalent for Unbuntu? ;)

Thanks,
Patrick
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,429
Nope :)
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,429
How do I change bhyve options for VMs I created with the UI?

Thanks
Patrick
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,429

dmshimself

Dabbler
Joined
Mar 20, 2017
Messages
41
Has anyone found a workaround for this at all? I don't seem to be able to use the date command in rancher to fix it, the documentation on using ros to set this sort of thing seems to be elusive and I have a number of date/time sensitive apps that I'd love to run as dockers in freenas and some of these don't run any sort time sync (well of course they don't!). Those that do are fine but the clock in rancher is a few minutes out.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,429
Well, as far as I'm concerned the clock is only off by an hour or two at boot time. Sooner or later NTP in the guest will fix it ...

Kind regards
Patrick
 

dmshimself

Dabbler
Joined
Mar 20, 2017
Messages
41
For those dockers I have with some sort of with NTP client, that indeed works. But I'm using some dockers which do not have that built in and rely on the host to have the correct time. My previous experiences of other similar techs makes me think that a synchronised time is really important, especially on the host OS and without that, things eventually trip up, usually when you least expect it. If rancheros supported /etc/localtime, that would probably be OK too as it could be passed on, but it doesn't seem to do that either. I have noticed that if you reboot the rancher VM, the time is again OK. For some people in a non-prod environment, this might be enough, but isn't ideal.
 

melar

Cadet
Joined
Nov 30, 2017
Messages
5
I've thought about the following, but didn't test it. (I'm still using iohyve)

bhyve is located under /usr/sbin/
rename bhyve to bhyve-renamed
add a new script named bhyve
Code:
#!/bin/sh
/usr/sbin/bhyve-renamed -u $*

chown root:wheel bhyve
chmod 555 bhyve

This adds the option -u to every use of bhyve. I think this shouldn't be a problem.
Maybe someone could test this.
 

dmshimself

Dabbler
Joined
Mar 20, 2017
Messages
41
I spoke a little too soon. The clock on rancher does drift off. It starts up fine after restarting the vm for the docker, but over time the clock runs slowly and there then becomes a gap. I'm making enquiries over in the rancheros forum too.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,429
@dmshimself are you running NTP inside the rancheros VM? Because you should ;)

Kind regards
Patrick
 

melar

Cadet
Joined
Nov 30, 2017
Messages
5
- Google for "bhyve clock sleep"
- Do a cat /sys/devices/system/clocksource/clocksource0/current_clocksource on your rancheros
Is it tsc ?
- Maybe a kern.timecounter.hardware="TSC-low" in /etc/sysctl.conf solves your clock problem.
 

dmshimself

Dabbler
Joined
Mar 20, 2017
Messages
41
@dmshimself are you running NTP inside the rancheros VM? Because you should ;)

Kind regards
Patrick

I'd love to know how to do that but I've searched high and low without any sucecss so far. It might be my own fault because I'm not running the full rancher, just using portainer. The server I have isn't that high a spec and rancher seems to load up a lot of containers. The last time I tried, the server was straining a bit. I've set up NTP loadsoftimes previously, but rancheros seems to be rightly locked down. Any advice appreciated.

Edited - I've just noticed that ntpd is running on the rancher box as default, so I'll have a good look at that and see why the clock is drifting.
 
Last edited:

dmshimself

Dabbler
Joined
Mar 20, 2017
Messages
41
- Google for "bhyve clock sleep"
- Do a cat /sys/devices/system/clocksource/clocksource0/current_clocksource on your rancheros
Is it tsc ?
- Maybe a kern.timecounter.hardware="TSC-low" in /etc/sysctl.conf solves your clock problem.
Cheers. I'll have a good rummage around later on today. The current_clocksource says refined-jifffies (I have no idea what that means)
 

dmshimself

Dabbler
Joined
Mar 20, 2017
Messages
41
I'd love to know how to do that but I've searched high and low without any sucecss so far. It might be my own fault because I'm not running the full rancher, just using portainer. The server I have isn't that high a spec and rancher seems to load up a lot of containers. The last time I tried, the server was straining a bit. I've set up NTP loadsoftimes previously, but rancheros seems to be rightly locked down. Any advice appreciated.

Edited - I've just noticed that ntpd is running on the rancher box as default, so I'll have a good look at that and see why the clock is drifting.

I've done a few system-docker logs ntp one after the other and the results may be just fine but look a little odd to me. The timestamp for the log is about 10 days out, but the date of the rancheros is about right as shown at the end. I'll leave it for a bit and check again, but maybe that's just the way it is. I'm used to having times being many hours out, but not many days.

Code:
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
+ ntpd -gq
13 May 18:14:07 ntpd[11]: ntpd 4.2.8p10@1.3728-o Wed Jul 26 20:13:20 UTC 2017 (1): Starting
13 May 18:14:07 ntpd[11]: Command line: ntpd -gq
13 May 18:14:08 ntpd[11]: proto: precision = 4000.250 usec (-8)
13 May 18:14:08 ntpd[11]: proto: fuzz beneath 0.342 usec
13 May 18:14:08 ntpd[11]: Listen and drop on 0 v6wildcard [::]:123
13 May 18:14:08 ntpd[11]: Listen and drop on 1 v4wildcard 0.0.0.0:123
13 May 18:14:08 ntpd[11]: Listen normally on 2 lo 127.0.0.1:123
13 May 18:14:08 ntpd[11]: Listen normally on 3 eth1 192.168.18.141:123
13 May 18:14:08 ntpd[11]: Listen normally on 4 lo [::1]:123
[rancher@rancher ~]$ date
Wed May 23 23:41:28 UTC 2018
[rancher@rancher ~]$

 

dmshimself

Dabbler
Joined
Mar 20, 2017
Messages
41
I left things running for a few hours and the time on the rancheros is about 5 minutes compared with the time in freenas (and everywhere else). If I look at the ntp log from the system-docker, I see:

Code:
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
+ ntpd -gq
13 May 18:14:07 ntpd[11]: ntpd 4.2.8p10@1.3728-o Wed Jul 26 20:13:20 UTC 2017 (1): Starting
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
+ ntpd -gq
13 May 18:14:07 ntpd[11]: ntpd 4.2.8p10@1.3728-o Wed Jul 26 20:13:20 UTC 2017 (1): Starting
13 May 18:14:07 ntpd[11]: Command line: ntpd -gq
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
[rancher@rancher ~]$ sudo system-docker logs ntp
+ echo 'starting in one shot mode to fix large time differences'
starting in one shot mode to fix large time differences
+ ntpd -gq


And this just repeats endlessly, the time never sees to get set even though the parameters should allow a large mismatch to be corrected.

if I then stop the ntp system docker and run ntpd by hand, I get a definite time change and the clock on rancheros is again correct for a while.

Code:
rancher@rancher ~]$ sudo ntpd -d -n
24 May 05:38:16 ntpd[26584]: ntpd 4.2.8p10@1.3728-o Wed Jul 26 20:13:20 UTC 2017 (1): Starting
24 May 05:38:16 ntpd[26584]: Command line: ntpd -d -n
24 May 05:38:16 ntpd[26584]: proto: precision = 4000.250 usec (-8)
24 May 05:38:16 ntpd[26584]: proto: fuzz beneath 0.280 usec
24 May 05:38:16 ntpd[26584]: Listen and drop on 0 v6wildcard [::]:123
24 May 05:38:16 ntpd[26584]: Listen and drop on 1 v4wildcard 0.0.0.0:123
24 May 05:38:16 ntpd[26584]: Listen normally on 2 lo 127.0.0.1:123
24 May 05:38:16 ntpd[26584]: Listen normally on 3 eth0 10.13.0.101:123
24 May 05:38:16 ntpd[26584]: Listen normally on 4 eth1 192.168.18.113:123
24 May 05:38:16 ntpd[26584]: Listen normally on 5 docker0 172.17.0.1:123
24 May 05:38:16 ntpd[26584]: Listen normally on 6 lo [::1]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 7 eth0 [2a00:1508:a0d:fe00:2a0:98ff:fe25:ca6f]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 8 eth0 [fe80::2a0:98ff:fe25:ca6f%2]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 9 eth1 [fe80::2a0:98ff:fe6c:7b93%3]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 10 docker0 [fe80::42:a4ff:fe08:af3e%6]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 11 veth3c89864 [fe80::988e:b0ff:feb0:ecdb%8]:123
24 May 05:38:16 ntpd[26584]: Listen normally on 12 veth767c584 [fe80::8833:f4ff:fea0:d42%11]:123
24 May 05:38:16 ntpd[26584]: Listening on routing socket on fd #29 for interface updates
24 May 05:38:41 ntpd[26584]: ntpd exiting on signal 2 (Interrupt)
24 May 05:38:41 ntpd[26584]: 203.118.151.32 local addr 10.13.0.101 -> <null>
24 May 05:38:41 ntpd[26584]: 122.252.188.99 local addr 10.13.0.101 -> <null>
24 May 05:38:41 ntpd[26584]: 130.217.226.51 local addr 10.13.0.101 -> <null>
24 May 05:38:41 ntpd[26584]: 202.78.240.38 local addr 10.13.0.101 -> <null>
[rancher@rancher ~]$ date
Thu May 24 05:38:44 UTC 2018


I also note that the running this manually shows a log that doesn't have the huge number of days out that the log for the system-docker seems to show. That might be a red herring, but note the jump form the 13th May to the 24th May on the logs.

It looks to my uninformed eye that the system-docker is trying to change the time with a big jump, but never gets that done and so repeats.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,429
I assume by "dockers" you mean docker containers, right? They run on top of a minimal Linux system very roughly similar to FreeBSD jails.

That minimal Linux is your rancheros VM. And you should run NTP in that VM outside of all docker containers. Otherwise your clock will always drift.

Now searching a bit about rancher it seems like they replaced all system services with containers. So essentially you need to run *one* privileged container that runs NTP and all other ones without, since they get their time from the host.

HTH, that's about as much as I know and can explain since I don't run docker ;)
Patrick
 
Top