NTP not syncing

[iLikeWaffles]

Also getting NTP problems.

Code:

root@freenas[~]# ntpdate -d 1.pl.pool.ntp.org
19 Oct 03:34:31 ntpdate[57347]: ntpdate 4.2.8p11-a (1)
transmit(162.159.200.1)
receive(162.159.200.1)
transmit(91.212.242.21)
receive(91.212.242.21)
transmit(162.159.200.123)
receive(162.159.200.123)
transmit(193.70.94.182)receive(193.70.94.182)
transmit(162.159.200.1)
receive(162.159.200.1)
transmit(91.212.242.21)
receive(91.212.242.21)
transmit(162.159.200.123)
receive(162.159.200.123)
transmit(193.70.94.182)
receive(193.70.94.182)
transmit(162.159.200.1)
receive(162.159.200.1)
transmit(91.212.242.21)
receive(91.212.242.21)
transmit(162.159.200.123)
receive(162.159.200.123)
transmit(193.70.94.182)
receive(193.70.94.182)
transmit(162.159.200.1)
receive(162.159.200.1)
transmit(91.212.242.21)
receive(91.212.242.21)
transmit(162.159.200.123)
receive(162.159.200.123)
transmit(193.70.94.182)
receive(193.70.94.182)
server 162.159.200.1, port 123
stratum 3, precision -26, leap 00, trust 000
refid [162.159.200.1], delay 0.03908, dispersion 0.00020
transmitted 4, in filter 4
reference time:    e154cac9.a2e5eb30  Sat, Oct 19 2019  1:35:37.636
originate timestamp: e154cafe.610feb1a  Sat, Oct 19 2019  1:36:30.379
transmit timestamp:  e154e6ad.f3244917  Sat, Oct 19 2019  3:34:37.949
filter delay:  0.03943  0.03908  0.03947  0.03937
         0.00000  0.00000  0.00000  0.00000
filter offset: -7087.57 -7087.57 -7087.57 -7087.57
         0.000000 0.000000 0.000000 0.000000
delay 0.03908, dispersion 0.00020
offset -7087.577890

server 91.212.242.21, port 123
stratum 2, precision -23, leap 00, trust 000
refid [91.212.242.21], delay 0.04564, dispersion 0.00005
transmitted 4, in filter 4
reference time:    e154c4c2.6bdccc8e  Sat, Oct 19 2019  1:09:54.421
originate timestamp: e154cafe.92ac1422  Sat, Oct 19 2019  1:36:30.572
transmit timestamp:  e154e6ae.23baa27e  Sat, Oct 19 2019  3:34:38.139
filter delay:  0.04605  0.04564  0.04567  0.04564
         0.00000  0.00000  0.00000  0.00000
filter offset: -7087.57 -7087.57 -7087.57 -7087.57
         0.000000 0.000000 0.000000 0.000000
delay 0.04564, dispersion 0.00005
offset -7087.576761

server 162.159.200.123, port 123
stratum 3, precision -25, leap 00, trust 000
refid [162.159.200.123], delay 0.03929, dispersion 0.00005
transmitted 4, in filter 4
reference time:    e154cae5.d29b7c3a  Sat, Oct 19 2019  1:36:05.822
originate timestamp: e154cafe.c779504e  Sat, Oct 19 2019  1:36:30.779
transmit timestamp:  e154e6ae.59878ab5  Sat, Oct 19 2019  3:34:38.349
filter delay:  0.03986  0.03961  0.03929  0.03938
         0.00000  0.00000  0.00000  0.00000
filter offset: -7087.57 -7087.57 -7087.57 -7087.57
         0.000000 0.000000 0.000000 0.000000
delay 0.03929, dispersion 0.00005
offset -7087.577443

server 193.70.94.182, port 123
stratum 2, precision -24, leap 00, trust 000
refid [193.70.94.182], delay 0.04001, dispersion 0.00035
transmitted 4, in filter 4
reference time:    e154c986.5dd74d31  Sat, Oct 19 2019  1:30:14.366
originate timestamp: e154cafe.fb015a23  Sat, Oct 19 2019  1:36:30.980
transmit timestamp:  e154e6ae.8cba5906  Sat, Oct 19 2019  3:34:38.549
filter delay:  0.04077  0.04001  0.04083  0.04036
         0.00000  0.00000  0.00000  0.00000
filter offset: -7087.57 -7087.57 -7087.57 -7087.57
         0.000000 0.000000 0.000000 0.000000
delay 0.04001, dispersion 0.00035
offset -7087.577071

19 Oct 03:34:38 ntpdate[57347]: step time server 193.70.94.182 offset -7087.577071 sec

The presence of receive messages indicates the NTP message does make the roundtrip, but for some reason FreeNAS chokes regardless. Doesn't matter which NTP server I choose, it always fails in the same way: no server suitable for synchronization found, if I take the -d debug flag off.

Other machines (not running FreeNAS) have no problems synchronizing. This includes a Windows, Android and Ubuntu box. This means it's not a firewall issue, or that whatever firewall issue there is, other OSes are able to deal with it.

 

[iLikeWaffles]

Important note:
If ntpdate -d appears to work, but ntpdate itself does not, chances are that it is a firewall issue. ntpdate -d uses unprivilidged ports. However, ntpdate -d does not actually update the date.

To update the date and also use the same unprivilidged ports as -d does, use ntpdate -d <server_ip>.

Now all I need is to find a way to permanently set ntpd to use unprivilidged ports as well, (not just the one-off ntpdate manual trick) and it should work for good.

 

[iLikeWaffles]

Sorry, I meant ntpdate -u <server_ip>

 

[nickt]

Alright - silly me. The battery was flat. My super micro board reported "green status" on 2.97 V, but replacing the battery sorted the issues straight away. Battery voltage is now 3.39 V, which is obviously more like it ought to be. Goodness knows how the VMs were able to keep NTP time under control. But it's fixed, so I'll stop wondering...

 

[rosivaldocorrea]

Is this thread opened? If yes, I have experiencing the same problem. The npt service is out of syncing.

I am using FreeNAS-11.1-U1. Set up the ntp server from the GUI (System > General > NTP Servers). I don't have any firewall blocking traffic between freenas and the ntp server. Any idea?

 

[Samuel Tai]

Also make sure FreeNAS chose the correct timer on boot: sysctl -a | grep quality. This will show all the candidate timers and their quality. FreeNAS ordinarily chooses the timer source with the highest quality, but on some systems, the quality value for some timers is incorrect. For example, on my system, TSC has quality 1000, but drifts uncontrollably, eventually leading to ntpd falling out of sync.

To fix this hardware quirk, you'll need to set a sysctl tunable kern.timecounter.hardware=HPET, and reboot.

 

[Parkinglot]

I scoured the net and no one has a definitive answer for this: Been running FreeNAS 11.x for over a year now and NTP has never worked correctly.
I would just go in regularly and run a ntpdate to get it corrected or restart the machine. The BIOS clock had been out of whack for a while because of battery, but I finally changed the battery, but NTP still seems to stray hours over a single day.

I configured in the FNAS GUI various time services. Tried the NIST, google, ubuntu, & others with no luck.
Is it the FW? -- I have a Palo Alto - running 9.1, hitting policy denys --- but this is udp out and back. seems to be OK.

running ntpq -p usually brings up something like this.

root@freenas[/etc]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
jell.yfish.us.l .STEP. 16 u - 1024 0 0.000 +0.000 0.000
time-a-g.nist.g .NIST. 1 u 55 64 375 7.270 +106050 50607.4
time3.google.co .GOOG. 1 u 19 64 377 11.080 +99623. 36153.5
root@freenas[/etc]#
root@freenas[/etc]# ntpq -c assoc
ind assid status conf reach auth condition last_event cnt
===========================================================
1 1564 8011 yes no none reject mobilize 1
2 1565 9024 yes yes none reject reachable 2
3 1566 9014 yes yes none reject reachable 1

Seems like I'm getting out to the servers [accept for the pool.ntp.org during this read came up 'no' for reach]. And like I said the ntpdate command works for any server I try; date and time gets updated and I'm good for a while.

Then I decided maybe the TrueNAS version might be better, so I loaded on a VirtualBox in my network. Configured it up to test.
Same result as above. NTP just won't synch -- to anything.

Then I decided to deploy my NTP server - CentOS 8, running chronyd. --- this is pointing to a bank of ntp servers.
[root@localhost etc]# chronyc sources
210 Number of sources = 4
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^+ clock.xmission.com 1 9 377 236 -1038us[ -873us] +/- 33ms
^* ntp1.wiktel.com 1 8 377 111 +4750us[+4923us] +/- 25ms <==== YEAH, that '*' means TIME is SYNCHED!!!!
^- radio-sunshine.org 2 9 377 114 -1039us[ -867us] +/- 100ms
^+ puppet.kenyonralph.com 3 9 377 54 -3973us[-3973us] +/- 40ms

From my FreeNAS, I took out all the public NTP servers and put my new server in. AND..... Voala

root@freenas[~]# ntpdate -s 192.168.1.222 <==== first synch date
root@freenas[~]# date
Thu Feb 4 17:06:22 EST 2021
root@freenas[~]# ntpq -p
ntpq: read: Connection refused
root@freenas[~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
192.168.1.222 .STEP. 16 u 71 64 0 0.000 +0.000 0.000 <==== not synch'ed
root@freenas[~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
192.168.1.222 69.89.207.99 2 u 2 64 1 0.474 +12854. 0.194 <==== This looks OK but not synched yet.
root@freenas[~]# ntpq -c assoc
ind assid status conf reach auth condition last_event cnt
===========================================================
1 7831 961a yes yes none sys.peer sys_peer 1 <==== Condition: sys.peer -- looks promising.

THEN - After about 15 minutes the clock is over a 1 minute off. Ugh!!!!

root@freenas[~]# ntpq -c assoc
ind assid status conf reach auth condition last_event cnt
===========================================================
1 7831 901a yes yes none reject sys_peer 1 <==== Condition: reject

SO --- I went to check the CentOS 8 Server. chronyd is still synched.
The FreeNAS server and the CENTOS 8 server are on the same subnet using the same DNS servers and going through the same Firewall.
What gives???? Freaking thing can't even synch to server on the same network.

Can someone tell me how to install Chronyd on the FreeNAS?

Or maybe iXsystems could just swap it out in the next release.
[Soo many "NTP not working" threads out there....]

 

[Samuel Tai]

Ah, yes, the HW clock drift issue. See https://www.truenas.com/community/threads/system-is-going-back-in-time-wrong-date.85686/post-593227.

 

[Parkinglot]

Ah, yes, the HW clock drift issue. See https://www.truenas.com/community/threads/system-is-going-back-in-time-wrong-date.85686/post-593227.

I'm not sure about that. The Virtual Box TrueNAS I wrote about has hardware clock that synched -- It's a fairly new Win10 machine running the latest VBox.

 

[Samuel Tai]

NTP's not syncing because your HW clock is drifting too far out of tolerance for NTP. This is a common problem. On your TrueNAS installation, run sysctl kern.timecounter.choice to display the timer candidates and their quality. In all likelihood, your system is using the TSC counter with quality 1000, but TSC is often a poor choice due to Spectre/Meltdown remediations which result in clock fuzzing. You may want to switch to the 2nd or 3rd best quality counter, like HPET. Set this via a sysctl tunable in System->Tunables kern.timecounter.hardware=HPET.

 

[Parkinglot]

OK - Not sure I added it correctly.

I rebooted the machine. But when I go to the SHELL -- it doesn't come up in the "choice" list. It looks the same as before I put the setting in the "Tunables" screen. Clock still not synching. --- Anyway, would it too much to ask to write out a step-by-step? Thanks,

 

[jgreco]

You have to use one of the choices from "kern.timecounter.choice".

 

[Samuel Tai]

Yes, and the type should be sysctl, not loader. In your case, the best option is probably ACPI-fast.

 

[Samuel Tai]

Use Variable=kern.timecounter.hardware, and Value=ACPI-fast.

 

[Parkinglot]

Do I reboot after this? Or should it take affect right away?

 

[Samuel Tai]

It takes effect immediately, but you may need to run ntpdate to get the clock sync'ed. Then monitor to make sure the sync holds.

To verify in the shell, run sysctl kern.timecounter.hardware.

 

[Parkinglot]

OK -- Finally a solution! The clock has now been synched for more than an hour and 45 minutes. Used to wander off in under 10 minutes.
Thank you!

 

[Steasenburger]

I also had massive time synchronization issues in the last time and apparently ntpd wasn't able to sync automatically.
The timecounter on my TrueNAS installation was also set to TSC, so i suspect the problem to be there.
I switched it to HPET and check on the command line that it is set correctly.

How can i check if the clock stays in sync with the NTP server?

 

[Constantin]

I’d let the system run for a while and then see if the time still drifts a lot. The clock on the NAS should always be dead nuts since so much on the TrueNAS depends on accurate timing. I once misconfigured my ntp servers and all sorts of hell broke loose after that.

Another thing to consider is running a NTP Server of your own. Centerclick makes two types of inexpensive Stratum 1 NTP servers that work really well. The 200 runs on 12VDC, the 250 can run on 12VDC or POE. It’s a really tidy package and a screaming deal for less than $200. No more worries re: your local isp blocking NTP access or ntp server pools getting annoyed with you due to overuse, etc. Home use loads on local ntp servers is so low as to be laughable, so crank down the refresh interval, if necessary.

As a side note, discovered that re-assigning the time.apple.com IP address to my local NTP also broke all IP Access for the local appleTV box. So time.apple.com seems to function for more than a NTP Server in the Apple universe.

 

[raelx]

I just went thru this after noticing my system time had drifted by about 5 minutes over a month or more (not exactly sure when it started). CMOS battery was good, but it seemed that none of the stock time servers could be reached (0.freebsd.pool.ntp.org 1.freebsd.pool.ntp.org 2.freebsd.pool.ntp.org). What worked for me was adding Google's server, you can see below it's the only one reachable. I'm on TrueNAS-12.0-U4

Code:

root@freenas[~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 europa.ellipse. .STEP.          16 u    -   64    0    0.000   +0.000   0.000
 linode.appus.or .STEP.          16 u    -   64    0    0.000   +0.000   0.000
 107-194-210-155 .STEP.          16 u    -   64    0    0.000   +0.000   0.000
*time1.google.co .GOOG.           1 u   20   64    3   39.280   -8.565   4.099
root@freenas[~]#