NTP health check failed - No NTP peers

Joined
Feb 23, 2022
Messages
3
I installed truenas scale in my old PC and tested the stability. And, the system send me a alert email.
New alerts:
  • NTP health check failed - No NTP peers: [{'122.117.253.246': 'REJECT'}, {'60.248.114.17': 'REJECT'}, {'103.169.212.1': 'REJECT'}]
I think that is mean my "NTP synchronized" is no. (I check the timedatectl status)
But, when I try to sync it, the system show it has been masked. So,I can't activate it.

Does someone have same problem?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Timeservice in Asia is a little bit dicey. This is an Internet thing, not really a TrueNAS thing. Have you tried using whatever your ISP recommends as NTP servers?
 

varet

Dabbler
Joined
Dec 6, 2021
Messages
19
I am in germany, same issues after last update!

all my NTP servers fail, even my locally hosted ones, which never ever failed.
I have:
1. Raspberry with rtc module
2. A Nuc
3. my router
all offer ntp service locally, worked for years, Freenas and until the latest update Truenas also! now I face the same randomly!

I tried with command line to sync and when starting the service I get also out of sync errors!!

More info:
root@lila[~]# systemctl stop ntp; ntpd -gq; systemctl start ntp
20 Apr 08:44:14 ntpd[3843909]: ntpd 4.2.8p15@1.3728-o Wed Sep 23 11:46:38 UTC 2020 (1): Starting
20 Apr 08:44:14 ntpd[3843909]: Command line: ntpd -gq
20 Apr 08:44:14 ntpd[3843909]: ----------------------------------------------------
20 Apr 08:44:14 ntpd[3843909]: ntp-4 is maintained by Network Time Foundation,
20 Apr 08:44:14 ntpd[3843909]: Inc. (NTF), a non-profit 501(c)(3) public-benefit
20 Apr 08:44:14 ntpd[3843909]: corporation. Support and training for ntp-4 are
20 Apr 08:44:14 ntpd[3843909]: available at https://www.nwtime.org/support
20 Apr 08:44:14 ntpd[3843909]: ----------------------------------------------------
20 Apr 08:44:14 ntpd[3843909]: proto: precision = 0.039 usec (-25)
20 Apr 08:44:14 ntpd[3843909]: basedate set to 2020-09-11
20 Apr 08:44:14 ntpd[3843909]: gps base set to 2020-09-13 (week 2123)
20 Apr 08:44:14 ntpd[3843909]: Listen and drop on 0 v6wildcard [::]:123
20 Apr 08:44:14 ntpd[3843909]: Listen and drop on 1 v4wildcard 0.0.0.0:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 2 lo 127.0.0.1:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 3 br0 10.10.0.200:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 4 kube-bridge 172.16.0.1:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 5 kube-dummy-if 172.17.0.1:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 6 kube-dummy-if 172.17.0.10:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 7 kube-dummy-if 172.17.48.27:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 8 kube-dummy-if 172.17.29.70:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 9 lo [::1]:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 10 enp8s0 [fe80::1efd:8ff:fe73:c849%3]:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 11 br0 [2003:a:d5d:7901:a01f:2fff:fe5d:39a4]:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 12 br0 [fe80::a01f:2fff:fe5d:39a4%4]:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 13 vnet0 [fe80::fca0:98ff:fe53:ba18%5]:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 14 kube-bridge [fe80::8c8e:f9ff:fed2:2629%6]:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 15 kube-dummy-if [fe80::284b:45ff:fe7f:ab65%7]:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 16 veth529af4bc [fe80::1086:92ff:fe25:ac74%11]:123
20 Apr 08:44:14 ntpd[3843909]: Listen normally on 17 veth9bf0496b [fe80::ac44:45ff:fecf:ad3e%12]:123
20 Apr 08:44:14 ntpd[3843909]: Listening on routing socket on fd #34 for interface updates
20 Apr 08:44:21 ntpd[3843909]: ntpd: time slew +0.071217 s
ntpd: time slew +0.071217s
root@lila[~]# journalctl -f -u ntp
-- Journal begins at Thu 2022-03-24 12:43:37 CET. --
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: Listen normally on 12 br0 [fe80::a01f:2fff:fe5d:39a4%4]:123
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: Listen normally on 13 vnet0 [fe80::fca0:98ff:fe53:ba18%5]:123
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: Listen normally on 14 kube-bridge [fe80::8c8e:f9ff:fed2:2629%6]:123
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: Listen normally on 15 kube-dummy-if [fe80::284b:45ff:fe7f:ab65%7]:123
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: Listen normally on 16 veth529af4bc [fe80::1086:92ff:fe25:ac74%11]:123
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: Listen normally on 17 veth9bf0496b [fe80::ac44:45ff:fecf:ad3e%12]:123
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: Listening on routing socket on fd #34 for interface updates
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Apr 20 08:44:21 lila.lan.local ntpd[3844379]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Apr 20 08:44:21 lila.lan.local systemd[1]: Started Network Time Service.
 

Chewie198

Cadet
Joined
Jan 15, 2014
Messages
8
I have also been seeing intermittent, yet infrequent, NTP alerts with the same error message, even though I have redundant locally hosted time servers. I suspect that there is some sort of bug in SCALE causing this as I didn't have any problems with the latest version of CORE, and none of my other dozens of servers are indicating any problems with the NTP service. Fortunately I can just simply dismiss the alert and eventually it clears itself, although it would be nice to root out the cause of the issue. I would suggest opening a ticket with ix to address it if you haven't already. I've noticed numerous bugs so far during my SCALE migration and while none of them are showstoppers, it is definitely rough around the edges. You might wait and see if it's resolved with the point update next week and then go from there.
 

varet

Dabbler
Joined
Dec 6, 2021
Messages
19
The Problem I face is.. server goes out of sync! and then
1. S3 uploads/sync fails
2. Server is really logging/storing data with wrong times.

It is an issue!
If there is a next week or month update. worth wait.
 

LarsR

Guru
Joined
Oct 23, 2020
Messages
716
I had the same issue, but simply deleated the stock debian ntp server and set my own german ntp server and since then never had issues again
 

DavidinGA

Explorer
Joined
Jun 8, 2022
Messages
62
SInce this hasn't been flagged as SOLVED, I thought I'd share my experience this Father's Day (USA) experience ...
WARNING
NTP health check failed - No NTP peers: [{'216.229.0.50': 'REJECT'}, {'137.184.81.69': 'REJECT'}, {'195.171.43.10': 'REJECT'}]
2022-06-18 22:18:10 (America/New_York)
'ntpq -pn' times out.
 
Joined
Jun 2, 2019
Messages
591
Personally, I host my own NTP server on my pfSense firewall using us.pool.ntp.org, then add a firewall rule to redirect all outbound NTP requests (port 123) for clients I can't set the server.

This solves four problems.
1. Eliminates risk of getting blacklisted for too frequent NTP requests.
2. Eliminates risk of fingerprinting based on the NTP servers clients reach out to.
3. Eliminates differences since all clients are using the same local NTP server.
4. In the unlikely event internet goes down, all clients can still retrieve NTP time.

 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Is any one of the persons with failing NTP running TN SCALE virtualised?
 
Joined
Jun 2, 2019
Messages
591
In order to get the highest accuracy and low jitter, you need a serial GPS with a PPS signal. USB is a polling interface, not interrupt driven.

I've tried the Garmin 16X serial GPS connected to my Protectli pfSense firewall, but because the com port does not have the DCD pin to wire the GPS PPS signal, the jitter is all over the place. I just use a pool (us.pool.ntp.org) and get significantly lower jitter.

Screen Shot 2022-06-19 at 4.40.06 PM.png
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
So we can rule out the known clock skewing problems inside VMs.
 

DavidinGA

Explorer
Joined
Jun 8, 2022
Messages
62
In order to get the highest accuracy and low jitter, you need a serial GPS with a PPS signal. USB is a polling interface, not interrupt driven.

I've tried the Garmin 16X serial GPS connected to my Protectli pfSense firewall, but because the com port does not have the DCD pin to wire the GPS PPS signal, the jitter is all over the place. I just use a pool (us.pool.ntp.org) and get significantly lower jitter.

View attachment 56235
Would this work? https://austinsnerdythings.com/2021/04/19/microsecond-accurate-ntp-with-a-raspberry-pi-and-pps-gps/ I'm not obsessed with dirt cheap - an under $100. solution seems reasonable.
 

HeXXy

Cadet
Joined
Jun 28, 2022
Messages
3
I'm also running into this problem, however I tracked it down to at one point the DHCP client wrote out /run/ntp.conf.dhcp before I disabled DHCP. When /run/ntp.conf.dhcp is present, /etc/ntp.conf is completely ignored, I removed /run/ntp.conf.dhcp and it started working. Hopefully this helps.
 

DavidinGA

Explorer
Joined
Jun 8, 2022
Messages
62
I'm also running into this problem, however I tracked it down to at one point the DHCP client wrote out /run/ntp.conf.dhcp before I disabled DHCP. When /run/ntp.conf.dhcp is present, /etc/ntp.conf is completely ignored, I removed /run/ntp.conf.dhcp and it started working. Hopefully this helps.
root@truenas[/run]# ls -l
...
drwxr-xr-x 2 root root 100 Jul 8 10:13 network
-rw-r--r-- 1 root root 5 Jul 8 10:13 nginx.pid
drwxr-xr-x 2 root root 80 Jul 8 18:13 nscd
drwxr-xr-x 2 nslcd nslcd 40 Jul 8 10:13 nslcd
-rw-r--r-- 1 root root 4 Jul 8 10:13 ntpd.pid
drwxrwx--- 2 root nut 40 Jul 8 10:12 nut
 

Deeda

Explorer
Joined
Feb 16, 2021
Messages
65
Same issue, man the bugs are piling up. NTP issue, UPS issue and Reporting page not loading issue.
 

Attachments

  • output.png
    output.png
    9.5 KB · Views: 674
  • ntp.png
    ntp.png
    25.8 KB · Views: 674
Top