NTP no longer syncing after poweroutage

[DonZalmrol]

So, today they've came by to install a new digital energy meter for my home. Had to turn everything off and after booting up, I've noticed that my Truenas is no longer syncing with NTP, nor is it in my Windows active directory domain. When I try to re-join manually I receive the following fancy message

Time offset from Active Directory domain exceeds maximum permitted value. This may indicate an NTP misconfiguration.

Now I had this error before and it went away after restarting my NTP service, however now it seems to be stopping after a few seconds...

timedatectl output:

Local time: Fri 2023-04-14 23:30:26 CEST
Universal time: Fri 2023-04-14 21:30:26 UTC
RTC time: Fri 2023-04-14 21:30:27
Time zone: Europe/Brussels (CEST, +0200)
System clock synchronized: no
NTP service: n/a
RTC in local TZ: no

systemctl status ntp output:

ntp.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2023-04-14 23:14:11 CEST; 14min ago
Docs: man:ntpd(8)
Process: 51314 ExecStart=/usr/lib/ntp/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
Main PID: 51319 (code=exited, status=255/EXCEPTION)
CPU: 82ms

Apr 14 23:14:04 GRE-NAS-03 ntpd[51319]: Listen normally on 49 veth448737ad [fe80::c48b:2dff:fe34:a4fb%32]:123
Apr 14 23:14:04 GRE-NAS-03 ntpd[51319]: Listen normally on 50 vethbfcd679c [fe80::f02e:2bff:fef5:8188%33]:123
Apr 14 23:14:04 GRE-NAS-03 ntpd[51319]: Listen normally on 51 vethda915bbb [fe80::f425:aaff:fe8b:155b%34]:123
Apr 14 23:14:04 GRE-NAS-03 ntpd[51319]: Listen normally on 52 vethd864b194 [fe80::58d2:15ff:fe56:58c7%35]:123
Apr 14 23:14:04 GRE-NAS-03 ntpd[51319]: Listening on routing socket on fd #69 for interface updates
Apr 14 23:14:04 GRE-NAS-03 ntpd[51319]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Apr 14 23:14:04 GRE-NAS-03 ntpd[51319]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Apr 14 23:14:04 GRE-NAS-03 systemd[1]: Started Network Time Service.
Apr 14 23:14:11 GRE-NAS-03 systemd[1]: ntp.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 14 23:14:11 GRE-NAS-03 systemd[1]: ntp.service: Failed with result 'exit-code'.

When I start the service again, I notice the "kernel reports TIME_ERROR: 0x41: Clock Unsynchronized" error message:

● ntp.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-04-14 23:29:36 CEST; 1s ago
Docs: man:ntpd(8)
Process: 206307 ExecStart=/usr/lib/ntp/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
Main PID: 206312 (ntpd)
Tasks: 2 (limit: 154012)
Memory: 1.4M
CPU: 80ms
CGroup: /system.slice/ntp.service
└─206312 /usr/sbin/ntpd -p /var/run/ntpd.pid -c /etc/ntp.conf -u 114:119

Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: Listen normally on 47 vetha0eaf602 [fe80::145c:dbff:fed1:bb0e%30]:123
Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: Listen normally on 48 vetha1e190fb [fe80::c15:3eff:fe7f:9433%31]:123
Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: Listen normally on 49 veth448737ad [fe80::c48b:2dff:fe34:a4fb%32]:123
Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: Listen normally on 50 vethbfcd679c [fe80::f02e:2bff:fef5:8188%33]:123
Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: Listen normally on 51 vethda915bbb [fe80::f425:aaff:fe8b:155b%34]:123
Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: Listen normally on 52 vethd864b194 [fe80::58d2:15ff:fe56:58c7%35]:123
Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: Listening on routing socket on fd #69 for interface updates
Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Apr 14 23:29:36 GRE-NAS-03 ntpd[206312]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Apr 14 23:29:36 GRE-NAS-03 systemd[1]: Started Network Time Service.

It seems that my TrueNAS is not using the BIOS (CMOS) time nor using NTP.
My configured NTP servers are my domain controllers, I also tried with reverting back to the default NTP servers provided by IXSystems/Linux.

What I tried:

1. Changing the NTP servers to my ADs/ Public NTPs/ Firewall (PFSense with NTP)
2. Manually setting my time
3. Checking & changing my BIOS time (currently in UTC with GMT+1 for Europe\Brussels and DST enabled
4. Checked and confirmed my time on my AD controllers, my PDC is correctly working
5. Browsed the forum and googled to resolve the issue
6. ....

Currently I'm clueless. Any help (like always) is greatly appreciated!

My specs:

• TrueNAS-SCALE-22.12.2
• HP DL380 Gen9
• LSI SAS 9300-16i
• 128GB RAM DDR4 ECC
• Intel Xeon E5-2630L v3 @ 1.80GHz
• Nvidia Quadro P2000 5GB
• 2x Intel SSD 120GB boot-pool
• 2x Samsung SSD 512 mirrored pool for APPS storage
• 5x 14TB HGST DC HC530 for DATA storage
• CMOS battery is OK, it was replaced a while ago!

 

[Arwen]

The normal fix is to;

- Disable NTP time service
- Manually set time
- Enable NTP time service
- Wait til NTP time service has a chance to stabilize
- Verify time

Linux sort of uses the below to prevent the above situation. However, I don't know if all Linux distro use such, (mine is Gentoo).

- Start NTP client to set current time more precisely than internal clock
- Start the NTP time service daemon to maintain the time

But, it is possible that your local NTP server is also off, so it would need to be fixed first. Plus, their are plenty of things about NTP I don't know...

 

[Patrick M. Hausen]

Also make sure your BIOS clock is set to UTC and not local time. This is different from Windows systems and applies to both Linux (SCALE) and FreeBSD (CORE).

 

[blademan]

The normal fix is to;

- Disable NTP time service
- Manually set time
- Enable NTP time service
- Wait til NTP time service has a chance to stabilize
- Verify time

Linux sort of uses the below to prevent the above situation. However, I don't know if all Linux distro use such, (mine is Gentoo).

- Start NTP client to set current time more precisely than internal clock
- Start the NTP time service daemon to maintain the time

But, it is possible that your local NTP server is also off, so it would need to be fixed first. Plus, their are plenty of things about NTP I don't know...

Where do you disable NTP service in scale? The only option in System Settings > General > NTP Servers is to add servers.

 

[samarium]

Where do you disable NTP service in scale? The only option in System Settings > General > NTP Servers is to add servers.

Code:

system->shell
$ sudo -s
# timedatectl
# systemctl status ntp
# systemctl stop ntp
# ntpd -g -q
# hwclock --systohc
# timedatectl
# systemctl start ntp
# systemctl status ntp
# timedatectl
#
### come back tomorrow and check timedatectl says System clock synced and ntp service active and RTC clock matches UTC
#

 

[Constantin]

As an alternative, consider acquiring the NTP200 Series of GPS-controlled NTP clocks, or the Ntp250 if you need a POE solution. Very easy to set up, works as advertised, great hardware / software support from developer and much lower price point than Uputronics hat on a raspberry pi.

 

[blademan]

Code:

system->shell
$ sudo -s
# timedatectl
# systemctl status ntp
# systemctl stop ntp
# ntpd -g -q
# hwclock --systohc
# timedatectl
# systemctl start ntp
# systemctl status ntp
# timedatectl
#
### come back tomorrow and check timedatectl says System clock synced and ntp service active and RTC clock matches UTC
#

Thanks for the step by step. That worked immediately.

Code:

ntpd -g -q, sync'ed the UTC time to ntp

sync'ed the UTC time to ntp

Code:

hwclock --systohc, sync'ed the RTC time in the computer to UTC

, sync'ed the RTC time in the computer to UTC

 

[Constantin]

Having good local time is not just important for the NAS to operate as expected, it’s also about enabling 2FA. With some ISPs fiddling with port 123 on occasion, it was important to me to have a local source or two. I also configured my pi-holes to supplement the stratum 1 NTP servers by adding TCXO’s to them and setting them up as stratum 2 peers.

 

[elvisimprsntr]

I connected a Garmin 18X LVC to pfSense via a RS232 port and added a NAT rule to redirect all outbound port 123 traffic to the local NTP server. Eliminates the need to change every client to use a local NTP server. You can get a 18X LVC off evilBay for < $50. Requires a bit of wiring to provide 5v via a USB port.

 

[Constantin]

That’s a great solution. I did something similar with the firewall (redirects) as well as advertising the local NTP pool via DHCP settings. Whatever approach you take, the cost is pretty low compared to having 2FA go kaploink.

 

[samarium]

Nice. I have the cheapo solution, RTC clock chip in the Pi, and just set it to lower stratum.
I might even have a USB GPS somewhere I could use, maybe even gpsd on an old phone/tablet, or is there is an app for that?
I can always lie and configure the Pi, or any other computer, as accurate, and 2FA usually is 30 seconds window.
Can also route traffic via the 4G link to evade the ISP if necessary, or via VPN to somewhere more open, although the latency and jitter will probably kill that idea.

 

[elvisimprsntr]

I might even have a USB GPS somewhere I could use, maybe even gpsd on an old phone/tablet, or is there is an app for that?

For accurate GPS time synchronization, you need a GPS puck or module that has a PPS output, which is wired to the DCD pin of a RS232 port. This is due to the fact the GPS NMEA 0183 messages are not synchronized with an actual time tick.

Using a USB GPS or a serial GPS without the PPS signal will result in offset and jitter that is all over the place (+/-100 msec). With the PPS signal you can achieve <10 usec offset/jitter. Certainly not accurate enough for time stamping financial transactions, but good enough for homelab use.

ConfiguringNMEARefclocks < Support < Network Time Foundation's NTP Support Wiki

support.ntp.org

https://github.com/elvisimprsntr/pfsense-ntp-gps (WIP)

 

[i8degrees]

Hi,

I know this reply is several months old now, but I figured that I would chime in just in case it does help somebody else in the future.

The last post advice is spot on. You definitely should get a GPS module with PPS feature. But, you certainly can use any old Android phone paired with the gpsdRelay application available from F-Droid app store. The minimum Android version required is v5.x (as is the case on my Motorola phone). The quality of the GPS inside the phone does not much matter as long as you can pick up 2..3 satellites -- even if the location precision is poor! The app simply begins relaying your phone's GPS to a chosen remote gpsd daemon.

Note that you can even use multiple phones for the sake of redundancy here as the app connects to the remote gpsd daemon via UDP. I have two phones setup only because the Motorola phone will drop the WiFi connection after some number of days and the only fix I have found so far is by rebooting the phone.

I have my local NTP (well, chronyd) and gpsd darmon running on an old Netgear router that I have reflashed with a recent version of OpenWRT. There is no reason why you couldn't set it up anywhere else you might have in mind, though.

Here is the configuration I have been running on here for many months now without fail:

```shell
# /etc/config/gpsd
config gpsd 'core'
option enabled '1'
option device 'udp://YOUR_GPSD_ADDR:2998'
option port '2947'
option listen_globally' 0'
```

Note that YOUR_GPSD_ADDR will be the host IP of the device that is running gpsd. I also included a screenshot of my phone running the Android app, and it, too uses the same IP.

```shell
# /etc/chrony/chrony.conf

# Load UCI configuration
confdir /var/etc/chrony.d

# Load NTP servers from DHCP if enabled in UCI
sourcedir /var/run/chrony-dhcp

# Log clock errors above 0.5 seconds
logchange 0.5

# Don't log client accesses
#noclientlog

# Mark the system clock as synchronized
rtcsync

# Record the clock's drift
driftfile /var/run/chrony/drift

leapsectz right/UTC
makestep 1.0 3

#sched_priority 40

include /etc/chrony/conf.d/*.conf
include /etc/chrony/sources.d/*.sources
```

```shell
# /etc/chrony/sources.d/gpsd.sources
allow
refclock SHM 0 refid GPS precision 1e-1 delay 0.2
```
Note that you can use NTP in place of chronyd if you so wish. I chose chronyd because it simplified something else entirely -- I can't recall what exactly it even was at this moment! I am sure that I am forgetting some minor detail here in the post, but in any case, you ought to find tutorials scattered all around the Internet if you do a quick Google search.

The worst time delta difference I have seen when using this method is no more than ~1s. Certainly good enough for local TFA without dependence on WAN / Internet. This is what prompted me to investigate how to setup this in the first place. I have been very pleased with the setup.

Feel free to respond back, although it may take me some time for me to realize that I have a response to this post! Good luck and happy hacking!

 

[i8degrees]

Doh, last, but not least, I wanted to add to my last post that there is a `ncurses` shell interface that is distributed with gpsd called `cgps` that can help out with debugging the daemon. I like to run the command with the `--silent --units i --llfmt d` switches, but this is purely a preference. I have included a screenshot of the output of `chronyc sources` to demonstrate what are typical values you could expect with such a setup. Note how my GPS source has its last sampling recorded taking 200ms compared to time.cloudflare.com showing 0ns. In practice, I have found no issue whatsoever with such jitters.