NTP not initializing

[David Dyer-Bennet]

Like, um, dozens of other people all over the net, I'm having NTP issues. The clock on my NAS sometimes drifts (and then this starts causing trouble with git repositories shared from there, or otherwise becomes noticeable). I've seen these issues off and on for years, by the way.

This is on my home network. FreeNAS is sometimes exhibiting clock drift, and on examination it seems to be that it never manages to initialize a relationship with any server. However, Windows and Linux boxes on the same LAN, behind the same very minimal firewall (on an ordinary ASUS AC-3200 home router), all track NTP just fine, and furthermore if I disable ntpd, then using ntpdate on the FreeNAS box also works (this seems to me to show that it's definitely not anything blocking the packet path between my FreeNAS box and the Internet for NTP).

I've read around a bit, and I have put in place (last night, so it's had many hours to init) the kern.timecounter.hardware change recommended (after verifying it was valid on my hardware. (It was originally TSC, before I changed it.)

Code:

fsfs% sysctl kern.timecounter.choice
kern.timecounter.choice: ACPI-safe(850) HPET(950) i8254(0) TSC-low(1000) dummy(-1000000)

fsfs% sysctl kern.timecounter.hardware
kern.timecounter.hardware: HPET

While the clock has not drifted that I can see by eyeball comparison since then, ntpq does seem to be telling me that NTP is NOT happy.

Code:

fsfs% ntpq -pwn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 45.79.192.248   .INIT.          16 u    - 1024    0    0.000   +0.000   0.000
 45.33.84.208    .INIT.          16 u    - 1024    0    0.000   +0.000   0.000
 154.16.245.246  .INIT.          16 u    - 1024    0    0.000   +0.000   0.000
 108.61.73.244   .INIT.          16 u    - 1024    0    0.000   +0.000   0.000
fsfs%

As I understand that, it says it is trying to initialize with all 4 servers it picked, but it has never (in nearly 12 hours) managed to ever reach any of them. Pings to them work fine, it's not a basic network connectivity issue.

I'm still using the default NTP server config, 0.us.pool.ntp.org through 3.us.pool.ntp.org.

So, any ideas for my particular case? And...why are we still seeing people having NTP problems on FreeNAS in particular so many years after it became a visible issue?

 

[elvisimprsntr]

Appears the NTP servers are not reachable and/or your WAN IP may be blocked due to frequent requests. Have you checked your global network settings to make sure you have correctly set the gateway and/or DNS server? Also, some ISPs (ATT) block port 123 source port, thus if your clients use port 123 as the source port, it would get blocked. This is to prevent residential customers from incorrectly setting up a WAN facing NTP server.

I run my own NTP server on my pfsense firewall and add a firewall rule to route outgoing NTP port 123 traffic to the localhost IP of my firewall.

NTP traffic is not encrypted, thus malicious actors can finger print what type of kit you have behind your firewall based on the NTP servers it reaches out to. By routing all NTP traffic to your own NTP server, the only kit that reaches out is your firewall. Also keeps all your kit in sync within a few microseconds.

 

[jgreco]

Due to historical amplification attacks, there are strange and wonderful port 123 blocks in a variety of places throughout the Internet that might astonish you. Some ISP's will not let you use NTP servers that are not their own, others have found their transits have blocked various stuff, etc.

The, ah, annoying bit is that some of the port 123 blocks tend to prohibit communications with ntpd because ntpd likes to use 123 as its local port while ntpdate uses a random high numbered port; this lets ntpd establish a peering and handle clients all using the same socket.

Are you using outbound NAT? Is the outbound NAT retaining the source port 123? If so, can you cause the NAT to randomize the port number and see if that works? If so, the problem's upstream from you, but randomizing the port number is probably sufficient as a fix.

 

[David Dyer-Bennet]

Are you using outbound NAT? Is the outbound NAT retaining the source port 123? If so, can you cause the NAT to randomize the port number and see if that works? If so, the problem's upstream from you, but randomizing the port number is probably sufficient as a fix.

It's a normal home installation, so it's all behind NAT. Is "outbound NAT" something special? I don't know the term; don't recall seeing it in the router menus for example. Normal NAT has to change the source port doesn't it? Otherwise there would be conflicts among the systems it serves?

There is some evidence that it's port-related, after some simple tests. ntpdate (after stopping ntpd) fails consistently saying no server available when I just give it a server name, but in -d mode it succeeds...and -d mode uses a non-privileged port. Dropping out one more variable, ntpdate -u works when ntpdate alone does not. That points very specifically at the port being somehow the issue.

I've asked my ISP about NTP, and about whether they run servers. For my use, it's not really a problem to just change over to using their server(s), if that's an option (can't find any info on their web page about ntp).

 

[David Dyer-Bennet]

Appears the NTP servers are not reachable and/or your WAN IP may be blocked due to frequent requests. Have you checked your global network settings to make sure you have correctly set the gateway and/or DNS server? Also, some ISPs (ATT) block port 123 source port, thus if your clients use port 123 as the source port, it would get blocked. This is to prevent residential customers from incorrectly setting up a WAN facing NTP server.

The timing params on the ntpq output seem to show I'm not making "fequent" requests, nowhere close to the 4-second limit commonly mentioned.

Access to the WAN in general has been fine for 6 years in this setup, and remains fine today. DNS resolution works fine. I can access, for example, www.truenas.com from my web browser (as I'm doing right now). I have no reason to be suspicious of any of that. All the evidence points to something specific to NTP (quite possibly port-related).

I've asked my ISP if they're doing anything weird, we'll see about that.

Meanwhile, is there a way to get ntpd to not use port 123 for it's requests? And in particular a way to configure that into FreeNAS? Some other tests do suggest that this problem here is port-related, see my other recent reply in this thread.

 

[David Dyer-Bennet]

And in fact I have found documentation that my ISP blocks inbound UDP packets to port 123. That does leave the question of why NTP seems to work for every other OS in the house!!!! (Presumably, everybody else has recognized that the well-known port is widely blocked and aren't using it.) Also testing using ntpdate (with ntpd stopped) and the -d and -u options verifies that it works fine when using a random port but does not work at all when using port 123.

So, is there a way to get NTPD to use some other port? Is there a way to configure this in FreeNAS? And -- why is the default something that doesn't work in many locations? Is there any downside to using a non-priv port for the NTP queries from ntpd?

Edited to add: looks like NTPD is an asshole about this, no way to make it use another port, according to https://stackoverflow.com/questions/232722/ntpd-use-an-unrestricted-port-for-communication

 

[elvisimprsntr]

The timing params on the ntpq output seem to show I'm not making "fequent" requests, nowhere close to the 4-second limit commonly mentioned.

Access to the WAN in general has been fine for 6 years in this setup, and remains fine today. DNS resolution works fine. I can access, for example, www.truenas.com from my web browser (as I'm doing right now). I have no reason to be suspicious of any of that. All the evidence points to something specific to NTP (quite possibly port-related).

I've asked my ISP if they're doing anything weird, we'll see about that.

Meanwhile, is there a way to get ntpd to not use port 123 for it's requests? And in particular a way to configure that into FreeNAS? Some other tests do suggest that this problem here is port-related, see my other recent reply in this thread.

What kit do you run as your firewall? Perhaps you can enable your own LAN side NTP server and point clients to it or add a firewall rule like I suggested?

 

[Constantin]

If you have access to a space under a roof, I'd consider installing your own NTP servers. I have one from centerclick, the other is running on a RPi

The NTP200 and NTP250 (POE powered) are set and forget, very simple appliances that simply get the job done. The RPI + Uputronics hat were a lot more challenging to set up but may offer more constellations and a better crystal. I got one of each since that gives some redundancy. The internet is the backup to both. I used Amazon metallized bubble-wrap bags as a ground plane below the puck antenna.

Neither antenna is installed outdoors and both receive 20+ satellite signals. Here is a NTP200 status graph as of this afternoon despite heavy cloud cover associated with hurricane Henri and all that.

 

[David Dyer-Bennet]

Would be cool to install my own tier-1 server, but it's one more thing to fail. I can't claim under $200 is "unaffordable", but I can't just burn 4 Benjamins casually, I'd notice the loss. And my ISP has carefully blocked things so I can't share it with outsiders.

I'm kind of annoyed that I can't just turn on a server in my household router/wifi AP; that should be a basic function in all of them (along with DNS masq). (It would have to use a port other than 123 of course for its external queries, but there are multiple choices for that, just not classic ntpd).

 

[David Dyer-Bennet]

Adding my own NTP server is something I'm avoiding fairly strenuously; it's another thing to fail and to maintain. If I could turn one on on my household router that would be good, but the ASUS AC-3200 doesn't have that option.

There's no real firewall, no rules configured in the AC-3200. Behind NAT, there's not much need. Don't see how a firewall can help me given that we've determined that the problem is that packets returning from the outside NTP servers are being blocked by my ISP when directed to port 123. If I could grab *outbound* packets and change their *source* and then recognize the UDP reply to them and change it back, that might help.

 

[elvisimprsntr]

My firewall is a NTP client using a pool of WAN facing NTP servers, each instance uses a random source port, thus not blocked by my ISP. Firewall then runs as a NTP server using its own acquired NTP time. All clients point to my firewall for NTP. The firewall rule forces all destination port 123 NTP requests to my localhost firewall NTP server. This prevents any clients from trying to access a WAN NTP server, even using a hard coded IP address. Doesn’t require any new hardware. Just enable NTP server in your firewall. Any decent router should allow you to run a NTP server. If the OEM firmware doesn’t, there are plenty of open source router firmware that you can flash over the OEM firmware. Example OpenWrt. I personally stepped up my game 7+ years ago when I switched to open source enterprise class firewall software http://pfsense.org

 

[David Dyer-Bennet]

Perhaps "any decent router", but not the ASUS ones I've seen, and they're among the better home ones I thought. And, in particular, not the one I'm currently using, an AC-3200.

The OpenWRT page doesn't list the AC-3200 as supported. I did experiment with Asuswrt-merlin for a while, but that didn't work with (or I couldn't figure out how to configure) the short-cut hardware routing necessary to get useful packet rates at 1gigabit, so that wasn't workable. (I use network disk a lot internally, but even the external path is gigabit, we've got gigabit fiber internet here.) I think it did offer ntp, probably, but since it didn't support useful packet rates, I switched back.

I've written commercial router internals (at Network Systems, who ended up not making it in the marketplace; they started as a company selling pre-standards ways to do cross-vendor networking) and configured rather complex test environments requiring multiple enterprise routers, so I'm not exactly "afraid" of going deep, but I don't have a lot of money to spend, and I don't want more pieces of gear that have to run 24/7 (and there are 3 other adults here that are also pretty deeply dependent on the Internet).

Even just adding a separate household router (easy to do; the interface to my ISP's fiber is a gigabit RJ-45, no authentication or anything, that's all handled in their hardware) would be an additional mission-critical box, another thing to go down, another drain on the UPS during power failures, etc. Replacing the household router with one box that was both a better router and a good-enough Wifi AP would avoid that problem, but would involve spending a few hundred dollars, which is not an amount I can do really casually. It doesn't really feel tempting just for this; I can hack something to cause ntpdate to run hourly, or something, and that's plenty good enough for my practical uses. Just annoying. Plus no doubt a few days of my time, spread over a week or so, to actually get it working right.

Still -- if I were going to go to a new router/Wifi box, whats a good one that handles gigabit throughput well, supports NTP usefully (syncs itself to outside servers without using port 123, presents port 123 to inside servers to sync to), and while we're at it does DNS-masq?

 

[jgreco]

If I could grab *outbound* packets and change their *source* and then recognize the UDP reply to them and change it back, that might help.

Well, that's what I asked about. A NAT engine has the option to rewrite both the IP address and port number, and retaining the port number actually requires more effort, because you have to handle the case where multiple devices are sending out with the same port number (in which case only one of them can get it). I would suggest looking at your NAT gateway's configuration options...

 

[David Dyer-Bennet]

Well, that's what I asked about. A NAT engine has the option to rewrite both the IP address and port number, and retaining the port number actually requires more effort, because you have to handle the case where multiple devices are sending out with the same port number (in which case only one of them can get it). I would suggest looking at your NAT gateway's configuration options...

No such controls anywhere I've seen them, anyway.

 

[Constantin]

I tried using my edgerouter as NTP server and either by sheer incompetence or some other mishap then sent my NAS into a timing tizzy. Everything broke down with transfers slowing to a crawl, etc.

No doubt, I had misconfigured something since the edgerouter should be capable of being a NTP server. But I also realized that if small timing errors could upset the Apple cart this badly, then it would be good to have a internal NTP pool also.

so I set up two different NTPs for some redundancy as relying on just the US satellites will also create a single point of failure. I don’t expect much upkeep as one appliance is purpose built and the other is a RPi running off the USB port on a Apple base station AP.

The NAS also uses external resources as a backup in case the internal ones fail.

The only devices I force to use the internal NTP resource (as opposed to encourage via DHCP settings) is my Sonos gear, which is categorically not allowed to contact the mothership other than to renew the SSL certificate.

 

[Heracles]

Adding my own NTP server is something I'm avoiding fairly strenuously; it's another thing to fail and to maintain.

Unfortunately, that is the best and easiest solution... As mentioned, too many NTP requests from a single IP will have you blacklisted easily. Here, I have a multi-site VPN for supporting family members. On 3 of these sites, I have pfSense firewalls that sync NTP with Internet pools by themselves. I then joined them in my private NTP pool and that is the one used by every other system in each of my sites. As such, there is only a single NTP client going out from each public IP I have. The risk to end up blacklisted is low.

With 3 references for myself, my private pool will identify and put aside one of them should its time be wrong.

Another benefit to use your own NTP internally is to ensure your own systems stay all together even when NTP is not that reliable. Should you reference ends up late or early, your entire network will remain consistent and will have the same gap. You will not be consistent with the world but will still be with yourself.

There are docker containers that can be used quickly and easily to deploy NTP service internally. You said that you have some Linux in your hands. Grab one of them and you should be in a much better position in no time.