Interesting NTP

[Constantin]

I recently acquired a NTP200 NTP server from centerclick.com, a company in NH. This seems like a good iOT use case as the device runs on about a watt of power and uses a simple puck antenna to get GPS signals.

Even though I am using the antenna indoors, it’s seeing 20+ satellites at a time and uses 14-18. To increase performance, I placed one of those Amazon Al-covered bubblewrap envelopes under the antenna (ground plane). Is working great, has a neat set of diagnostic pages, and simply does what it is supposed to. Should the satellite fix fail, the GPS module also incorporates a TCXO. For $159, this is a great deal. Some stats below (part of the webGUI) and obviously the jitter will be much lower once the initial spike has aged out of the graph.

Centerclick also offers the slightly more expensive NTP250, which is identical save the addition of PoE, which also makes it a dual-power supply unit. That POE capability is likely very interesting to those who want to use the NTP250 outdoors (though inside an enclosure). No doubt, the antenna would likely improve its performance even further if mounted outdoors, and high up. But given that I only need 4 satellites for a good time fix, I am perfectly content to leave this unit indoors.

A second NTP Pi hat from Uputronics will join this one for redundancy sake. It seems to use a more-capable GPS chip per the spec sheet (up to 72 satellites, multiple constellations, SBAS, etc.) but its RTC uses a uncompensated crystal. The cost point is about the same once you add the antenna, Pi, power supply, case, and so on. If you're only in the market for one local NTP server, I'd go with the centerclick solution - it just works, the price point is right, and the TCXO is the right component for the times that GPS signals are lost.

The NTP250 is an even better deal for those of us who want to mount the rig outdoors for a great signal, etc.

 

[Constantin]

Several months later, it just works. I’ll likely penetrate the case of the NTP200 a few times with a drill bit to enhance its convective cooling capacity. The NTP200’s internal temperature on top of my armoire reaches 100*F routinely and I don’t like that for the sake of any electrolytic capacitors inside.

The uputronics hat has since been encased in the custom case they also make for the RPi and hat combination. It’s likely not much cooler than the NTP200 but all the caps there are ceramic / tantalum, just like the RPi.

I never experienced issues with NTP servers and Comcast but at the same time also like having “Backups” given how sensitive the TrueNAS is to timing differences. The local ntp stuff is the primary time keeper and the Internet pool of ntp servers is the backup. Reduces the load on internet ntp servers and likely improves the reliability of my internal network also.

 

[elvisimprsntr]

I had a TimeMachine TM1000A, which got bricked during the GPS week rollover firmware update. I then tried connecting a Garmin GPS 16X LVS to my pfsense box, but I could never get the PPS signal to work. It would receive ~12 satellites, but without PPS, jitter was worse than just using a us.pool.ntp.org NTP pool. So I just implemented a FW rule to redirect all client NTP traffic to pfsense NTP server.

 

[Constantin]

From a hardware point, getting a clean NTP server set up is fairly trivial. The uputronics board illustrates that nicely with its very clean but also sparse layout. Not a lot going on. Nice thing about GPS dies is that you can connect to them serially, so they are relatively easy to deal with.

Regular clocks / crystals usually rely on a simple 32.768khz crystal that just happens to produce enough “ticks” every second to make a 16 bit register roll over. Funny coincidence! That arrangement is the heart of a Real time clock (RTC).

But simple oscillators should be thermally compensated for fine work, ie add a thermal sensor, bond it to the crystal, then adjust the number of ticks that cause the register to turn over based on the error curve of the crystal and the ambient temperature. Usually, RTC crystals are optimized re: error around 25*C.

TXCOTCXOs do all this compensation internally (to varying degrees of success) and typically also put out a RTC timing signal and a PPS. Many of these are programmed with I2C which means getting very familiar with the spec sheets. Depending on your price point, you can get very good TXCOTCXO performance.

the next step up are oven based systems where the oscillator is heated to a known-good temperature and kept there. Adds a lot of cost, heat, and power to a project but can be done by hobbyists.

The Uputronics appears to have a better GPS die than the NTP200 yet the NTP200 features a TXCOTCXO while the Uputronics features a simple oscillator optimized for use at 25*C. So in my view the Uputronics may offer better satellite performance (more constellations) while the NTP200 will behave better should the satellite signal get lost.

On balance, I’d prefer the NTP200 because it’s a purpose-built, efficient little rig that does exactly what it’s supposed to do and nothing more. For the most part, much better diagnostics, graphing, etc than the RPi solution. FAR better illustration of errors due to jitter and so on.

The only area where the RPi solution has even the slightest leg up is the satellite diagnostics, ie the ability to see which satellites the rig is getting signals from and hence being able to verify that the die is getting signals from multiple constellations. On balance, this feature is not as relevant as the status screens that the NTP200 has re: NTP performance.

 

[Ericloewe]

I missed this originally, but it's interesting and I have a few thoughts:

• Multipath can have a significant effect on location and time accuracy. Depending on your environment, mounting the antenna outside can improve things significantly.
• Four satellites is the absolute minimum to get a fix, since there are four unknowns (X, Y, Z and deltaT) and you get one equation per satellite. You can get below that if the receiver knows its position (e.g. it was previously determined and is stored on-board). As a sidenote, extra constellations are only useful if at least two satellites are visible, because every constellation has its own independent deltaT.
• The 30-60 minutes to get a cold fix they mention is a completely obscene amount of time. Does this match what you see?
• It's TCXO - Temperature-Controlled Crystal (Xtal) Oscillator

 

[rvassar]

I've had a GPS hat kicking around my desk for more than a year, waiting for me to build a Stratum 0 server. Last time I had stratum 0 at home, it was via 10 Mhz shortwave.

Sadly, it appears ntpd has been on shaky ground since Dr. Mills disability. Rather than shore up the project there's been some effort to split things up & recreate. ESR has a white paper out that I've looked at, but I'm not quite ready to jump to NTPsec simply because I have no experience with it. What's everyone recommending these days?

 

[Constantin]

Multipath can have a significant effect on location and time accuracy. Depending on your environment, mounting the antenna outside can improve things significantly.

Based on my very limited experience, this is very true in the context of inner city navigation, sometimes in canyons and like natural environments also. Hence the addition of dead-reckoning systems to car GPS receivers via MEMs or similar accelerometers. Usually not a consideration in more open environments with clear sky in most directions. My home stands on a hill, the roof above is rubber, so the opportunities for interference are relatively low.

There are external receivers that allow you to pipe a GPS signal from the outside into more shielded/problematic environments. I have no doubt that has an impact on the signal, etc. but better than nothing. Where possible, I like to keep receivers and other like equipment inside the building envelope, even if there is a performance penalty, in order to shield them from environmental degradation (sun, lightning, etc.).

As a sidenote, extra constellations are only useful if at least two satellites are visible, because every constellation has its own independent deltaT.

Absolutely. However, multiple constellations also provides a level of backup in case the US GPS system goes down altogether (Control Station hack, or whatever). Then there is Galileo, GLONASS, and Beidou.

The 30-60 minutes to get a cold fix they mention is a completely obscene amount of time. Does this match what you see?

That depends really on a couple of things. If you have a strong signal, a good viewing window, etc. then its ~12.5 minutes to download the almanac on a cold start that details where the GPS should roughly be looking for satellites. The more satellites in view the better as you'll have more chances to download the whole almanac before the satellite potentially drops out of view.

I found my NTP200 got a fix pretty quickly and that may have to do with the battery still having enough charge to keep the ephemeris, almanac, time, etc. data relatively intact from when the NTP200 came to me from NH. Once it has started up, the batteries are charged, and gotten a fix, all the relevant data for hot starts is maintained, and subsequent hot starts are almost instant. However, it takes some time to get the jitter into the ludicrously low range.

Both the NTP200 and the Uputronic RPi hat have backup batteries and RTCs to help them hot start quickly once the power is restored.

TCXO - Temperature-Controlled Crystal (Xtal) Oscillator

Thanks for the correction. For some reason I always write TXCO (it's been corrected above). That said, a TCXO is temperature-compensated, not controlled. You may have been thinking of a oven-controlled crystal oscillator (OCXO)?

Building a TCXO was quite instructive and I subjected the rig to dry ice and hot cycles to push my Teensy / DS18B20 / simple crystal to the limits of what the digital thermometer / crystal / etc. could manage. Not trivial unless you apply a lot of conformal coating to keep the ill effects of condensation to a minimum. But worked well enough. Least squares then allows estimation of a nearly-perfect parabola to compensate for temperature-induced crystal error. All that takes time. The TCXO folk do all that in the factory, laser-trimming the hardware to near-perfection before they put a lid on the chip.

 

[Ericloewe]

Thanks for the correction. For some reason I always write TXCO (it's been corrected above). That said, a TCXO is temperature-compensated, not controlled. You may have been thinking of a oven-controlled crystal oscillator (OCXO)?

You're right of course, I was thinking of "controlled for" and got the terminology mixed up. Come to think of it, what would also be rather cool is a setup with one of those PCIe atomic clock cards. Hilariously overkill for home use, and yet it's not inconceivable.

My home stands on a hill, the roof above is rubber, so the opportunities for interference are relatively low.

Sounds like it's not worth the hassle, then, as long as the signal is strong enough (which it clearly is).

 

[Constantin]

I know this will make WISP operators howl in indignation but after having two external PtP antennas fried by lightning I got smart, got higher gain dishes and simply installed the antennas indoors. They have to shoot through two walls, some brush + tree in the fresnel zone, 900ft…. and yet the signal is still good for 400+ Mbit/s, ie 4x of Comcast on a good day.

No doubt, the signal would be stronger if the antennas were mounted outside but with frequent lightning hits and a copper roof, lightning rods, etc the antennas are better off indoors.

 

[Ericloewe]

Neighbors increase multipath, but they do spread out the risk of a lightning strike...

 

[jgreco]

Sadly, it appears ntpd has been on shaky ground since Dr. Mills disability. Rather than shore up the project there's been some effort to split things up & recreate. ESR has a white paper out that I've looked at, but I'm not quite ready to jump to NTPsec simply because I have no experience with it. What's everyone recommending these days?

Hey, look, I can put one of my other hats on.

Hi, I'm the guy who does infrastructure engineering for NTP.ORG.

The Network Time Foundation supports the development of ntpd, but as with lots of free software projects, including things like FreeBSD, OpenSSL, etc., it has been poorly funded by those making money by integrating it into their products and businesses.

https://www.informationweek.com/sec...tackles-ntp-security-issues-as-big-move-looms

That was five or six years ago. I was the one that orchestrated and performed the vast majority of the migration from ISC to Markley Cloud in Boston, and then arranged for colocation space at ServerCentral in Chicago -- items discussed in that article. I then had one of my companies rotate out some older hypervisors, donated them to NTF in Chicago, where they've been doing lots of work for the last five years. Ironically, as I write this, migrations are underway to retire those, because they're now more than ten years old... HP stuff just keeps on ticking. In the meantime, Markley backed out of their agreement to provide cloud hosting, but I've arranged for more colocation space, and NTF now has three data centers for fully redundant operations, a bunch of donated hypervisors, etc.

Over at NTP.ORG, we're maintaining archives and producing documentation and looking at how to carry things forward, providing development environments, production testing, working with vendors of time sources, and all the real work involved in making NTP practical.

Many of us volunteer to keep NTP.ORG up and running; I've donated the services of the shop here to build and refurb gear at cost, and have done a lot of on-site work at distant sites at cost as well. I spent a career bringing the Internet to Wisconsin, then making it commercially viable, and then making it usable. I didn't get rich or anything, but along the way I've always tried to give back to the community. Everyone here on these forums know I share my knowledge freely and eagerly, but I think I make an even bigger contribution to the wellbeing of the Internet community by keeping the infrastructure for NTP.ORG running smoothly.

I wouldn't go jumping over to ESR's NTPsec, simply because ESR seems to feel NTPsec is a part-time side project and from what I can tell hasn't invested much time in trying to support the development and maintenance of the NTP ecosystem, which is much more than just a tarball to download and whitepapers to write.

ntpd is very stable software, but it is admittedly an old codebase with large portions having been written in a less secure era. This isn't really any different from many other really old codebases. But a lot of the issues people seem to have with "ntpd" such as DDoS attacks were really configuration issues rather than inherent issues with the ntpd software; default configurations from numerous vendors left ntpd wide open to queries, and that's ~~always bad for a UDP protocol. I will even concede that I have complained at length about the lack of clear documentation for people who are simply looking to implement timeservice and not have a minor degree in the complexities of time and ntpd config.

So I would answer this:

What's everyone recommending these days?

with "why, continue to use ntpd, of course."

 

[Constantin]

I agree with @jgreco that NTPD is the way to go for now and also want to send a huge THANK YOU his way. That's a lot of volunteer work to put in, that we all benefit from without an opportunity to acknowledge!

In the context of a home NTP system, exactly what would one gain with one of the alternatives to ntpd? The issues that have been identified in a broader security context are unlikely to have much applicability in a SOHO setting with one or two stratum 1 NTP servers serving the local subnet. Commercial enterprises likely use triple-redundant sources internally on a prefer/burst basis and external resources as a fallback.

An argument could be made for a government agency like NIST or FEMA to provide multiple NTP servers with reference atomic clocks dispersed around the country to serve as a fallback if GPS goes down. Simply too much depends now on a single, agreed-to time scale.

 

[jgreco]

I agree with @jgreco that NTPD is the way to go for now and also want to send a huge THANK YOU his way. That's a lot of volunteer work to put in, that we all benefit from without an opportunity to acknowledge!

You're welcome. I believe in a pay-it-back model where practical; I came to these forums to pay back my use of FreeNAS to the developers by helping provide some support. I do a lot of syseng but learned years ago that the finer points of Samba and AD and AFP were things I despised with a passion, so I try not to build those things myself anymore. These forums seemed like an ideal mixing of a bunch of my interests and experience, so participating here has been fun too. NTP is a bit different, in that it is critical to the functioning of the modern Internet, yet just like some other projects, it limps forward on meager funding from a handful of donors. I get a wry sense of pride that my business stands alongside some industry giants in supporting the Project, but mostly I do it because I enjoy a challenge.

In the context of a home NTP system, exactly what would one gain with one of the alternatives to ntpd? The issues that have been identified in a broader security context are unlikely to have much applicability in a SOHO setting with one or two stratum 1 NTP servers serving the local subnet.

For non-time-wonks, I doubt that there's much difference between them. Some clients out there still just periodically sync clocks with SNTP, and there is ntpd, which has been around since the '80's, and OpenNTPD, which IIRC was a rewrite, and Chrony, which is a from-scratch implementation that eschews being able to act as a Stratum 1 in favor of doing a better job of just doing it all over the network, and NTPsec, which is ESR's more recent fork of ntpd, with a bunch of ntpd's legacy stuff stripped out and added NTS. Each has strengths and weaknesses, targeted at particular use cases. If you were going to set up as a S1 clock, I suspect ntpd is the best place to be. There are time-wonk issues with both OpenNTPD and Chrony, as I understand it, but if it came down to it, running SOMETHING to sync your time is always better than not. And there are some client-only implementations that do not provide timeservice. Please note that I am regurgitating a bunch of stuff that I don't pay a ton of attention to from fuzzy memory, so please excuse errors.

Commercial enterprises likely use triple-redundant sources internally on a prefer/burst basis and external resources as a fallback.

An argument could be made for a government agency like NIST or FEMA to provide multiple NTP servers with reference atomic clocks dispersed around the country to serve as a fallback if GPS goes down. Simply too much depends now on a single, agreed-to time scale.

Um, they do.

https://www.nist.gov/pml/time-and-frequency-division/popular-links/web-clock-faq

I don't remember all the specifics.

 

[Constantin]

I was thinking more along the lines of sufficient infrastructure around US colocation facilities to handle all NTP requests rather than relying on charitable impulses from sponsors. As best as I can tell, a pro-quality, triple-redundant stratum 1 NTP can be put together for less than $1500. That's a pittance in the context of running a colocation facility. For $15K or so, you'd have three rubidium oscillators rubbing elbows and having a grand time.

Just like DNS servers, it's the kind of thing that all US colocation facilities should feature to make the internet more resilient.

 

[jgreco]

Well, probably not "colocation facilities" unless we're talking rinky-dink ma and pa shops. A high quality carrier neutral colocation center like Equinix or Dupont Fabros (now actually the same company) sells space and power, not bandwidth. There's really no guarantee that you can reach an arbitrary network within the building directly, depending on who you peer with or buy transit from -- that's the nature of the interconnection beast. The datacenter environment is sufficiently RF-noisy that you'd need to buy roof space to get GPS. And that's sort of the rub, Equinix is happy to sell that to you with some significant MRC, so there's little incentive for them to give it away.

Twenty years ago, during the heyday of the private non-evil-deathstar-cableco ISP's, it was actually fairly common for smaller providers to run their own NTP servers, and some of the surviving ones like XMission (see also here) and Sonic still do. I have no idea what percentage of their userbases actually make use of their local clocks. Both of those are open access though.

Smaller colocation providers have been drying up, thanks largely to "teh Cloudz", and most are no longer going to be interested in trying to offer value-add services that require cost PLUS some amount of technical knowledge to maintain. There are some exceptions, of course. For example, Hurricane Electric operates their two colocation centers in Fremont, and while they are not carrier neutral, they are pretty happy to let you peer with other networks, or several IXC's with presence there, AND they offer an NTP server (servers?).

The other factor is that it has become harder to offer open services like DNS or NTP without the risk of becoming a ... mmm let's call it an "attractive nuisance." When someone can turn your public service into a weapon against a third party, such as with DDoS, possibly rendering your service unusable in the process, well, that sucks.

So. I totally hear what you're saying, but I don't see a clear path to a win.

The NTP ecosystem works best when a large number of clocks with diverse sources can be used as the upstream time source.

It would be nice if every autonomous system on the Internet hosted a high quality stratum 1 clock, or better yet, at least three, with diversity in technology, and opened them for queries to the rest of the Internet, whether open or via arrangement. But this is expensive in NRC and potentially MRC; many of us operate networks where the optimal siting for such gear would be in a data center where the landlord is going to charge MRC for roof access.

The Internet's use of of a limited number of open access stratum 1 servers is probably a hazard. Not counting people running Raspberry Pi's on the end of a residential ISP line with DHCP, I am guessing that there's probably on the order of 100, possibly less, such clocks with long term stability.

 

[Constantin]

That was super illuminating and underscores my naïveté re: the realities of the underpinnings of the US internet. In my ideal world, every facility would offer NTP, DNS, and like Infrastructure Services to any local tenants that desire them. Perhaps limit responses to local tenants (Ie not the wider internet) but at least have some semblance of resiliency in case there is a crack in the space-time continuum. Doesn’t have to be rubidium either, even a OCXO-based, GPS-disciplined NTP server likely would hold up long enough to cover the most likely downtimes.

I looked into this a little further…Gepetto Electronics offers a RPi interface for a FE-5680a rubidium reference and a Skyworks GPS-time optimized receiver module that could be combined to discipline the FE-5680 to perfection. For less than $1000, you’d end up with a very accurate DIY time piece good for 100 days or so without GPS. Bit of a room heater though!

He also offers a GPS-disciplined 10MHz OCXO reference for a bit less money, less power and heat also. That might make a really nice combination with the RPi for a NTP server. The only “downside” is that there is no access to the skytrax NMEA output for troubleshooting purposes with this rig.

With either approach there is the issue of converting from exactly 10MHz to a nice 1PPS signal that the RTC on the RPi expects, but that should not be too hard with something like the PICDIV from leapsecond.