"Official" way to replace systemd-timesyncd with a "proper" ntpd?

[RR4711]

Does anyone know how the "proper" Debian style way looks like to replace the standard (for our application too crappy) systemd-timesyncd with the "real" ntpd?

I need proper "soft-pulling" of the system time and not this periodic update of the systemd-timesyncd (which may be fine for desktop applications).
Also I would need to be able to use ntpq or the API to check programmatically if the time is in sync with the master ntp server and what the offsets are.

Will there be issues with the realtime-kernel and possibly the communication with the modules? Anyone running a stretch image with "real" ntpd? Any issues with the RTC?

For now I just installed NTP with

"apt-get install ntp ntp-doc"

and it is running, but the timedatectl tool is still there and telling me the time is not synced

pi@sulfurpi:~ $ timedatectl
Local time: Tue 2020-04-07 09:32:04 UTC
Universal time: Tue 2020-04-07 09:32:04 UTC
RTC time: Tue 2020-04-07 09:32:06
Time zone: Etc/UTC (UTC, +0000)
Network time on: yes
NTP synchronized: no
RTC in local TZ: no
pi@sulfurpi:~ $

i assume it tries to get this information from the not-running systemd-timesyncd and silently fails. Does anybody know if that is intended behaviour?

ntp seems running ok:

pi@sulfurpi:~ $ ntpstat
synchronised to NTP server (134.34.3.19) at stratum 2
time correct to within 19 ms
polling server every 64 s

Am I overlooking something or is this how it is supposed to be? I'm a bit worried about the HW clock beeing off by 2 seconds. Should the Hw clock not be set to current system time on every shutdown?

Markus

[lukas]

We don't have experience ourselves with replacing systemd-timesyncd with ntpd, but what you've done looks fine to me, assuming that you've disabled systemd-timesyncd to prevent it from interfering with ntpd, i.e. "sudo systemctl stop systemd-timesyncd ; sudo systemctl disable systemd-timesyncd".

I don't know where the 2 second difference reported by timedatectl is originating from. I wouldn't trust timedatectl though to do the right thing when systemd-timesyncd isn't used.

As to your question whether the system time should be synced back to the RTC on shutdown: That's how it was done in the old days, with CMOS clocks on x86. Nowadays the kernel handles this itself behind the scenes. When ntpd or systemd-timesyncd invokes the adjtimex() system call, sync_rtc_clock() is called via do_adjtimex() -> ntp_notify_cmos_timer(). So an adjtimex() call results in immediate synchronization of the updated system clock with the RTC. In addition, sync_rtc_clock() reschedules synchronization every ~11 minutes.

The realtime functionality of the kernel is irrelevant in this context. It just means that latency is lower than on a vanilla kernel, mainly because interrupt handlers run as (preemptible) kernel threads and most spinlocks become sleeping locks.

[RR4711]

Thanks for the explanation. I play around with it and get a constant 2 seconds difference to the RTC after reboot, it takes NTP quite a few minutes to pull the system time to within 20ms of the NTP Server. I don't know what's going wrong there. I check the sync status with the ntpstat tool.

The 2 seconds also are there when the system has run for hours (Kernel had plenty of time to write RTC back according to your 11 Minutes statement, and I hope the RevPi RTC doesn't lose 2 Seconds in 11 Minutes). Something is fishy there...
After a few minutes things are stable but I need to have the time synced up very fast after bootup (we have an aicraft application and I don't have the time to wait 5 Minutes till the time is finally within spec) so I need to have the RTC as close to system time as possible. I need to play with hwclock utility and maybe manually set it and see if that makes a difference.

[lukas]

After spending a few days debugging this problem I've cooked up a kernel patch to sync the system time more precisely to the RTC. Could you provide me with your ntpd config so I can test it here in the lab?

[RR4711]

After spending a few days debugging this problem I've cooked up a kernel patch to sync the system time more precisely to the RTC. Could you provide me with your ntpd config so I can test it here in the lab?

Cool, that was unexpected

Sure, I'm afraid for now it's pretty standard with 2 Servers from PTB and one from NIST (most likely not optimal due to round trip time, but its a test setup only)

Later there will be one local NTP that sits in the aircraft's local LAN and is synced to GPS (I don't have access to that now, hopefully we can fly in a few weeks, it's all very complicated with the COVID-19 on top),

Code: Select all

# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift

# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable


# You do need to talk to an NTP server or two (or three).
server ptbtime1.ptb.de
server ptbtime2.ptb.de
server time.nist.gov

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
#pool 0.debian.pool.ntp.org iburst
#pool 1.debian.pool.ntp.org iburst
#pool 2.debian.pool.ntp.org iburst
#pool 3.debian.pool.ntp.org iburst


# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details.  The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Needed for adding pool entries
restrict source notrap nomodify noquery

# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust


# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255

# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines.  Please do this only if you trust everybody on the network!
#disable auth
#broadcastclient

[lukas]

The topmost commits on our revpi-4.19 branch seek to make RTC synchronization more accurate. They're still kind of experimental and need more tweaking, which I'll do over the coming weeks.

One thing I noticed during testing is it takes about 153 sec for ntpd to set the system time (and thus the RTC). So if the RTC doesn't contain a valid time, the system time will be wrong for about 2.5 minutes until ntpd updates it. It seems ntpd waits until it has acquired a sufficiently precise time. If the system is rebooted before those 2.5 minutes have elapsed, the system time will still be incorrect on the next reboot.

With systemd-timesync, the system clock is "already" updated after 41 sec, but subsequently declared unsynchronized again. Then at the 137 second mark it will again be updated and is permanently marked synchronized. So that's a somewhat odd behavior, but it means that the system clock and the RTC are updated faster than with ntpd if there's no valid time.

The supercap which powers the RTC on the RevPi will sustain the clock for a couple of hours, after that the 153 (or 41) sec delay must be endured upon the next bootup before the system clock (and the RTC) has a trustworthy value again. Just something to be aware of.

[RR4711]

The topmost commits on our revpi-4.19 branch seek to make RTC synchronization more accurate. They're still kind of experimental and need more tweaking, which I'll do over the coming weeks.

Thanks. Looking forward to test it. We may have first flights around 10th of May so that could work. Could you provide me a .DEB package for testing as I don't have the time right now to figure out how the kernel bakery works in detail as I'm busy with improving the application software.

One thing I noticed during testing is it takes about 153 sec for ntpd to set the system time (and thus the RTC). So if the RTC doesn't contain a valid time, the system time will be wrong for about 2.5 minutes until ntpd updates it. It seems ntpd waits until it has acquired a sufficiently precise time. If the system is rebooted before those 2.5 minutes have elapsed, the system time will still be incorrect on the next reboot.

That's why it is even more important that the RTC time isn't too far off. Typical scenario is everything is setup & tested inside the hangar running from ground power unit (the timeserver can synchronise itself to GPS via a GPS repeater in the hangar roof). Before rollout we shut everything down and power off, tow the plane outside and then usually switch everything back on after takeoff. Ideally this is less then half an hour, which should be easily backed up by the Goldcap.

With systemd-timesync, the system clock is "already" updated after 41 sec, but subsequently declared unsynchronized again. Then at the 137 second mark it will again be updated and is permanently marked synchronized. So that's a somewhat odd behavior, but it means that the system clock and the RTC are updated faster than with ntpd if there's no valid time.

Hmm, ntpd also makes fine corrections over time to compensate for drift. With systemd-timesync I expect to have "jumps" in my dataset's timestamp which is not what we want.

The supercap which powers the RTC on the RevPi will sustain the clock for a couple of hours, after that the 153 (or 41) sec delay must be endured upon the next bootup before the system clock (and the RTC) has a trustworthy value again. Just something to be aware of.

Has someone ever measured the "true" holdover time? I think 24 hrs is spec in the datasheet?

Markus