Techie fooled a panicked daemon and manipulated time itself to get servers in sync
- Reference: 1756452610
- News link: https://www.theregister.co.uk/2025/08/29/on_call/
- Source link:
This week, meet a reader we'll Regomize as "Kim" who was asked to investigate an air-gapped lab in which time had become unhelpfully inconsistent.
"The problem was that an application server was more than five minutes out of sync with the domain controller that served as the lab’s Network Time Protocol (NTP) server," Kim told On Call.
[1]
His inquiries led to the discovery that all the servers were out of synch with the NTP server, and that while staff had known about the problem for over a year nothing had been done.
[2]
[3]
Higher-ups eventually got wind of the situation and issued an edict: The application’s timestamps must be within one minute of real time.
Kim dove into the problem again, and found the controller's NTP service configured correctly, but attempts to manually sync the machines saw the NTP daemon report it could not find any servers.
[4]
Which was odd, because the NTP server was clearly broadcasting the time, and other machines in the lab requested and received time information.
[5]Basic projector repair job turns into armed encounter at secret bunker
[6]Sysadmin cured a medical mystery by shifting a single cable
[7]Tech support team won pay rise for teaching customers how to RTFM
[8]Servers hated Mondays until techie quit quaffing coffee in their company
Kim eventually remembered that the NTP daemon has a "panic threshold."
NTP's job is to keep time and broadcast it so that clocks on all machines connected to a network are in synch. This matters for many reasons, among them consistency of log files – if every machine on a network kept different time it would be very hard to investigate an incident.
The panic threshold exists to stop a client of an NTP server synching if there's a big discrepancy between time as experienced by client and server.
The discrepancy on the network Kim tried to fix was well beyond the preset panic threshold, so the NTP daemon did as it was told and panicked.
[9]
The fix was simple: increase the panic threshold so the daemon stopped panicking.
Kim made the change, and declared to On Call that doing so made him an "Air-gapped time cop!"
Have you messed with time and space to fix tech? If so, hop in your TARDIS and [10]click here to send On Call an email so we can tell your story on a future Friday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aLF6PzSDfC_4SyVw9YSc4AAAAFQ&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aLF6PzSDfC_4SyVw9YSc4AAAAFQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aLF6PzSDfC_4SyVw9YSc4AAAAFQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aLF6PzSDfC_4SyVw9YSc4AAAAFQ&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2025/08/22/on_call/
[6] https://www.theregister.com/2025/08/15/on_call/
[7] https://www.theregister.com/2025/08/08/on_call/
[8] https://www.theregister.com/2025/08/01/on_call/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offbeat/columnists&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aLF6PzSDfC_4SyVw9YSc4AAAAFQ&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:oncall@theregister.com
[11] https://whitepapers.theregister.com/
That was almost too Prefect. Have a Pan Galactic Gargle Blaster on me.
Our current Mondeo has never given us a problem since we brought it 7 years ago. We often refer to it as the "Ford Perfect".
It's time to take a second to congratulate Kim
(it's early enough for a beer somewhere, including Vulture Towers)
been there, done that ...
Been in a similar situation.
Rather than chaning the panic treshold, stopping the ntpd stuff, manually setting the clock to within a few seconds of the correct time, starting ntpd did fix the problem
(be careful when doing this, if you set the clock forward this is usually fine, setting the clock backward will lead to all kinds of interesting and amusing phenomena
Re: amusing phenomena
Nice euphemism.
Re: been there, done that ...
Great ! a lesson in not messing with time from Mr. "I'm my own grandfather" !
Re: been there, done that ...
Every time I've encountered such a problem, I've just used the 'force update' parameter of the ntp client.
Under normal circumstances if the clocks drift enough to be out of tolerance then that indicates that something needs to be corrected so I'd be reluctant to change the threshold.
Re: been there, done that ...
Yeah, this... I'd've ntpupdated with the force option and seen if the problem re-asserts itself.
As you say, the option is there for a good reason, and lesson #43 in IT is 'Things are there for a reason'
I had a similar one. I once worked somewhere there was an Acopia ARX virtualising various Windows shares. The Acopia was setup to use the PKI infrastructure for time, the Windows clients Active Directory. I came in one Monday morning to find that the PKI clock had drifted sufficiently far from AD time (actually the correct one) that the clients would no longer talk to it.
It was a very quick fix, but took a while to figure out what's wrong as you kind of assume PKI would be correct...
You know what they say about "assume" . . .
That every time you say it someone will gleefully trot out that exact tired response , like polly wanting a cracker, regardless of context and whether assuming was strategically justified?
Sometime you just have to assume, whether its because its 99.99% likely correct , or because its an intermediate measure while you diagnose more likely culprits.
( tbf the response may have been warranted in this case )
Life is full of assumptions. The trick is to understand when and where you are relying on the assumption and what the consequences of the assumption being wrong might be.
Spent ten years of my (working) life drilling the importance of fully documented and managed Risks, Assumptions, Issues and Dependency (RAID) registers into project teams for one of the (then) major outsourcing players. Some folks never really got it no matter how hard we worked.
> You know what they say about "assume" . . .
No, I don't actually. But you assuming that I do probably says something.
[1]Obligatory XKCD
[1] https://xkcd.com/1339/
>> You know what they say about "assume" . . . <<
Hi Mum!
Lotus Notes
I had one as a contractor where people were intermittently unable to retrieve their mail.
As I recall the clients were attached to a Novell Network, and the notes was on a different server (neither my responsibility) managed by different teams, and so the time had drifted between them.
Once I let them know the Novell server was adjusted to be more correct and mail started working again..
I don't know if they actually got NTP or anything like that working on it .. like I say not my job at that place
My NTP-AD default...
Apart from the obvious, to actually configure ntp servers via GPO, I change "MaxAllowedPhaseOffset" to 30 seconds. The default of up to 15 minute tolerance before doing anything lead too often to the effect of being between five to ten minutes off over several days, which is irritating in many of todays environments. The rest "default stuff" is fine.
Some special picky services get less than 1 second as tolerable offset, but that is rarely needed.
Re: My NTP-AD default...
AD joined machines do not need to have an NTP server set. They automatically sync with their DCs. Only the root DC, it needs an external time source.
Since Kerberos tokens are time based, a domain will work properly as long as machines are in sync with DCs, even if the DCs time is wrong.
Today with IAMs using time-based tokens, syncing time became far more important than log correlation, authentication may fail and services stop working.
MaxAllowedPhaseOffset only controls if the time is adjusted immediately or gradually.
https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-time-service-tools-and-settings
This is bringing back vague memories of adjusting the time (date) in order to get software installed .
Cant remember why , might have been something to do with trial periods
Norton used to do a 90 day trial which I regularly extended to about 10 years by setting the clock forward before installing it.
Errmm, "allegedly"
Those installs all expired long ago though.
I remember it being a cheat on certain Xbox games (Fable titles iirc) where advancing the console system date by a few years would break the time tracking built into the game, showering you with money from any income you had set up.
So the NTP server was essentially told the advice on the cover of the Hitchhikers' Guide to the Galaxy (in large, friendly letters).
I'll get me coat. The one with the cassette tapes of the radio plays of the HHGTTG in the pocket, please.