NIST tried to pull the pin on NTP servers after blackout caused atomic clock drift
(2025/12/21)
- Reference: 1766302841
- News link: https://www.theregister.co.uk/2025/12/21/nist_ntp_outage_warning/
- Source link:
A staffer at the USA’s National Institute of Standards and Technology (NIST) tried to disable backup generators powering some of its Network Time Protocol infrastructure, after a power outage around Boulder, Colorado, led to errors.
As explained in a [1]mailing list post by Jeffrey Sherman, a NIST supervisory physicist who maintains the institute’s atomic clocks, “The atomic ensemble time scale at our Boulder campus has failed due to a prolonged utility power outage.”
Sherman, whose LinkedIn bio proclaims he is “One of the few federal employee actually paid to watch the clocks all day,” says one impact of the incident “is that the Boulder Internet Time Services no longer have an accurate time reference.”
[2]
That’s bad because one of the things NIST uses its atomic clocks for is to provide a Network Time Protocol service, the authoritative source of timing information that the computing world relies on so that diverse systems can synchronize events. If NTP isn’t working, outcomes can include difficulties authenticating between systems, meaning applications can become unstable.
[3]
[4]
At this point, readers might wonder why NIST can’t just turn off the inaccurate service. Sherman said a backup generator kicked in and kept the servers running.
“I will attempt to disable them [the generators] to avoid disseminating incorrect time,” he wrote.
[5]
But the storms that caused the outage were so severe, only emergency services personnel are allowed to visit the site.
His post says he has seen “strong evidence one of the crucial generators has failed. In the downstream path is the primary signal distribution chain, including to the Boulder Internet Time Service.”
“Another campus building houses additional clocks backed up by a different power generator; if these survive it will allow us to re-align the primary time scale when site stability returns without making use of external clocks or reference signals,” he added.
[6]NIST discovers DevSecOps, thinks world should really check this out
[7]NIST trains AI to hear the 'oh crap' moment before batteries explode
[8]Techie fooled a panicked daemon and manipulated time itself to get servers in sync
[9]Indian government signals changes to infosec rules after industry consultation
Xcel Energy, the local utility, [10]blamed the outage on strong winds and as of Saturday night local time (7:00PM MT Dec 20, 2:00AM UTC Dec 21) advised most customers would have power again within three hours.
However at the time of writing – 00:15 MT Dec 21 – NIST’s [11]status page states the Boulder site is experiencing “Facility outages” and a “< 4.8us clock error.” That’s about four microseconds.
[12]
NIST [13]told CBS News it warned users such as telcos and aerospace organizations that weather around Boulder could cause problems, and advised them to tap the org’s other sources of time information.
That’s sound advice as best practice for using NTP is to specify multiple servers, and failover from troubled sources of time info to accurate ones. This incident therefore shouldn’t trouble the prudent, but may leave some NTP feed users unawares if they rely solely on the Boulder facility’s time feeds. ®
Get our [14]Tech Resources
[1] https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/o0dDDcr1a8I
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/07/31/nist_devsecops_guide/
[7] https://www.theregister.com/2024/11/18/battery_fail_sound_ai/
[8] https://www.theregister.com/2025/08/29/on_call/
[9] https://www.theregister.com/2022/06/14/indian_infosec_rules_possible_changes/
[10] https://co.my.xcelenergy.com/s/outage-safety/wildfires/power-shutoffs/event-update
[11] https://tf.nist.gov/tf-cgi/servers.cgi
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://www.cbsnews.com/colorado/news/power-outage-boulder-atomic-clock-nist/
[14] https://whitepapers.theregister.com/
As explained in a [1]mailing list post by Jeffrey Sherman, a NIST supervisory physicist who maintains the institute’s atomic clocks, “The atomic ensemble time scale at our Boulder campus has failed due to a prolonged utility power outage.”
Sherman, whose LinkedIn bio proclaims he is “One of the few federal employee actually paid to watch the clocks all day,” says one impact of the incident “is that the Boulder Internet Time Services no longer have an accurate time reference.”
[2]
That’s bad because one of the things NIST uses its atomic clocks for is to provide a Network Time Protocol service, the authoritative source of timing information that the computing world relies on so that diverse systems can synchronize events. If NTP isn’t working, outcomes can include difficulties authenticating between systems, meaning applications can become unstable.
[3]
[4]
At this point, readers might wonder why NIST can’t just turn off the inaccurate service. Sherman said a backup generator kicked in and kept the servers running.
“I will attempt to disable them [the generators] to avoid disseminating incorrect time,” he wrote.
[5]
But the storms that caused the outage were so severe, only emergency services personnel are allowed to visit the site.
His post says he has seen “strong evidence one of the crucial generators has failed. In the downstream path is the primary signal distribution chain, including to the Boulder Internet Time Service.”
“Another campus building houses additional clocks backed up by a different power generator; if these survive it will allow us to re-align the primary time scale when site stability returns without making use of external clocks or reference signals,” he added.
[6]NIST discovers DevSecOps, thinks world should really check this out
[7]NIST trains AI to hear the 'oh crap' moment before batteries explode
[8]Techie fooled a panicked daemon and manipulated time itself to get servers in sync
[9]Indian government signals changes to infosec rules after industry consultation
Xcel Energy, the local utility, [10]blamed the outage on strong winds and as of Saturday night local time (7:00PM MT Dec 20, 2:00AM UTC Dec 21) advised most customers would have power again within three hours.
However at the time of writing – 00:15 MT Dec 21 – NIST’s [11]status page states the Boulder site is experiencing “Facility outages” and a “< 4.8us clock error.” That’s about four microseconds.
[12]
NIST [13]told CBS News it warned users such as telcos and aerospace organizations that weather around Boulder could cause problems, and advised them to tap the org’s other sources of time information.
That’s sound advice as best practice for using NTP is to specify multiple servers, and failover from troubled sources of time info to accurate ones. This incident therefore shouldn’t trouble the prudent, but may leave some NTP feed users unawares if they rely solely on the Boulder facility’s time feeds. ®
Get our [14]Tech Resources
[1] https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/o0dDDcr1a8I
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2025/07/31/nist_devsecops_guide/
[7] https://www.theregister.com/2024/11/18/battery_fail_sound_ai/
[8] https://www.theregister.com/2025/08/29/on_call/
[9] https://www.theregister.com/2022/06/14/indian_infosec_rules_possible_changes/
[10] https://co.my.xcelenergy.com/s/outage-safety/wildfires/power-shutoffs/event-update
[11] https://tf.nist.gov/tf-cgi/servers.cgi
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aUfTTE7lnxrSRDd2pRkSWgAAAA0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://www.cbsnews.com/colorado/news/power-outage-boulder-atomic-clock-nist/
[14] https://whitepapers.theregister.com/
b0llchit
...and whole economies evaporate in a puff of high energy protons.
One can only hope that lightning strikes
JamesTGrant
Access to the generator, but no access to the room where the plugs and switches for all the gear sit?
NTP — remind me what the N stands for…
Bebu sa Ware
I would have imagined blocking the ports on the network switches through which the time services communicate with the rest of world might have been do·able remotely. Possibly not by a supervising physicist, perhaps by the local network engineer or BoFH.
Groo The Wanderer - A Canuck
NETWORK time protocol.
NETWORK.
That means it is inherently UNRELIABLE.
Phil O'Sophical
Unreliable and Incorrect are not the same thing. Servers may or may not be available (which is why it's a redundant network design), but when they are available they MUST be correct.
From the article:
This incident therefore shouldn’t trouble the prudent,
The prudent are a vanishingly rare species. They're never accountants, rarely policy makers, and seldom shareholders. Critical infrastructure is critical no matter how it's funded or how resilient it is, and too often the incautious learn far too late. The only difference between any past disasters and future ones is that our high tech world means we have fewer back up systems. The classic one is chimneys; houses don't have them anymore. In the old days if the mains or gas went off you could always burn something to keep a home warm. Now, you cannot!
The lunancy of those in financial control is that, at the same time they withold funding from propely beefing up critical infrastructure, they're probably personally diversifying their financial interests, investments, etc. The lunacy is that none of that diversification is worth a damn if the modern technological world suffers a major shock (e.g. a Carrington event) and whole economies evaporate in a puff of high energy protons.