GitHub rolls back database change after breaking itself
- Reference: 1723679533
- News link: https://www.theregister.co.uk/2024/08/14/github_rollback/
- Source link:
The Microsoft-owned code-hosting outfit says it made a change involving its database infrastructure, which sparked a global outage of its various services. The biz is now in the process of rolling back that update to recover.
"We are experiencing interruptions in multiple public GitHub services," the source code silo said in an [1]advisory on its status page. "We suspect the impact is due to a database infrastructure related change that we are working on rolling back."
[2]
What the GitHub.com homepage looked like today during the breakdown ... Click to enlarge
The downtime started just after 2300 UTC (1600 PT), according to GitHub. Affected services are: GitHub actions, pages, issues, pull requests, Copilot and Codespaces, packages, Git operations, and webhooks.
GitHub.com and the GitHub API were also unavailable – the website was showing just a unicorn and error message at one point – as was SSH-based access to repos.
[3]
"No server is currently available to service your request," an error message on the dot-com's homepage read earlier. "Sorry about that. Please try refreshing and contact us if the problem persists."
[4]'Error' in Microsoft's DDoS defenses amplified 8-hour Azure outage
[5]Microsoft's Azure networking takes a worldwide tumble
[6]Can't get Minecraft, MongoDB Cloud, others to work today? Blame that Azure outage
[7]Failure to follow proper procedures caused US-wide AT&T outage, FCC says
By 2329 UTC, GitHub made the decision to roll back its infrastructure change. And just now, at 2345 UTC, it's starting to right itself and return to normal.
"The database infrastructure change is being rolled back," the biz said in an update. "We are seeing improvements in service health and are monitoring for full recovery."
[8]
Happens to the best of us, clearly. ®
Get our [9]Tech Resources
[1] https://www.githubstatus.com/incidents/kz4khcgdsfdv
[2] https://regmedia.co.uk/2024/08/14/screennshot_github_down.jpg
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_specialfeatures/cloudinfrastructuremonth&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zr19Z3WlSz1sq7b5zolTPwAAAIw&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/2024/07/31/microsoft_ddos_azure/
[5] https://www.theregister.com/2024/07/30/microsofts_azure_portal_outage/
[6] https://www.theregister.com/2024/07/30/azure_outage_impact/
[7] https://www.theregister.com/2024/07/23/atandt_outage_fcc_report/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_specialfeatures/cloudinfrastructuremonth&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zr19Z3WlSz1sq7b5zolTPwAAAIw&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://whitepapers.theregister.com/
30 minutes to decide to do a rollback?
Classic. When I started a chain of actions that triggered a Hangouts outage (mostly internal to Google, thankfully, but still really bad), it took me less than 10 minutes to decide that. Seriously, if you last change corresponds to the start of an outage, now is NOT the time to say " [1]correlation does not imply causation ". You aren't Aristotle. Be [2]Beyes instead. Undoing that change is far more likely to fix things than to harm them.
[1] https://xkcd.com/552/
[2] https://en.wikipedia.org/wiki/Bayesian_statistics
Re: 30 minutes to decide to do a rollback?
I am thinking it might have taken 30' for the changes to propagate to the full extent of github's infrastructure.
From bitter experience hastily reversing an undesirable change midway through that change taking full effect is often just adding another locomotive to the train wreck.
Taking time to collect the evidence and think through everything before potentially compounding the felony is one of the hallmarks of experience.
BTW Bayes for the typo Beyes.
Take your bets!
Which was it this time? Was it DNS? Wrong window syndrome? Regex gone wild? Far fingered an rm -rf? Or did someone forget to pay the onion bhaji tax to the local BOFH???