Cigarette break burned out a huge chunk of Africa's internet
- Reference: 1723448166
- News link: https://www.theregister.co.uk/2024/08/12/who_me/
- Source link:
This week our hero is someone we'll Regomize as "Paton" who, several decades ago, worked for a prominent ISP in South Africa. Indeed it was a very prominent ISP, as its DNS servers were authoritative for thousands of domains including country code top-level domains (ccTLDs) in South Africa and elsewhere in the region. This will be important later on.
Paton worked as a "backbone engineer" which, particularly in the context of such a crucial ISP, sounds like a seriously important job. Not the sort of thing one does in a hurry, or while distracted. This, too, will be important later on.
[1]
One of Paton’s responsibilities was maintaining Access Control Lists (ACLs) specifying which users and domains had access to certain network resources. The ACLs in Paton's organization were extensive and complex, as you might imagine. They controlled not only the netblocks for the ISP's customers, but also for its internal infrastructure – including those authoritative DNS servers.
[2]
[3]
One fine afternoon, Paton was handed the task of updating the netblocks and propagating them via BGP to peers and transit providers on the network. This meant editing the ACLs with the new information – not a difficult task.
However, he was under pressure from a group of his colleagues, who were keen for him to join them outside for a bit of fresh air and a cigarette (somewhat negating the benefit of the fresh air). So Paton hurried – which, as noted above, he should not have done – and got the job done quickly so he could go enjoy that rich tobacco flavor.
[4]Hello? Emergency services? I'd like to report a wrong number
[5]Never put off until tomorrow what someone could erase today
[6]Facebook prank sent techie straight to Excel hell
[7]There is no honor among RAM thieves – but sometimes there is karma
When he got back, not very long afterwards, he found what he described as "pandemonium and chaos." The Network Operations Center was being flooded with calls from irate customers as a result of what was, to that date, the largest internet outage to hit the African continent.
To make matters worse, someone pretending to be a hacker contacted a local tech news service and claimed that it was their l33t skillz that had caused the outage. The news outlet ran with it, meaning Paton's bosses had both the technical problem and a potential security headache to deal with – on a multi-country scale.
[8]
It transpired, of course, that there was no hack. The ISP's security had not been breached. But that was hardly of comfort from a PR perspective, given the actual cause.
In adding to the ACLs in his haste, Paton had inadvertently replaced them. All of them. So the new netblocks were fine, but none of the other complex and interlocking control lists directing internet traffic for much of sub-Saharan Africa were in place.
Paton of course had to restore the ACLs, as well as updating the netblocks. Then, for his sins, he got to write the organization's very first change-management protocol. No doubt it included a bit about checking that you haven't wiped half a continent off the internet before you get a smoke.
[9]
OK, not everyone's little mistakes end up blowing up to nationwide scale. But if you've ever done one little thing wrong and it ended up being a really big thing, we'd like to hear about it. [10]Click here to send an email to Who, Me? and we may share your story on some other manic Monday. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZrndRplnzNgoOkMSF2pNdQAAAA8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZrndRplnzNgoOkMSF2pNdQAAAA8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZrndRplnzNgoOkMSF2pNdQAAAA8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2024/08/05/who_me/
[5] https://www.theregister.com/2024/07/29/who_me/
[6] https://www.theregister.com/2024/07/22/who_me/
[7] https://www.theregister.com/2024/07/15/who_me/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZrndRplnzNgoOkMSF2pNdQAAAA8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZrndRplnzNgoOkMSF2pNdQAAAA8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] mailto:whome@theregister.com
[11] https://whitepapers.theregister.com/
Re: Bad luck, really
To Err is Human
Re: Bad luck, really
To Covfefe is Trump
Murphy's Law Corollary #13
The more innocuous a change appears, the wider reaching its effects
Own up to it
At least our backbone engineer wasn't a spineless coward, and owned up to his mistake.
Re: Own up to it
But realistically, a change like that should require a detailed process document - with screenshots, examples and backup strategy - as well as having two pairs of eyes to cross check everything before committing the changes.
Re: Own up to it
Shoulda Woulda Coulda
always plays in my head when I do changes.
Sadly best practice of proper documentation, four eyes etc were not common 20-30 years ago. ITIL made a serious impact when adopted.
Ahh a manager who is telling the truth
I was staff to a senior manager. One day he wrote an email to the managers, giving the raw truth(all the bad news) about the status of the project we were all working on, and gave an action plan. He started typing the distribution list devm... expecting it to word complete to devmanagers, but he typed devg for devgroup and pressed return. He realised his mistake when the email system said sent to 600 people.
The manager asked me to go down to IT to stop the email from being distributed. Too late!
The outcome was better than expected.
Several people dropped into his office to say things like .. "Good email. We knew we were knee deep in the brown stuff, it is nice to know that you know" "That was a brave email - good for you".
People started giving a better truth such as "we are two weeks behind schedule" rather than "we are almost there". Instead of people sitting around whinging how late the project was, they got on and fixed it!
As a management team we thought "phew we were lucky it wasn't worse"
Re: Ahh a manager who is telling the truth
Transparency in organisations is a rare thing.
Accurate Status-Reporting Upward
... is inhibited by managers whom, upon receiving bad news about the project's progress-vs-the-schedule, leap into action via drive-by micromanaging and "doing things themselves".
yeah I've forgot add when configuring vlans on important links before, always fun
created a directory
I was diagnosing an authentication issue on a newly installed live server cluster and created a temporary directory and copied some log files into it for further analysis. What harm could that do? I also did the same on the other node of the cluster. Went off to lunch and apparently all hell broke out. Over lunchtime an automatic code replication process ran and tried to get rid of my unexpected directory but couldn't due to permissions and crashed leaving the server in a half upgraded state. The other node did the same. A colleague of mine who wasn't at lunch spotted the issue and silently deleted my directories and fixed the servers. We blamed the issue on the original problem under investigation and so no one ever found out what really happened! We also fixed the scheduling of the replication process so if one node failed the other node wouldn't try and do the same, so in fact it was a very good "test"...
Re: created a directory
At least it happened before it was deeply into critical production territory...
False alarms also count as alarms
Many jobs ago, our NOC monitoring systems were sitting in a /25 subnet routed by a pair of Catalyst 5500 switches running IOS in a bolted on RSM module (I said it was many jobs ago). This subnet was protected by an ACL, a few hundred lines long, that required semi-frequent changes. These changes sometimes went wrong, so at any given point we had two ACLs on the boxes - 150 and 151, one applied and the previous one for backup. Our procedure was to copy the currently active ACL into a text editor, renumber it from 150 to 151[0] (or the other way around), delete the currently unused backup one, paste the new one to the routers and then change the interface configuration to use the newly updated one. Simple, right?
Well, there were always people (I'll admit it: including myself, once or twice) that messed up and accidentally deleted the currently active ACL[1] by mistake - which, in Cisco terms means "as long as the applied ACL is empty, all traffic is allowed". This in itself is not a huge issue since you'll be pasting in a new ACL version a few seconds later. But that's where the fun begins. Remember the "as long as the applied ACL is empty" part? When you put in the first line of that ACL, IOS does a complete 180 and says "if there's even a single line in the applied ACL, anything not explicitly matching that line is dropped" (which, when you think about it makes sense for an ACL). But this means that while you're in the process of pasting the updated ACL, your system is running in an incomplete state and is dropping traffic that would actually be legitimate as per the COMPLETE list.
And of course, pasting several hundred lines of configuration into a heavily-loaded router will take some time, each line would take about a second or so for the router to process going back in. Of course, the system running on the main screens in the NOC had an IP address ending in 240-something, and of course the ACL was organised numerically by destination IP. Which meant that when network devices around the world were up for their next poll cycle, the SNMP requests would time out. This would slowly start an avalanche of alarms in the NOC, entire maps turning from green to red, even audible alarms going off stating that this and that country had gone offline... And without fail, every time this happened there was a tour through the NOC by customers or higher-ups... It would clear itself up in about ten minutes as the list pasting finished, but it was always amusing seeing the NOC staff sinking into panic and managers turning the same shade of red as the screens.
[0] When doing a search-and-replace you had to be VERY careful to not just replace "150" with "151"; you needed to do "-list 150 " with "-list 151 "; otherwise you'd have a bunch of angry Scandinavians on the phone within minutes as they hade IP addresses containing 150 in one octet.
[1] This is in itself a risky mistake. I never had it happen to me, but there were stories circulating that if you deleted an ACL line that a packet was being evaluated against in that very moment, it was prone to crash the box entirely.
Re: False alarms also count as alarms
Sounds like good fun.
Nothing like pressure to really sharpen one's situational awareness in order to avoid an "oops"...
Re: False alarms also count as alarms
I tend to schedule a nightly copy to a network share for such things. Expecting people to backup before making changes is fairly optimistic.
Bad luck, really
But you know what they say : it takes a computer to really fuck things up.