Stop shaming service providers for outages, argues APNIC chief scientist
- Reference: 1700789411
- News link: https://www.theregister.co.uk/2023/11/24/routing_police/
- Source link:
In a [1]post that opens by considering the [2]major outage at Australian telco Optus , APNIC chief scientist Geoff Huston opined "If this were a bank heist the site would no doubt be saturated with investigators from the police force."
But the cause of the Optus incident was a "routing heist."
[3]
"So where are the routing police to investigate the incident?" he asked. "How can we understand the exact nature of the triggers for this outage and identify if there was some level of contributory negligence from the network operator or their suppliers that amplified a minor issue of a route leak into a major issue that impacted millions of consumers?"
[4]
[5]
Huston lamented that routing police don't exist, and argued that internet governance organizations, standards bodies, and network operator groups are not appropriate entities to take on the task.
He's also not keen on national regulators doing the job, given the internet crosses borders and governments do not.
[6]APNIC close to completing delegation of its final /8 IPv4 block
[7]APNIC: Big Tech's use of carrier-grade NAT is holding back internet innovation
[8]APNIC left a dump from its Whois SQL database in a public Google Cloud bucket
[9]Starlink creates risk of internet investment doom cycle, says APNIC researcher
The aviation industry, he suggested, offers a better model.
"We need to respond to outages and related incidents in the internet in a way that does not immediately attempt to sweep it under the closest rug and deny that anything untoward ever happened at all," he wrote. He cited the airline industry as "a case in point where the object of an investigation is not necessarily to apportion blame, but to unearth the root causes and potentially propose measures that … operators can adopt that would prevent a recurrence of the mishap."
[10]
"It would be good if all service providers in the public internet spent the time and effort post-rectification of operational problems to produce detailed and thorough outage reports as a matter of standard operating procedure," Huston argued.
Doing so is "not about apportioning blame or admitting liability." Rather, sharing incident analyses is "all about positioning these services as the essential foundation of the public digital environment and stressing the benefit of adopting a common culture of open disclosure and constant improvement as a way of improving the robustness of these services."
"It's about appreciating that these days these services are very much within the sphere of public safety and their operation should be managed in the same way," he added. ®
Get our [11]Tech Resources
[1] https://blog.apnic.net/2023/11/23/call-the-routing-police/
[2] https://www.theregister.com/2023/11/08/internet_phone_train_and_payments/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZWAt9ElRoh8OdTAFZLuAdwAAAME&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZWAt9ElRoh8OdTAFZLuAdwAAAME&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZWAt9ElRoh8OdTAFZLuAdwAAAME&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/10/11/apnic_ipv4_exhaustion_milestone/
[7] https://www.theregister.com/2022/01/20/ipv4_nats_slow_ipv6_transition/
[8] https://www.theregister.com/2021/06/22/apnic_whois_data_exposed/
[9] https://www.theregister.com/2021/04/23/starlinks_catch_22/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZWAt9ElRoh8OdTAFZLuAdwAAAME&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://whitepapers.theregister.com/
Kind of see his point.
But what I do know (from experience) is that in the event of any cockup blamestorming always follows as does searching for a scapegoat to punish.
No employer of mine would ever come out and share with the broader community/competitors a "Hey we fucked up, this is what you guys can do to make sure it wont happen to you later". They'd say bugger all and find someone to shitcan.
Indeed...
... Don't blame the ISPs, blame the carriers - but in this case the two are conveniently one and the same.
All I've been able to do when thinking about this fiasco was PMSL, in utter disbelief that there are still idiots willing to entrust Sloptus with anything. Even Dodo's given them a run for their money, and that's really saying something.
Re: Indeed...
Err.. as far as I know (and you suggest), Dodo (in Australia) is a (mobile) virtual network operator which solely uses the Optus network. The other (physical) choices are Telstra and Vodaphone (unsure about iiNet).
Soo.. the only thing Dodo did, was run (away) with your money (and give it to Optus)
Perhaps they gave some (money/data) back due to the outage providing better value for money?
.... a quick search suggest Dodo did Jack
First step is having some kind of Integrity...
In my professional capacity of being a network administrator/engineer for over 20 years Optus is the only carrier I have ever encountered where their troubleshooting technician actively lied to me.
This speaks volumes about the internal culture of the company (i.e. When the tech feels that they need to lie to the customer because otherwise they will get a bollocking from Management because the customer is entitled to partial refund)
Optus at least will continue to sweep errors under the rug as long the company believes that honesty and integrity and just meaningless works you include in advertising material.
Aviation Investigations
May not apportion blame, but lawyers use their accident investigation reports to apportion liability. I suspect the telecommunications industry would fear that.