Ten Mistakes Marred Firewall Upgrade At Australian Telco, Contributing To Two Deaths (theregister.com)
- Reference: 0180425929
- News link: https://it.slashdot.org/story/25/12/19/2241221/ten-mistakes-marred-firewall-upgrade-at-australian-telco-contributing-to-two-deaths
- Source link: https://www.theregister.com/2025/12/19/optus_emergency_outages_cause_report/
> On Thursday, Optus published an independent [2]report (PDF) on the matter written by Dr Kerry Schott, an Australian executive who has held senior management roles at many of the country's most significant businesses. The report found that Optus planned 18 firewall upgrades and had executed 15 without incident. But on the 16th upgrade, Optus issued incorrect instructions to its outsourced provider Nokia. [...] Schott summarized the incident as follows: "Three issues are clear during this incident. The first is the very poor management and performance within [Optus] Networks and their contractor, Nokia. Process was not followed, and incorrect procedures were selected. Checks were inadequate, controls avoided and alerts given insufficient attention. There appeared to be reticence in seeking more experienced advice within Networks and a focus on speed and getting the task done, rather than an emphasis on doing things properly."
>
> The review also found that Optus' call center didn't appreciate it could be "the first alert channel for Triple Zero difficulties." The document also notes that Australian telcos try to route 000 calls during outages, but that doing so is not easy and is made harder by the fact that different smartphones behave in different ways. Optus does warn customers if their devices have not been tested for their ability to connect to 000, and maintains a list of known bad devices. But the report notes Optus's process "does not capture so-called 'grey' devices that have been bought online or overseas and may not be compliant."
"To have a standard firewall upgrade go so badly is inexcusable," the document states. "Execution was poor and seemed more focussed on getting things done than on being right. Supervision of both network staff and Nokia must be more disciplined to get things right."
[1] https://www.theregister.com/2025/12/19/optus_emergency_outages_cause_report/
[2] https://regmedia.co.uk/2025/12/19/supplied_independent_report_triple_zero_outage_at_optus_18_september_2025.pdf
Liability (Score:2)
So is someone gonna get sued for those two deaths?
455 calls failed and two callers died (Score:1)
Two dead out of 455. That's just a bad afternoon at the Brontosaurus Steakhouse in Omaha, on Wednesday, when the fried ice cream cutlets go two for one.
An incidence of 0.0043956.
Or for you, not exactly mathletes out there, less than 1/2 of 1%.
Clearly, AI is the new cholesterol.
Re: (Score:2, Insightful)
It's certainly a testament to how many of those calls didn't need to happen, and I'm not seeing here that we're sure those people would have made it if the calls had gone through, or they'd even have had a good chance.
On the flip side, if anyone certainly died because of negligence around a critical emergency service, some heads should roll. Unfortunately, it won't be the ones which it needs to in order to prevent it from happening again.
Re: (Score:2)
Well, as long as some heads roll. right?
I mean, the whole scheme of things nowadays is to identify a scapegoat, shame him/her profoundly, and move along.
It gives one armor?
Re: (Score:2)
It helps to read to the end.
Lots of lessons for us all in the official report. (Score:1)
Well worth skimming the official technical investigation report. Ten human mistakes are discussed in detail -- many are project management and organizational-culture issues that anyone in technology may run into. (I've certainly run into such issues in my career.) Official report:
[1]https://regmedia.co.uk/2025/12... [regmedia.co.uk]
[1] https://regmedia.co.uk/2025/12/19/supplied_independent_report_triple_zero_outage_at_optus_18_september_2025.pdf
lock on a gateway what network hardware has LOTO? (Score:2)
lock on a gateway what network hardware has LOTO?
Re: (Score:2)
Thanks for the reference. I will use this as one more teaching example of exceptional incompetence when running critical IT systems.
And, gross negligence or not? (Score:2)
Because I think the ones responsible should be charged with manslaughter. Gross negligence could allow that.
what about stop useing so meny contractors and sub (Score:2)
what about stop useing so meny contractors and subcontractors?
Re: (Score:2)
Optus said they could do it cheaper and better than Telstra.
Follow the money.
Re: (Score:2)
Companies want to save money and decide to employ operators rather than engineers. Then when they need to contract "experts" for projects they go with the lowest bidder. What do they expect to happen? You get what you pay for.... sometimes less.