Comms watchdog to probe errors that left Brits unable to make emergency calls
- Reference: 1687951745
- News link: https://www.theregister.co.uk/2023/06/28/ofcome_probing_bt_999_calls_disruption/
- Source link:
The technical glitches at the telecoms operator first showed up on Sunday morning at 0830 local time, and the dominant telco was forced to move to a backup system around 90 minutes later. BT was able to see incoming calls to 999, the UK's emergency services number, and re-routed them to the backup solution but it admitted there was a window of time when calls could not get through.
Upwards of 50 regions and counties flagged up the issue, and police, fire and ambulance services asked members of the public to call 101 (typically for non-emergencies) and local numbers.
[1]
Things were back to normal by the evening but emergency services teams reported a backlog of calls from the morning.
[2]
[3]
[4]Ofcom said today that its investigation will try to set out the "facts surrounding the incident" and ascertain if the watchdog has reasonable grounds to think BT failed to meet regulatory obligations.
"Our rules require BT and other providers to take all the necessary measures to ensure uninterrupted access to emergency organizations as part of anti call services offered. They also require providers to take all necessary measures to ensure the fullest possible availability of calls and internet in the event of catastrophic network breakdown or in cases of force majeure."
[5]
"Separately, providers are required to take appropriate and proportionate measures to identify and reduce the risks of, and prepare for the occurrence of, anything that compromises the availability, performance, or functionality of their network or service.
"Providers are also required to take appropriate and proportionate measures to prevent adverse effects arising from any such compromise. Where there is an adverse effect on the network or service, the provider must take appropriate and proportionate measures to remedy or mitigate that effect."
[6]£2B in UK taxpayer cash later, and still no Emergency Services Network
[7]It's official: Vodafone and Three to tie the knot in the UK
[8]UK's Emergency Services Network unlikely to start operating until 2029
[9]BT is ditching workers faster than your internet connection with 55,000 for chop by 2030
[10]UK emergency services take DIY approach amid 12-year wait for comms upgrade
On Sunday afternoon, local time, the Metropolitan Police, who cover the Greater London area, said the technical fault "could mean that we don't get to someone in danger in time." Others impacted included Police Scotland, South Wales Police, and the force in Greater Manchester.
BT is itself investigating what caused the technical error and intends to reveal those findings to the government, Ofcom and emergency services by tomorrow.
"This will examine the technical aspects of what triggered Sunday's incident, the process of moving over to the backup system, and the timings of communications to the emergency services, Ofcom and the government," it said. ®
Get our [11]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZJxZHoD4kp@FpBt45W3PCAAAAMs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZJxZHoD4kp@FpBt45W3PCAAAAMs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZJxZHoD4kp@FpBt45W3PCAAAAMs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.ofcom.org.uk/news-centre/2023/ofcom-launches-investigation-into-bt-following-999-disruption
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZJxZHoD4kp@FpBt45W3PCAAAAMs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/03/09/esn_nao_report/
[7] https://www.theregister.com/2023/06/14/its_official_vodafone_and_three/
[8] https://www.theregister.com/2023/04/06/emergency_services_network_2029/
[9] https://www.theregister.com/2023/05/18/bt_redundancies_2030/
[10] https://www.theregister.com/2023/04/28/uk_esn_alternatives/
[11] https://whitepapers.theregister.com/
Ensure uninterrupted access?
"Our rules require BT and other providers to take all the necessary measures to ensure uninterrupted access to emergency organizations as part of anti call services offered. They also require providers to take all necessary measures to ensure the fullest possible availability of calls and internet in the event of catastrophic network breakdown or in cases of force majeure."
This is the typical kind of bullshit spouted by people who have zero understanding of how anything works on a technical level. People who think because something is "regulated" that means everything will be ok.
There seems to come a point very quickly where systems can (will, and do) fall over. Whether their application is something serious like emergency services or banking doesn't make them exempt from this premise.
Whether it comes down to incompetence, mistakes or mis-management - and let's be honest if it involves BT that's entirely possible - is a separate matter. But I cannot stand this naivety that critical infrastructure is somehow 100% fault tolerant. It isn't, and no amount of policies, regulations or highly paid consultants are going to change that. Ever.
It's quite scary how much masking tape and spreadsheets keep infrastructure running in the first place. Add in some human error and nothing is guaranteed.
Government expectations
I love the comment from the government reported by the BBC - "The government has said it took BT nearly three hours to alert ministers to the problems it was experiencing."
which seems like a typical non-technical failure to grasp the idea of priorities. If the issue started at 08:30 on Sunday I'd take a wild guess that the only relevant BT staff working (either actively or on call) would be the engineers, and they'd be busy trying to fix the problem. They're engineers, so they're sure as hell not gonna be calling Whitehall who likely can't contribute anything practical to fixing the issue, they'll just be updating their immediately line managers. I imagine you'll have to go up a few levels of management (none of whom would be actively working on a Sunday morning, so possibly not immediately contactable) before you reach someone with the authority to speak direct to the government on behalf of BT. They don't indicate whether any status pages or similar were updated with information, but that's presumably more likely to happen in the short term, than finding someone willing and senior enough to place the call, especially when initially they'd likely have zero information to pass on anyway.
Standard "I can speak to you about the issue, or I can fix the issue... I can't do both".
Local time?....
Please enlighten me...
"On Sunday afternoon, local time, the Metropolitan Police...."
Reminds me of the joke about the local police station having their toilets stolen. The Police commented later that "they had nothing to go on".
Toodle pip.
Finally TheReg posts an article...
I know you've gone all American recently, but this was a pretty big story.
As someone in the know about these things I value the commentard's opinions very much.
This kind of thing is going to be increasingly common. Wouldn't have happened back in the heyday of Nortel.
Anonymous, for reasons above.
Obviously....
Your call is really important to us [plays muzak]
Dry Run
It's just a dry run for what's going to happen when BT turns off the copper phone network and everything goes VoIP.