New York City 911 System Missed 1,700 Calls Due To Software Update (medicaldaily.com)
- Reference: 0185261294
- News link: https://tech.slashdot.org/story/26/08/28/2140220/new-york-city-911-system-missed-1700-calls-due-to-software-update
- Source link: https://www.medicaldaily.com/nyc-911-outage-bronx-motorola-software-update-cause-477756
> Part of New York City's 911 emergency system was [2]offline from about 3:13 a.m. to 10 a.m. on August 18th due to a software update , according to Medical Daily. Of 5,659 calls to the Bronx call center during that period, about 1,700 calls connected but could not be heard by emergency operators. Some were rerouted to other call centers, but an estimated 400 human callers never got through. The potentially serious consequences of delayed response to emergencies are not yet known. The city did not notify the public and only released information when journalists inquired. How buggy software came to be installed on a running system is not yet known.
The early-morning software update was pushed by Motorola, though exactly what it changed or why it caused the failure remains unknown. The city is still investigating whether the update was adequately tested and whether the same software release was deployed elsewhere.
[1] https://slashdot.org/~belmolis
[2] https://www.medicaldaily.com/nyc-911-outage-bronx-motorola-software-update-cause-477756
Like strangely more things lately... (Score:1)
Probably AI-generated code, with agentic testing and debugging, with some junior saying "whelp looks good I guess or whatever" while he pushes it to production. All the while the original seasoned "bioware" devs are collecting unemployment.
1K crises per hour? (Score:4, Interesting)
So does NYC really have on the order of 1,000 justified crises every hour or is there just a lot of abuse and drama queens in that city? That averages out to around a call per year for each inhabitant.
Re: (Score:2)
Well, a quick search tells me there are 8,546,038 New York City residents. 24k calls a day would be 0.002808319 of the population has a problem per day. Doesn't seem unreasonable when you look at it that way.
Re: 1K crises per hour? (Score:1)
> 8,546,038 New York City residents. 24k calls a day would be 0.002808319
Uh, what?
Are you trying to determine a percentage?
24,000 different callers out of 8,546,038 New Yorkers means that 0.280832% of the population calls 911 in a given day...
Re: (Score:3)
Still doesn't seem out of line, especially given [1]the things people call 911 over. [buzzfeed.com]
[1] https://www.buzzfeed.com/ravenishak/911-dispatchers-dumb-stories
Re: (Score:2)
I've called 911 once in my life and it was over 40 years ago.
Re: (Score:1)
I'm going to point out that NYC is a dense population metropolis. So in an average town maybe 3 people see something alarming and call 911; however, in NYC there could be 10-15 people that witness something and call. Same activity, but more observers.
Re: (Score:2)
Oh I've seen alarming things, including alarming things in NYC. I just never called 911. Of course I didn't have a mobile phone when I saw those things in NYC but I doubt if I would have called anyway.
Re: (Score:2)
Some calls are repeats, either different people calling about the same event or the same person calling when he or she doesn't get through. 911 also gets automated calls from alarms, some of which are false alarms. That's why, of the 1700 calls that could not be heard, they estimate that about 400 represent human callers who did not get through.
The bots! I knew it! (Score:1)
> 911 also gets automated calls from alarms, some of which are false alarms.
Just like social media, 911 is also stuffed with bots.
Re: (Score:1)
No, but a lot of people simply don't know what else to do. Plus: in many cases, there isn't anything else to do because chronically underfunded and understaffed city and state agencies just can't keep up.
So 911 operators wind up spending a substantial amount of their time redirecting people, e.g., "wait until business hours and then call XYZ". Or they forward calls. Or they do the best they can to help out (sometimes very upset) people who've called 911 because they're desperate.
In a better world,
Re: (Score:2)
> So 911 operators wind up spending a substantial amount of their time redirecting people, e.g., "wait until business hours and then call XYZ". Or they forward calls. Or they do the best they can to help out (sometimes very upset) people who've called 911 because they're desperate.
my experience over the years calling the police non-emergency number (which also routes to 911 operators in my area) or 911 for urgent but not critical things in a few towns has had the operators be at best "curt, but professional". rarely had one who seemed to care about what was going on or that wanted to help or dispensed actually useful routing advice.
it was usually a "ugh.... fine I'll take down the info because the law probably requires that I do so" vibe. like my existence was tolerated... but
Re: (Score:2)
oh, but NYC has 311. I haven't used theirs but in the 2 places I have used it... they were more responsive and intellectual curious and engaged than the police/911 ones... at least usually.
Re: (Score:3)
> "rarely had one who seemed to care about what was going on"
Defense mechanism and perfectly understandable. They have to deal with the worst situations all day long. If they had too much emotion or empathy, they would burn out immediately.
> "ugh.... fine I'll take down the info because the law probably requires that I do so" vibe.
They also have to deal with constant abuse of the system. Not saying your calls were, but such things tend to make people very dispassionate.
Remember that other article rec
Re: (Score:2)
Automated alarm calls are apparently another issue. I haven't seen good statistics on what percentage of 911 calls are alarm calls, but it seems that they are at least 10% and that the percentage is rising. Over 95% of automated alarm calls are reportedly false alarms.
Re: (Score:2, Insightful)
Drama queens.
But then what do you expect in a city where people have been conditioned to expect mommy government to wipe their noses for them? 911 is just the public facing contact for the public.
Ignorant hicks (Score:2)
Have no idea what a real city is like. Get back to us when you've visited somewhere you have to wear shoes.
Re: 1K crises per hour? (Score:1)
I don't see 1,700 "missed calls," I see about 400:
> Of 5,659 calls to the Bronx call center during that period, about 1,700 calls connected but could not be heard by emergency operators. Some were rerouted to other call centers, but an estimated 400 human callers never got through.
That quote indicates that only 400 calls were "missed" if we define "missed" as "never got through."
A and B systems (Score:4, Interesting)
Back in the last quarter of the 20th century, many POTS telephone switches had an A side and a B side.
During software upgrades - and yes those were a thing - they would route all new calls to the A side and wait until the last call on the B side ended before they would take the B side down for the upgrade.
Then they would do some testing before bringing up the B side.
Only after they were confident that the B side was working would they upgrade the A side.
This worked in part because they could schedule upgrades when the load was low enough to be handled only by half of the switch's capacity.
Re: (Score:2)
I was thinking that, in a perfect world, there would be two parallel (and completely isolated from each other) systems running simultaneously, but capable of handling the full load of both. Upgrades would involve shifting the full load to one while the other is upgraded, then very careful testing before switching.
I used to work in the industry (Score:3)
Not at Motorola, but another company in the space.
Software upgrades were extremely methodical in nature. The change control calls had dozens of people on them, and the upgrade document was hundreds of pages long. The ops team (which was not the dev team) rehearsed the upgrade in lower environments for weeks before doing it for real. I was on upgrade calls mostly just as a security blanket in case developer intervention was required.
I feel bad for the folks who have to deal with the FCC paperwork. You never wanted to get invited to one of those calls, it was like having the Inquisition show up at your doorstep.
> How buggy software came to be installed on a running system is not yet known.
It may not even have been the software. Well, it might have been, but it might also not have been. I got paged out at 2am one time because the domain controller, which I do not own, rotated all the passwords for the managed service accounts, which I do not control, and the SQL database, which I do not manage, suddenly lost its shit and wouldn't let any apps log in. It fixed itself in about 15 minutes, but the state noticed, and we were doing paperwork for about a week after that.
Unknown? (Score:1)
"How buggy software can be installed on a running system is unknown..." LOL. Breaking things is easy. It's keeping them from breaking that's hard.
Re: (Score:2)
I have a shirt with the text "I don't usually test my code, but when I do, I do it in production." Seems appropriate here...
Tested ? (Score:3)
"The city is still investigating whether the update was adequately tested..."
"No."
Hopefully I just saved the city a bunch of money.
No longer 911 (Score:3)
Now its 0118999881999119725
-3