Americans wake to widespread cellular outages, cause unclear
- Reference: 1708610413
- News link: https://www.theregister.co.uk/2024/02/22/americans_wake_to_widespread_cellular/
- Source link:
Problems with AT&T, Verizon, and T-Mobile USA services began flooding into crowd-sourced outage monitor Down Detector at roughly 0400 Eastern (0100 Pacific) and haven't slowed since, with more than 73,000 [1]AT&T outage reports submitted at the time of writing. T-Mobile US and Verizon, the other two major cell network operators in America, have also seen widespread outage reports, though far fewer in number than AT&T.
It does appear that AT&T is the carrier primarily hit by this IT breakdown, and T-Mo and Verizon subscribers are feeling secondary effects.
[2]
"T-Mobile did not experience an outage. Our network is operating normally," a T-Mo spokesperson told The Register . "Down Detector is likely reflecting challenges our customers were having attempting to connect to users on other networks."
[3]
[4]
Verizon had much the same to say, telling us that its network is operating normally, and the issues likely relate to Verizon customers trying to reach those on other carriers. With most of the reports pertaining to AT&T, and T-Mo and Verizon's denials of anything wrong on their end, the outage appears to be an AT&T issue, which the carrier has confirmed.
"Some of our customers are experiencing wireless service interruptions this morning. We are working urgently to restore service to them," an AT&T spokesperson told us. "We encourage the use of Wi-Fi calling until service is restored."
[5]
No explanation for the outage, which is ongoing as of publication, was given.
[6]Snow day in corporate world thanks to another frustrating Microsoft Teams outage
[7]Stop shaming service providers for outages, argues APNIC chief scientist
[8]Bank boss has pay slashed after presiding over tech outages in 2023
[9]Dish multi-day outage rolls on as ransomware fears grow
The outage is severe enough that affected AT&T customers appear to be unable to place or receive any phone calls, including to 911, the San Francisco Fire Department [10]warned customers on Twitter.
"If you are an AT&T customer and cannot get through to 911, then please try calling from a landline," the SFFD said - but "do not call or text 911 simply to test your phone service," the Fire Department added.
[11]
A map of AT&T outages as of 8:38 am eastern time, Feb 22 - Click to enlarge
A map of AT&T outage reports shows several major metropolitan areas are affected at the time of publication, including Atlanta, Dallas and Houston.
Cellular towers have classically been owned by the carriers, but that's changed in the US in recent years - most towers are now owned by tower real estate investment trusts like American Tower and Crown Castle, who've bought up [12]tens of thousands of cell towers. Many of those towers were purchased from AT&T, Verizon and T-Mobile, making even the largest US cellular providers look more like the small virtual network operators their hardware has classically supported.
With that in mind, we reached out to some of the major cell tower firms to see if they had anything to report. We'll update this story if we hear back, and as the situation evolves. ®
Updated to add at 1650 UTC
A spokesperson for AT&T has let us know the carrier thinks its network is recovering: "Some of our customers are experiencing wireless service interruptions this morning. Our network teams took immediate action and so far three-quarters of our network has been restored. We are working as quickly as possible to restore service to remaining customers."
Cloudflare has a [13]graph showing a significant drop off of IPv4 and IPv6 connectivity for AT&T, leading some network administrators to speculate it's a BGP issue.
Get our [14]Tech Resources
[1] https://downdetector.com/status/att/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zdd9t0HegN1th4caYXbN3gAAAUU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zdd9t0HegN1th4caYXbN3gAAAUU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zdd9t0HegN1th4caYXbN3gAAAUU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zdd9t0HegN1th4caYXbN3gAAAUU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2024/01/27/teams_outage_again/
[7] https://www.theregister.com/2023/11/24/routing_police/
[8] https://www.theregister.com/2024/02/07/dbs_ceo_pay_docked/
[9] https://www.theregister.com/2023/02/27/dish_outage_ransomware/
[10] https://twitter.com/SFFDPIO/status/1760618741422072177?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1760618741422072177%7Ctwgr%5E459f8456d641759d26fbb52d8401d1113fad45eb%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fwww.reuters.com%2Ftechnology%2Fcellular-outage-us-hits-att-t-mobile-verizon-users-downdetector-shows-2024-02-22%2F
[11] https://regmedia.co.uk/2024/02/22/att-feb-22-outage-map.jpg
[12] https://www.steelintheair.com/cell-tower-companies/
[13] https://twitter.com/CloudflareRadar/status/1760694607070073021
[14] https://whitepapers.theregister.com/
Carry on, nothing to see here...
"T-Mobile did not experience an outage. Our network is operating normally,"
Because with T-Mob you'd never notice an outage anyway, it's all part of the standard customer experience.
Well that's interesting...
Was this an attack, or a naturally-occuring issue? It'll be interesting to see...
Re: Well that's interesting...
"A communications breakdown can only mean one thing - I-N-V-A-S-I-O-N "
- Star Wars, Episode 1, The Phantom Menace
Re: Well that's interesting...
No point in invading, the US are on an excellent trajectory to destroy themselves. At least in respect to what they were founded for a few years ago.
Re: Well that's interesting...
> ...a few years ago...
Said an ignorant someone to the second-oldest continuously functioning nation on the planet, and the oldest representative democracy...
>"If you are an AT&T customer and cannot get through to 911, then please try calling from a landline," the SFFD said
Ah yeah, the copper POTS landline network that AT&T has been systemically dismantling since 2016 because it's expensive to maintain and not a revenue generator. Those landlines. Just use those.
Maybe you can unplug a fax in your office and hook your rotary phone to the last still working line on the block
RE: Phone on a fax line
Hey! That's still an option in [my office / my industry]
I still have a "landline", it is just VOIP over the mixed coax/fiber network the cable company has. But yes, copper POTS is dying in America.
>But yes, copper POTS is dying in America.
Just another point of ignorance between urban and rural America.
I've got to drive 8 miles to get the first bar of cell service. POTS with DSL is the only land-based Telco/Internet option available. No one is bringing fiber to my location, much less to my residence. $tarlink is an option if I cut down a bunch of trees for a broad view of the sky.
Being a bit of a retro-tech geek... I have two Western Electric 500 rotary phones - one desk set and one wall mount. Pure analog wired service. Those phones have better audio than any of the modern digital wireless phones around the house. Far better than the iPhone / Bluetooth combination when I am in cell range. It is surprising how much we have compromised on audio quality over the years with the so-called modern tech.
Verizon sent me a letter a few years ago, telling me that if I didn't allow them to replace my copper landline with fiber (at no cost to me), that I would lose service. I accepted their offer, kept the new fiber landline for a month, then terminated service. When COVID rolled around, I called them back and had them activate the fiber with FIOS internet on it.
My landline is now through a VOIP provider, over that fiber, can use both pulse (rotary) and DTMF phones on it, and the number is the same one I've had for 40 years, at a small fraction of what a Verizon landline would cost me. It works just fine, and I used it to verify that I could call my AT&T cellphone this morning. Whatever rot is in the AT&T infrastructure, it hasn't yet made it to my neck of the woods
Cellphones aren't tied to their primary network for calling 911; this is built into the cellular standard. In fact, you can call 911 without a SIM, as long as your cellphone can see a network. So if people are unable to call 911 but there is still an active network (of any description) in their area, it'll be one of two things:
1. the problem is indeed broader than 'just' AT&T, and other networks are experiencing problems too.
2. OR the cellphone thinks it's still connected to AT&T and therefore won't try other networks automatically.
If 2., then removing the SIM should allow you to call 911, as the cellphone will then not remain tethered to AT&T, but will default to whichever (working) network it can pick up.
This is all true, but I think it's easier for them to just say "have a neighbor call it in" or "time travel to 2009" than get into SIM support calls on the 911 lines. Can you imagine the number of dowagers sitting at home with their cell phones in ruins alternating between 911 calls and bothering their grandchildren?
"I do have the simp right here. It says Sandisk."
"Alright I've rebooted the phone, what do I do next? Hello? Hello?"
"I just wanted to call back say thank you for the patience you showed when you spent 90 minutes fixing my Tracfone. 911 operators are the best."
Towers
Not sure I understand the penultimate paragraph completely.
The towers are just metal. Yes, they're owned by tower companies, but the towers themselves (as long as they're standing), and who owns them, have no effect on the cellular service provided through the equipment bolted to them and in the shelters at their bases. THAT is owned by the cellular providers, who pay rent to the tower companies for space on the towers.
I doubt very much if the ownership of the towers has anything to do with this outage. A network misconfiguration would be my guess.
Re: Towers
The towers are just metal. Yes, they're owned by tower companies, but the towers themselves (as long as they're standing), and who owns them, have no effect on the cellular service provided through the equipment bolted to them and in the shelters at their bases. THAT is owned by the cellular providers, who pay rent to the tower companies for space on the towers.
That's how it worked yonks ago when I designed cellsite radio gear. I'm VERY skeptical that it has since changed so that now companies like American Tower own and run the actual radios and telephone networks — and that Verizon, ATT, and T-Mobile are just reselling access/minutes/GB the way that Cricket et alia do.
Cellular network operators have been using rented antenna and equipment space on/in other people's towers and buildings since the very beginning back in 1980.
Read the end of that title as "cause nuclear" at first and was very confused.
Me too :)
Sun Spots
Sunspots? At this time of year, at this time of day, in this part of the country, localized entirely within AT&T's network?!
Open flame for steaming hams ----->
ATT user here
I'm in Eastern North Carolina have ATT cellular.
My phone has been working fine, no problems, and I've heard plenty of reports of people with ATT phones who also haven't been having problems. However, my car had no connectivity, and I believe it connects using ATT's LTE network.
From what I've been hearing is the problem is with the LTE network, my phone is on the 5G network and has been unaffected.
Sun Flares?
NASA reporting two major sun flares several hours ago. not sure if that's it, but that would do it.
Reddit reporting it's a problem with the cisco hardware if that can be believed.
It's believable that the sun flares hit something that knocked out cisco's side of things.
Please do not call 911 simply to test your phone line
The outage will finish shortly. We are simply testing our systems.
Sincerely,
Your Friendly CCP Senior Electronic Warfare Officer