Verizon wobbles on the East Coast, outage cuts off night owls
(2024/11/12)
- Reference: 1731436210
- News link: https://www.theregister.co.uk/2024/11/12/verizon_east_coast_outage/
- Source link:
The holiday weekend in the US ended badly for Verizon FiOS users on the East Coast after over 40,000 customers reported being forced offline.
While Downdetector showed some other problems for the service in Los Angeles and the Midwest, cities like Washington DC and Philadelphia had the bulk of the problems, although the impact in New York and Pittsburgh appeared to be minimal. The outage kicked off at 1228 ET (0528 UTC) and was resolved within four hours for the majority of affected customers.
"A network issue early this morning disrupted service for some Verizon FiOS customers in the Northeast for a short period of time," the telco [1]tweeted . "As soon as the issue was identified, our engineering teams quickly restored the service."
[2]
The alternative site Outage Report [3]says over 40 US cities were affected by the network issues.
[4]
Verizon's rocky night - Click to enlarge. Source: Downdetector.
Given the timing, it's possible that this was a midnight patch gone wrong, although one member of the Outages mailing list spotted some very odd traffic patterns on Verizon networks.
"Seeing sites down in northern Virginia and Maryland, but interestingly I actually have two sites still up on FiOS in Maryland not far from other sites that are down," they report.
[5]
[6]
"Also my one site with Verizon enterprise is up. Routers on [DHCP] can get [DHCP] leases and ping the gateway provided, but traceroutes fail to even get a reply from the gateway, traceroutes from outside fail with only one reply from inside Verizon's network, seems to be a pretty major software and/or routing failure."
Verizon isn't answering questions at the moment but we'll let you know if it does.
[7]Washington courts grapple with statewide outage after 'unauthorized activity'
[8]Microsoft admits Outlook crashes, says impact 'mitigated'
[9]Chinese cyberspies reportedly breached Verizon, AT&T, Lumen
[10]Financial institutions told to get their house in order before the next CrowdStrike strikes
Curiously T-Mobile US and Comcast also had a minor service outage at around the same time. It may be that there was a brief knock-on effect or people misreporting the Verizon outage.
It's the [11]second time in two months that Verizon has seen outage problems. In September, wide swaths of its service went down. Part of this was down to the obvious aftereffects of Hurricane Helene but serious problems were reported elsewhere in the country.
[12]
We'll update this piece as more information comes in. ®
Get our [13]Tech Resources
[1] https://x.com/VerizonSupport/status/1856334754888438158
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZzPeF-8-7pcEO11KTVU55wAAAIs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://outage.report/us/verizon-fios
[4] https://regmedia.co.uk/2024/11/12/verizon.jpg
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZzPeF-8-7pcEO11KTVU55wAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZzPeF-8-7pcEO11KTVU55wAAAIs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2024/11/05/washington_courts_outage/
[8] https://www.theregister.com/2024/10/10/microsoft_outlook_outage/
[9] https://www.theregister.com/2024/10/07/verizon_att_lumen_salt_typhoon/
[10] https://www.theregister.com/2024/11/02/fca_it_resilience/
[11] https://www.theregister.com/2024/09/30/verizon_outages_report_us/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZzPeF-8-7pcEO11KTVU55wAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
While Downdetector showed some other problems for the service in Los Angeles and the Midwest, cities like Washington DC and Philadelphia had the bulk of the problems, although the impact in New York and Pittsburgh appeared to be minimal. The outage kicked off at 1228 ET (0528 UTC) and was resolved within four hours for the majority of affected customers.
"A network issue early this morning disrupted service for some Verizon FiOS customers in the Northeast for a short period of time," the telco [1]tweeted . "As soon as the issue was identified, our engineering teams quickly restored the service."
[2]
The alternative site Outage Report [3]says over 40 US cities were affected by the network issues.
[4]
Verizon's rocky night - Click to enlarge. Source: Downdetector.
Given the timing, it's possible that this was a midnight patch gone wrong, although one member of the Outages mailing list spotted some very odd traffic patterns on Verizon networks.
"Seeing sites down in northern Virginia and Maryland, but interestingly I actually have two sites still up on FiOS in Maryland not far from other sites that are down," they report.
[5]
[6]
"Also my one site with Verizon enterprise is up. Routers on [DHCP] can get [DHCP] leases and ping the gateway provided, but traceroutes fail to even get a reply from the gateway, traceroutes from outside fail with only one reply from inside Verizon's network, seems to be a pretty major software and/or routing failure."
Verizon isn't answering questions at the moment but we'll let you know if it does.
[7]Washington courts grapple with statewide outage after 'unauthorized activity'
[8]Microsoft admits Outlook crashes, says impact 'mitigated'
[9]Chinese cyberspies reportedly breached Verizon, AT&T, Lumen
[10]Financial institutions told to get their house in order before the next CrowdStrike strikes
Curiously T-Mobile US and Comcast also had a minor service outage at around the same time. It may be that there was a brief knock-on effect or people misreporting the Verizon outage.
It's the [11]second time in two months that Verizon has seen outage problems. In September, wide swaths of its service went down. Part of this was down to the obvious aftereffects of Hurricane Helene but serious problems were reported elsewhere in the country.
[12]
We'll update this piece as more information comes in. ®
Get our [13]Tech Resources
[1] https://x.com/VerizonSupport/status/1856334754888438158
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZzPeF-8-7pcEO11KTVU55wAAAIs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://outage.report/us/verizon-fios
[4] https://regmedia.co.uk/2024/11/12/verizon.jpg
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZzPeF-8-7pcEO11KTVU55wAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZzPeF-8-7pcEO11KTVU55wAAAIs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2024/11/05/washington_courts_outage/
[8] https://www.theregister.com/2024/10/10/microsoft_outlook_outage/
[9] https://www.theregister.com/2024/10/07/verizon_att_lumen_salt_typhoon/
[10] https://www.theregister.com/2024/11/02/fca_it_resilience/
[11] https://www.theregister.com/2024/09/30/verizon_outages_report_us/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZzPeF-8-7pcEO11KTVU55wAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
Yeah, I Experienced the Outage in Suburban Baltimore
I figured that it was the Internet Gods telling me to go the f%$3 to sleep.