Canadian ISP Rogers falls over for hours, takes out broadband, cable, cellphones
- Reference: 1657307430
- News link: https://www.theregister.co.uk/2022/07/08/canada_rogers_outage/
- Source link:
ATMs and debit card payments were also affected, leaving folks unable to take out cash and buy stuff in stores, and some were unable to make 911 emergency calls. Rogers has more than 10 million wireless subscribers in Canada. The snafu cut the nation's connectivity by a quarter, [1]according to network watchers at NetBlocks, and may have knock-on effects for other ISPs and service providers.
The outage has been going on for several hours, seemingly nearly 11 hours at time of writing. The communications giant acknowledged the broken connectivity at about 1300 UTC (0900 ET) but it appears from online [2]complaints and the Cloudflare [3]internet radar , the trouble began much earlier, around 0830 UTC (0430 ET).
[4]
According to that radar, there was a surge in BGP routing announcements at around 0845 UTC for Roger's AS812 block of the internet. Other Rogers' blocks [5]show networking difficulties.
[6]
Cloudflare Radar's graph showing a spike in BGP announcements at 0845 UTC for Rogers' AS812 around the same time connectivity died
A few hours ago, Rogers [7]tweeted : "On behalf of all of us at Rogers, we sincerely apologize to our customers, and we will continue to keep you updated as we have more information to share, including when we expect service to be back up."
And on its support message board, it [8]confirmed : "We are currently experiencing an outage across our wireline and wireless networks and our technical teams are working hard to restore services as quickly as possible."
[9]
[10]
We are seeing some reports by netizens that their Rogers services are returning, and plenty others saying it's still down, so your mileage may vary. The comms giant [11]told CBC it is making progress in repairing its networking.
According to the broadcaster, Rogers-owned brands, such as Fido and Chatr, were also knocked offline, and access to travel and financial networks as well as emergency services, was derailed, too. CBC's radio station in Kitchener, Ontario, was even taken off the air. Police stressed that 911 call centers were still working, even if Rogers' subscribers couldn't get through, and rival communications provider Bell said it was unaffected.
[12]
As this article was published, Rogers tweeted to say it is working with its techies and suppliers to bring back connectivity:
Our technical teams are working to restore our services alongside our global technology partners, and are making progress. [13]pic.twitter.com/BsufX2Q92Q — RogersHelps (@RogersHelps) [14]July 8, 2022
Suffice to say, Rogers' customers are furious. "Welcome to Rogers-ocalypse, where a third of Canada's populace becomes a hostage to their service provider," as one put it.
We'll update this story as more information arrives. ®
Updated to add at 2200 UTC
This outage is still ongoing. In the past hour, Rogers emitted this statement:
We acknowledge the impact our outage is having on your life. We have every technical resource and partner fully deployed to solve the problem. As soon as we know the specific time the networks will be fully operational, we will share that with you. Right now, we are focused on the solution. Some of our customers have raised the question of credits and of course we will be proactively crediting all customers and will share more information soon.
It sounds like it's been a long day,. not just for the ISP but also its subscribers.
Get our [15]Tech Resources
[1] https://twitter.com/netblocks/status/1545388751370739712
[2] https://downdetector.ca/status/rogers/
[3] https://radar.cloudflare.com/asn/812?date_filter=last_24_hours
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YsipBibysiyEgnd6mvEaqwAAAIs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://radar.cloudflare.com/asn/12021?date_filter=last_7_days
[6] https://regmedia.co.uk/2022/07/08/handout_cloudflare_radar_bgp_as812_rogers.png
[7] https://twitter.com/RogersHelps/status/1545429159341694976
[8] https://communityforums.rogers.com/t5/Service-Interruption-Status/Service-Interruption/td-p/493355
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YsipBibysiyEgnd6mvEaqwAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YsipBibysiyEgnd6mvEaqwAAAIs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://www.cbc.ca/news/business/rogers-outage-cell-mobile-wifi-1.6514373
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YsipBibysiyEgnd6mvEaqwAAAIs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://t.co/BsufX2Q92Q
[14] https://twitter.com/RogersHelps/status/1545487964465254402?ref_src=twsrc%5Etfw
[15] https://whitepapers.theregister.com/
They can only fix it between 1pm and 3 pm, next Wednesday.
On another note, given the midnight timing of the start the outage, I'm betting celebrations in FSB HQ are underway.
Insert something pithy about ...
.... eggs and baskets right about here.
Annoying As It Is
I wangled a corporate rate with them a few years back, the last time they screwed up my next bill was half the normal amount.
At least they are being upfront about it.
Freedom are worse, I jumped ship after spending a late morning & afternoon ringing the support lines as all incoming calls were going to VM & all their "solutions" didn't work & neither did their call backs (Go figure). Finally after about 5 hours one support rep admitted there was a national outage. Phone services were finally restored 23 hours according to my best guess of when the "services" went down.
Time of writing
It's still down.
must have been something big. will be interresting to follow.
im not with rogers thank goodness but i can say yes .. interac transactions are also affected.
will be quite a lot of people abandoning them.
the outage is WAY too long for WAY too many people.
Started affecting us at ~0940 UK time, that's when we lost all communication with the mothership in Canada so email / network drives etc were unavailable. Made for a quiet day truth told.
Long history of snafus
I recall when they first entered the "business market" with email and stuff. Called it Rogers for Business. For a period of a few weeks, email was being dropped on the floor, never delivered, lost forever, and customers not told.
The problem is Roger's started as a cable company where "sorry, we can't fix it till Tuesday" was acceptable. They never have learned how to deliver the quality of service and reliability that a backbone provider should. This is not the first time they've screwed up this bad, either.
Make no mistake, though, they'll blame someone else for their gross incompetence - last time they blamed unnamed "vendors" for the problem, despite the fact that they don't use any vendors that aren't also used by other telcos in the country, none of whom had an issue.