Snow day in corporate world thanks to another frustrating Microsoft Teams outage
- Reference: 1706317151
- News link: https://www.theregister.co.uk/2024/01/27/teams_outage_again/
- Source link:
The IT goliath said it became aware of the breakdown around 1455 UTC, with a service bulletin [1]reporting that at least some customers may experience errors, missing or delayed messages, and/or the inability to upload or download documents using Teams.
"We apologize for the inconvenience; our engineering team is aware and actively working to resolve the current Teams outage," Redmond [2]shared via Xitter.
[3]
The issue was quickly attributed to a "networking issue," but whatever caused the problem was limited to the online chat'n'meeting app Teams. "We've completed a failover in the Europe, Middle East, and Africa (EMEA) region and telemetry is showing improvement," the bulletin read.
[4]
[5]
In the Americas, however, attempts to bring Teams back online failed to achieve the desired effect. "Our failover operation did not provide the anticipated relief to end users in North and South America regions and we are now working to optimize traffic patterns as part of the mitigation effort," the service bulletin read.
By 2212 UTC, Microsoft reported that its efforts to resolve the outage were showing signs of improvement, with netizens [6]commenting that messages were once again mostly flowing. However as of writing, Microsoft's Office Service Health page still showed Teams as down.
[7]
What caused the global partial outage and why the network fix worked in EMEA but not in the Americas remains unclear. We've reached out to Microsoft for comment; we'll let you know if we hear anything back.
[8]Major IT outage at Europe's largest caravan and RV club makes for not-so-happy campers
[9]COVID-19 test lab accused of exposing 1.3 million patient records to open internet
[10]Tesco techies and Azure jockeys hit the floor during weekend of outages
[11]Linus Torvalds postpones Linux 6.8 merge window after being taken offline by storms
The timing of the outage suggests the issue may have been related to a broader infrastructure problem. Around the time Teams began acting up, Ookla's Downdetector showed a marked uptick in issues accessing [12]Google and [13]Azure services.
Whether these incidents are just coincidental, it's worth noting that this is hardly the first time Teams has logged off early for the day leaving users in the lurch. In 2023 Microsoft suffered repeated outages across its cloud and Office 365 properties, including one that [14]occurred almost exactly a year ago yesterday. ®
Get our [15]Tech Resources
[1] https://portal.office.com/servicestatus
[2] https://twitter.com/MicrosoftTeams/status/1751008161610465508
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZbSN917pPAZMXQUlFYWiKQAAAcY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZbSN917pPAZMXQUlFYWiKQAAAcY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZbSN917pPAZMXQUlFYWiKQAAAcY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://twitter.com/MSFT365Status/status/1751005104528859594
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZbSN917pPAZMXQUlFYWiKQAAAcY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2024/01/24/major_it_outage_at_caravan/
[9] https://www.theregister.com/2024/01/24/dutch_covid_testing_firm_ignored_warnings/
[10] https://www.theregister.com/2024/01/22/a_weekend_of_outages_tesco_azure/
[11] https://www.theregister.com/2024/01/15/linux_kernel_merge_window_postponed/
[12] https://downdetector.com/status/google/
[13] https://downdetector.com/status/windows-azure/
[14] https://www.theregister.com/2023/01/25/network_issues_causing_outage_in/
[15] https://whitepapers.theregister.com/
I was wondering why things were so quiet today
Also, much more productive.
I can't help but be amused by all of these outages. IT and IS departments convinced CTOs to spend massive amounts of money to outsource all of their infrastructure to the cloud, so that it would be more reliable, and yet many companies are experiencing more downtime and data loss.
It reminds me of the time some execs ordered us to save money by getting rid of those "pointless" co-located backup servers and the "useless" in-house redundant server, and just put everything into one really big box. Simple, clean, none of that "replication" nonsense that slowed things down.
It wasn't until it was fully in production (which we did under protest) that I was asked what the machine name spof.companyname.com meant. When I explained that SPOF mean " single point of failure ", the CEO (the CTO's boss) went white as a sheet, and wanted us to explain what would happen if it we to fail.
One rendition of Monty Python's dead parrot sketch (" it's pinin' for the fjords; it's ceased to be; it shall be an ex-server ") later, he demanded we explain and justify "our" decision to do this. Several CYA emails were displayed, and the new CTO that arrived the next month promptly reversed the decision, and we were able to restore multi-site before there was any disaster.
Today, "SPOF" is becoming synonymous with "the cloud". AWS, Office 365, and the like mean that if your net connection goes down, so do you.
I thought "Cloud" was supposed to make all this stuff nigh-unbreakable, with seamless failover in the event of an issue, everything tested to destruction before being put into service, absolutely safe for us to give up all control and put all our eggs into their single basket at a much higher cost than just doing it ourselves.