Global network outage hits Microsoft: Azure, Teams, Outlook all down
- Reference: 1674646234
- News link: https://www.theregister.co.uk/2023/01/25/network_issues_causing_outage_in/
- Source link:
Users all over the world are venting their anger on social media at not being able to send or receive email, connect to Teams meetings or access many other services.
The problems started at around 0705 UTC on January 25 and involved issues with network connectivity for millions of users of Azure, Teams and Outlook, when they attempted to connect with Azure resources, which naturally includes Microsoft's cloud-hosted services.
[1]
A [2]Service Health Status update from the company at 0810 UTC admitted that: "Users may be unable to access multiple Microsoft 365 services," which include but were not limited to Teams, Exchange Online, Outlook, SharePoint Online, and OneDrive for Business.
[3]
[4]
It appears the cause of the issue is a networking configuration change, with the Windows giant admitting: "We've identified that a wide-area networking (WAN) routing change caused impact to the service. We've rolled back the change and monitoring the service as it recovers. Some of the customers who had previously reported impact are also reporting recovery."
Plenty of irate customers are contacting Microsoft via Twitter to share their problems, with a [5]typical example being one who said that: "VPN, SQL, Web Apps all affected."
[6]
Another [7]reported they were "unable to send/receive emails in outlook, Teams meetings are disconnecting in between, and O365 portals are loading very slowly. Critical Business activities are impacted majorly this morning."
Over on [8]Reddit , the chatter among sysadmins revealed that it was a global issue, with users from Germany, Switzerland Netherlands, Australia as well as the UK all reporting latency and other issues, with one in particular commenting that: "I guess the layoffs at MS didn't go down that well..." and another saying: "I feel bad for the guy who pulled the trigger on the change that has caused this firestorm."
A more recent status update from Microsoft stated that: "Our telemetry shows consistent signs of recovery from 0945 UTC onwards across multiple regions and services. Most customers should now see full recovery as WAN networking has recovered fully."
[9]Windows 10 paid downloads end but buyers need not fear ISO-lation
[10]Gonna run System Restore in Windows 11? Microsoft says some of its apps won't
[11]Microsoft locks door to default guest authentication in Windows Pro
[12]Microsoft fixes Windows database connections it broke in November
This latest outage follows [13]one in December that also affected cloud services such as Exchange Online and Teams across the Asia-Pacific region, while [14]another outage in July likewise affected Microsoft 365 services globally, but these are just a few of the incidents that happened within the last year.
Meanwhile, Microsoft reported only modest growth in its [15]earnings results for the quarter to the end of 2022, with consumer-centric products showing marked revenue dips and the company warning of slowing growth for Azure and associated cloud services.
[16]
CEO Satya Nadella ascribed this to customers "optimizing" their cloud spend, and predicted that growth would resume once Azure can deliver the AI customers want – assuming they can access those services, of course.
We asked Microsoft for a statement explaining today's service outage and will update this article when we get a response. ®
Get our [17]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://status.office365.com/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://twitter.com/al_kernow/status/1618158294459363329
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://twitter.com/ShoaibIshaqPAK/status/1618156508981923841
[8] https://www.reddit.com/r/sysadmin/comments/10kt06f/office_com_down_local_or_global/
[9] https://www.theregister.com/2023/01/24/windows_10_paid_downloads_end/
[10] https://www.theregister.com/2023/01/19/microsoft_windows_apps_restore/
[11] https://www.theregister.com/2023/01/17/microsoft_windows_pro_guest/
[12] https://www.theregister.com/2023/01/11/microsoft_database_connection_fix/
[13] https://www.theregister.com/2022/12/02/microsoft_teams_exchange_apac_outage/
[14] https://www.theregister.com/2022/07/21/teams_outage/
[15] https://www.theregister.com/2023/01/25/microsoft_q2_2023/
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://whitepapers.theregister.com/
Re: Advertising Standards
They've been secretly continuing the numbering since Windows for Workgroups, it should be Microsoft 3.65
Re: Advertising Standards
Don't be immature there is absolutely nothing wrong with Microsoft 365, it's reliable and our staff depend upon it to help facilitate meetings talking about their pets. If it wasn't for M364 I'm sure the bosses would have more of the staff back in the office. The only way we got M363 is due to COVID but at this rate the business case for M360 is looking weaker each year.
Re: Advertising Standards
Well I guess "365" is ok as it gives them a day of outage spread across 4 years?
Re: Advertising Standards
Isn't it funny, however, how all those outages seem to happen at the start of any such period?
If Microsoft keep using Europe as beta testers for the US I'll be investigating moving our entire company to Apple or Google (three major outages in as many weeks - all hitting Europe at/before 9am and all barely fixed before the US wakes up)
Google apps on Mac or even Chromebook would probably be best. Apple office apps aren’t man enough, and Office 365 on Mac has some features missing. 365 on iPad has so many features missing they should call it Office Lite. Macs probably have the longest software support, though there’s some oddities; 2017 MacBook Air gone out of support a year later than a 2012 MacBook Pro.
Not in a month of Sundays. Microsoft is already sniffing around trying to see if it can get a spy global operation up like Google (remember the "we will scan for your Office 365 copies using the next security update " thing?), but it's merely doing what it always does: trying to imitate another company which is already making a mint doing exactly that, and that is Google.
Here is a hint: read the Terms you have to agree to. Plan for a lot of coffee, and make some time for it, don't try to do it all in one go (they're designed to tire you out before you get to the dodgy bits, which is admittedly standard contract practice but MS, Facebook and Google have elevated that to almost an art form).
Once you have read them and actually understand what you've agreed to you may never touch Google again, ever.
Back to MS: the good thing about Microsoft trying to imitate another company's services is that it invariably cocks it up, badly, and that effort then joing a long line of other attempts to broaden their revenue stream.
They have since long given up being innovative themselves. The only innovation MS has really come up with over the years is the various methods by which they lock in and then fleece their victims customers..
I was called by a colleague this morning to say that email, teams etc. were down. I pointed out that I was on my day off and actually didn’t care. They said it was Microsoft that had cocked things up to which I said so I can’t do anything about it. Let IT support worry about it, I said that MSFT would doubtless fix it at some point and I was going back to sleep.
*looks smugly at LibreOffice icon*
Well gosh, maybe having everything depend on *the cloud* ain't such a good idea?
I tend to think of cloud services as a form of ransomware - you have to keep paying to have access to your data, and there's no guarantee that it is any more secure than it would have been when held locally. The only thing it seems to be good for is the profits of the cloud companies. I seem to remember when Bill Gates was in charge, one of his dreams was to have a recurring monthly income stream.
cloud services are not a good idea ...
... when everyone except you can access your data, and you have to pay th' Mill Owner for the privilege.
When will the day when torches and pitchforks be used against beancounters come, if it will ever?
Yes but no good if you can't access anything.
I couldn't access anything (Windows Laptop and Android handset) thanks to the global corp using Microsoft Authenticator, not mentioned but this wasn't working either.
Lovely morning, because I couldn't auth with Microsoft MFA, my VPN would connect but refuse to auth further, I couldn't access the internet because the Proxy software used couldn't authenticate me.
Also as per usual the Microsoft status website showed no issues until after 8AM, couldn't get through to helldesk so I was very confused (Although that is becoming more normal these days).
Yet again a morning of whinging at yet another single point of failure.
While in my private life I stay clear of cloud stuff, I cannot not see the vast advantage for businesses to have everything in the cloud. Namely: every business in one, single cloud. When The Cloud's offline, every business is offline so no one is bothered that their counterparts are not responsive.
Yup, it's a stupid idea, for many, many reasons, this just being in the top 3.
How are you conflating online services being down with a local install of Libre Office?
Teams outage
Productivity is up significantly
Re: Teams outage
Employees are also happier, power consumption has gone down
Re: Teams outage
power consumption has gone down
Teams makes the difference whether I'm cold in the office.
Icon, 'cause it's been cold.
"Critical Business activities are impacted majorly this morning."
Yeah, that's the white streak of fear that I made damned sure Manglement knew was a possibility when we migrated to Office365 several years ago. That we could come in one day, and Microsoft could have blown itself out of the water during the night, and nothing would work for the entire day. It was felt that the possibility of that was more than offset by us not having to manage (or pay for) an on-prem mail system, so we went full speed ahead. But yeah, Microsoft soiling itself was, and remains, a major menace to the system they're selling.
"CEO Satya Nadella ascribed this to customers "optimizing" their cloud spend, and predicted that growth would resume once Azure can deliver the AI customers want"
The only AI I "want" is one that tells the Microsoft techs "If you push out this config change, you will likely break all Office365 operations across all Microsoft domains. Are you Really Sure you want to do this?". Too bad Microsoft hasn't figured out how to do that yet. They certainly have enough failure event data points to train the system with, though.
Time for MS to pay for these outages
'F' the small print in their 100,000 page contracts.
Their ineptitude has cost businesses billions over recent weeks. Get that class action ready folks.
Start with $100B. Companies like MS need to be taught a lesson that while businesses depend on you, if you screw up then you should carry the can.
Re: Time for MS to pay for these outages
Why do you think they get all the beta testing done before it's 7am EST? The chances of class actions are remote.
Re: Time for MS to pay for these outages
" Get that class action ready folks "
If that happens and they they win, the lawyers will be laughing all the way to the bank and those represented will get five bucks each.
25 Days in to 2023
Office 365 is already at Office 364.
The first domino?
I'm wondering if this will be indicative of other tech companies showing lots of employees the door. Timber!
Re: The first domino?
Lotus Domino would mean having to use Lotus Notes
<
Re: The first domino?
Ahh but, I used Notes during a 8 year spell at IBM and do you know how many days it failed to work?
Re: The first domino?
I would guess at zero days.
Re: The first domino?
"...do you know how many days it failed to work?"
Yes but that's not the point of many comments regarding Lotus Notes.
The bloody program might have worked but with all the convoluted ins and outs did you manage to get any work done while struggling to get it to do what you wanted?
Re: The first domino?
Depends on the setup, I managed 98% uptime at my worst, but then it was a few Windows servers and not the infrastructure I wanted (i.e Lotus Domino made it super simple to replicate everything to a backup server or a server dedicated for webmail. All could be controlled through Notes networks, which no-one I spoke to ever knew wtf it was, often setup for dialup and never updated). Also had to deal with fighting for resources on the SAN, in the end I found using the inbuilt NSF compression lowered I/O requirements and the Blade had more than enough CPU resources that actually made it faster to respond. Never did use the de duplication (I did find in testing that was a problem)
I really did like Notes 9 when released, but at the time it was bloated and wasn't Outlook and I can't make people like Notes just because I did.
Re: The first domino?
We were working on a new project and the company had Lotus Notes, our company used Microsoft Exchange with all the trimmings.
And then the ILoveYou virus hit (which gives you some idea how long ago that was, it was even then already rubbish, and this was pre cloud, so on prem).
Our own company: about a 7 day outage (people working over the weekend included).
Company with Lotus Notes: not as much as a twitch during the original wave, and one infection about a week later when a secretary came back from holiday.
The key reason for the difference was because it was way, WAY harder to launch anything executable from Lotus Notes.
I recall another customer from those days who had two Lotus Notes setups. One they had running on a Windows setup, and the other one on Linux. The Linux guy spent most of his days drinking coffee watching the Windows people trying to keep the thing running - his ran just fine :).
Does not make sense
"Plenty of irate customers are contacting Microsoft via Twitter to share their problems"
Is Twitter still working?
Re: Does not make sense
Is Twitter still working?
Well they still have servers up, is that enough?
Re: Does not make sense
Duck Duck Go was down because Azure was wobbly I assume.
Re: Does not make sense
Doesn't it get its answers from Bing?
Re: Does not make sense
Yes, although no idea why you were downvoted for that.
Important cloudy metric: hours of unavailability (aka downtime)
Before you even consider moving cloudwards, you need to know for many hours the service will be unavailable per year and broken down into planned & planned downtime as well as which geographical region, amongst other dimensions.
It's usually written into SLAs and the figure offered by the cloudy offerers will have to be higher than what you currently get with your on-premises hardware and their support staff.
Re: Important cloudy metric: hours of unavailability (aka downtime)
you need to know for many hours the service will be unavailable per year
And how exactly are you going to get that data when it's all unplanned? Oh, wait, you'll ask Microsoft sales people on the golf course, of course, and naturally you even believe them..
Downtime?
Those deriding cloud seem to be missing a rather key issue - downtime happens onsite as well. And many organisations don't have the funds or expertise to maintain the same level of service as the big cloud players do.
Schools, for example. Running an on-prem email server is very difficult, vs "use Office 365". Having a fully resilient system is expensive - vs "use Office 365"...
Re: Downtime?
Is it possible that the impact of a local server(s) failing temporarily has an impact on one business, but when Microsoft fuck up it impacts many businesses, and the total cost is vastly more than a single business issue ?
I wonder how much this costs the UK for the outage as it affects so many people.
Re: Downtime?
But Office(insert number less than 365 here) is not resilient. What is the real cost of local support vs. cloud subscriptions? What is the real cost of downtime to an organisation? What about somewhere big, like the NHS, with Office offline?
When the cloud was first offered as a solution, it was assumed that your data would be available from anywhere at any time and it would be held in multiple locations to make it resilient in terms of access, security and backups. Everything is fine while it works, but one screw-up seems to be able to take it all down at once.
Re: Downtime?
Schools don't exist in isolation, or at least they don't in the UK, they are part of a local authority which are massive organisations, for instance the LA I work for, has something like 25,000 users, so not too sure why any school would have it's own email server. Prior to the cloud their email server would have been in a council computer centre somewhere. And yes there can be local outages, but there is usually a backup server, or traffic can be rerouted, whatever, my point being there are options. With the cloud all that is out of your hands. It's like the difference between driving and flying, at least when you're driving you are in control.
Re: Downtime?
LAs are not "in control" directly of most school IT. It is delegated and not centralised.
Trusts like Somerset, that used to run an email service for them, have abandoned that service - as the cost of running it is significant compared with Office 365 or Google.
The idea that a school or even an LA has the same level of support or capability as, say, Microsoft or Google is hilarious.
Re: Downtime?
downtime happens onsite as well
Yes, but then you can at least DO something about it, divert facilities, come up with alternatives - a whole raft of options exist (and can be planned for) that you do not have with the cloudy stuff.
Of course, they will tell you THEY have all that, but MS has yet again proven that they don't.
The difference between on prem and cloud is that when on prem goes down it's your problem and you can act, in the cloud it's EVERYBODY'S problem and you can do sod all other than wait and moan a lot. On a global scale, the impact on productivity is so much greater. Oh, and you don't have a third party holding your data hostage either.
After the already negative impact of having to use Microsoft products in the first place, going cloudy merely exacerbates it..
Re: Downtime?
You can do ALL of that with "cloudy stuff" as well. Don't mistake most organisations putting all their eggs in one basket as the only way of using cloud services.
Rollback procedure
Tomorrow in the morning meeting: "Ok, let's not ever install Windows on those servers again."
Hmm
Hmm, does seem to be a fundamental flaw with the cloud model, if someone snips the cord, your cloud will float away. But what do I know, I'm just an old timer who's been in the industry since the early 80's, I take it these young'uns know what they're doing? Can't lie it seemed better when you could go in to the machine room, and add another sack of coal to old Betsie, the mainframe node, when things got a bit sketchy.
Teams offline. Massive global upsurge in productivity reported.
The Cloud...
Other peoples computers you have no control over.
Advertising Standards
I think the ASA needs to have a word with them about the number '365' and what it means.
It's probably a warning to people out there not to put all your baskets into one cloud.