News: 1674646234

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Global network outage hits Microsoft: Azure, Teams, Outlook all down

(2023/01/25)


Microsoft is currently rolling back a network change across its wide area network that it believes toppled over a raft of its cloud services this morning, perhaps in solidarity with the company's tumbling profits.

Users all over the world are venting their anger on social media at not being able to send or receive email, connect to Teams meetings or access many other services.

The problems started at around 0705 UTC on January 25 and involved issues with network connectivity for millions of users of Azure, Teams and Outlook, when they attempted to connect with Azure resources, which naturally includes Microsoft's cloud-hosted services.

[1]

A [2]Service Health Status update from the company at 0810 UTC admitted that: "Users may be unable to access multiple Microsoft 365 services," which include but were not limited to Teams, Exchange Online, Outlook, SharePoint Online, and OneDrive for Business.

[3]

[4]

It appears the cause of the issue is a networking configuration change, with the Windows giant admitting: "We've identified that a wide-area networking (WAN) routing change caused impact to the service. We've rolled back the change and monitoring the service as it recovers. Some of the customers who had previously reported impact are also reporting recovery."

Plenty of irate customers are contacting Microsoft via Twitter to share their problems, with a [5]typical example being one who said that: "VPN, SQL, Web Apps all affected."

[6]

Another [7]reported they were "unable to send/receive emails in outlook, Teams meetings are disconnecting in between, and O365 portals are loading very slowly. Critical Business activities are impacted majorly this morning."

Over on [8]Reddit , the chatter among sysadmins revealed that it was a global issue, with users from Germany, Switzerland Netherlands, Australia as well as the UK all reporting latency and other issues, with one in particular commenting that: "I guess the layoffs at MS didn't go down that well..." and another saying: "I feel bad for the guy who pulled the trigger on the change that has caused this firestorm."

A more recent status update from Microsoft stated that: "Our telemetry shows consistent signs of recovery from 0945 UTC onwards across multiple regions and services. Most customers should now see full recovery as WAN networking has recovered fully."

[9]Windows 10 paid downloads end but buyers need not fear ISO-lation

[10]Gonna run System Restore in Windows 11? Microsoft says some of its apps won't

[11]Microsoft locks door to default guest authentication in Windows Pro

[12]Microsoft fixes Windows database connections it broke in November

This latest outage follows [13]one in December that also affected cloud services such as Exchange Online and Teams across the Asia-Pacific region, while [14]another outage in July likewise affected Microsoft 365 services globally, but these are just a few of the incidents that happened within the last year.

Meanwhile, Microsoft reported only modest growth in its [15]earnings results for the quarter to the end of 2022, with consumer-centric products showing marked revenue dips and the company warning of slowing growth for Azure and associated cloud services.

[16]

CEO Satya Nadella ascribed this to customers "optimizing" their cloud spend, and predicted that growth would resume once Azure can deliver the AI customers want – assuming they can access those services, of course.

We asked Microsoft for a statement explaining today's service outage and will update this article when we get a response. ®

Get our [17]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://status.office365.com/

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://twitter.com/al_kernow/status/1618158294459363329

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[7] https://twitter.com/ShoaibIshaqPAK/status/1618156508981923841

[8] https://www.reddit.com/r/sysadmin/comments/10kt06f/office_com_down_local_or_global/

[9] https://www.theregister.com/2023/01/24/windows_10_paid_downloads_end/

[10] https://www.theregister.com/2023/01/19/microsoft_windows_apps_restore/

[11] https://www.theregister.com/2023/01/17/microsoft_windows_pro_guest/

[12] https://www.theregister.com/2023/01/11/microsoft_database_connection_fix/

[13] https://www.theregister.com/2022/12/02/microsoft_teams_exchange_apac_outage/

[14] https://www.theregister.com/2022/07/21/teams_outage/

[15] https://www.theregister.com/2023/01/25/microsoft_q2_2023/

[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y9FgO70sVgj0XociU4zQ3QAAAMI&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[17] https://whitepapers.theregister.com/



Advertising Standards

AnotherName

I think the ASA needs to have a word with them about the number '365' and what it means.

It's probably a warning to people out there not to put all your baskets into one cloud.

Re: Advertising Standards

Zippy´s Sausage Factory

They've been secretly continuing the numbering since Windows for Workgroups, it should be Microsoft 3.65

Re: Advertising Standards

Anonymous Coward

Don't be immature there is absolutely nothing wrong with Microsoft 365, it's reliable and our staff depend upon it to help facilitate meetings talking about their pets. If it wasn't for M364 I'm sure the bosses would have more of the staff back in the office. The only way we got M363 is due to COVID but at this rate the business case for M360 is looking weaker each year.

Re: Advertising Standards

Anonymous Coward

Well I guess "365" is ok as it gives them a day of outage spread across 4 years?

Re: Advertising Standards

Charlie Clark

Isn't it funny, however, how all those outages seem to happen at the start of any such period?

Anonymous Coward

If Microsoft keep using Europe as beta testers for the US I'll be investigating moving our entire company to Apple or Google (three major outages in as many weeks - all hitting Europe at/before 9am and all barely fixed before the US wakes up)

FirstTangoInParis

Google apps on Mac or even Chromebook would probably be best. Apple office apps aren’t man enough, and Office 365 on Mac has some features missing. 365 on iPad has so many features missing they should call it Office Lite. Macs probably have the longest software support, though there’s some oddities; 2017 MacBook Air gone out of support a year later than a 2012 MacBook Pro.

Anonymous Coward

Not in a month of Sundays. Microsoft is already sniffing around trying to see if it can get a spy global operation up like Google (remember the "we will scan for your Office 365 copies using the next security update " thing?), but it's merely doing what it always does: trying to imitate another company which is already making a mint doing exactly that, and that is Google.

Here is a hint: read the Terms you have to agree to. Plan for a lot of coffee, and make some time for it, don't try to do it all in one go (they're designed to tire you out before you get to the dodgy bits, which is admittedly standard contract practice but MS, Facebook and Google have elevated that to almost an art form).

Once you have read them and actually understand what you've agreed to you may never touch Google again, ever.

Back to MS: the good thing about Microsoft trying to imitate another company's services is that it invariably cocks it up, badly, and that effort then joing a long line of other attempts to broaden their revenue stream.

They have since long given up being innovative themselves. The only innovation MS has really come up with over the years is the various methods by which they lock in and then fleece their victims customers..

JimboSmith

I was called by a colleague this morning to say that email, teams etc. were down. I pointed out that I was on my day off and actually didn’t care. They said it was Microsoft that had cocked things up to which I said so I can’t do anything about it. Let IT support worry about it, I said that MSFT would doubtless fix it at some point and I was going back to sleep.

Antony Shepherd

*looks smugly at LibreOffice icon*

Well gosh, maybe having everything depend on *the cloud* ain't such a good idea?

AnotherName

I tend to think of cloud services as a form of ransomware - you have to keep paying to have access to your data, and there's no guarantee that it is any more secure than it would have been when held locally. The only thing it seems to be good for is the profits of the cloud companies. I seem to remember when Bill Gates was in charge, one of his dreams was to have a recurring monthly income stream.

cloud services are not a good idea ...

Anonymous Coward

... when everyone except you can access your data, and you have to pay th' Mill Owner for the privilege.

Potemkine!

When will the day when torches and pitchforks be used against beancounters come, if it will ever?

Captain Scarlet

Yes but no good if you can't access anything.

I couldn't access anything (Windows Laptop and Android handset) thanks to the global corp using Microsoft Authenticator, not mentioned but this wasn't working either.

Lovely morning, because I couldn't auth with Microsoft MFA, my VPN would connect but refuse to auth further, I couldn't access the internet because the Proxy software used couldn't authenticate me.

Also as per usual the Microsoft status website showed no issues until after 8AM, couldn't get through to helldesk so I was very confused (Although that is becoming more normal these days).

Yet again a morning of whinging at yet another single point of failure.

Evil Auditor

While in my private life I stay clear of cloud stuff, I cannot not see the vast advantage for businesses to have everything in the cloud. Namely: every business in one, single cloud. When The Cloud's offline, every business is offline so no one is bothered that their counterparts are not responsive.

yakacm

Yup, it's a stupid idea, for many, many reasons, this just being in the top 3.

Dave Null

How are you conflating online services being down with a local install of Libre Office?

Teams outage

I Am Spartacus

Productivity is up significantly

Re: Teams outage

elsergiovolador

Employees are also happier, power consumption has gone down

Re: Teams outage

Evil Auditor

power consumption has gone down

Teams makes the difference whether I'm cold in the office.

Icon, 'cause it's been cold.

Pirate Dave

"Critical Business activities are impacted majorly this morning."

Yeah, that's the white streak of fear that I made damned sure Manglement knew was a possibility when we migrated to Office365 several years ago. That we could come in one day, and Microsoft could have blown itself out of the water during the night, and nothing would work for the entire day. It was felt that the possibility of that was more than offset by us not having to manage (or pay for) an on-prem mail system, so we went full speed ahead. But yeah, Microsoft soiling itself was, and remains, a major menace to the system they're selling.

"CEO Satya Nadella ascribed this to customers "optimizing" their cloud spend, and predicted that growth would resume once Azure can deliver the AI customers want"

The only AI I "want" is one that tells the Microsoft techs "If you push out this config change, you will likely break all Office365 operations across all Microsoft domains. Are you Really Sure you want to do this?". Too bad Microsoft hasn't figured out how to do that yet. They certainly have enough failure event data points to train the system with, though.

Time for MS to pay for these outages

Steve Davies 3

'F' the small print in their 100,000 page contracts.

Their ineptitude has cost businesses billions over recent weeks. Get that class action ready folks.

Start with $100B. Companies like MS need to be taught a lesson that while businesses depend on you, if you screw up then you should carry the can.

Re: Time for MS to pay for these outages

Dan 55

Why do you think they get all the beta testing done before it's 7am EST? The chances of class actions are remote.

Re: Time for MS to pay for these outages

Mike 137

" Get that class action ready folks "

If that happens and they they win, the lawyers will be laughing all the way to the bank and those represented will get five bucks each.

25 Days in to 2023

wolfetone

Office 365 is already at Office 364.

The first domino?

chivo243

I'm wondering if this will be indicative of other tech companies showing lots of employees the door. Timber!

Re: The first domino?

Korev

Lotus Domino would mean having to use Lotus Notes

<> and nukes from orbit (to make sure)

Re: The first domino?

Nursing A Semi

Ahh but, I used Notes during a 8 year spell at IBM and do you know how many days it failed to work?

Re: The first domino?

Scott Wichall

I would guess at zero days.

Re: The first domino?

nematoad

"...do you know how many days it failed to work?"

Yes but that's not the point of many comments regarding Lotus Notes.

The bloody program might have worked but with all the convoluted ins and outs did you manage to get any work done while struggling to get it to do what you wanted?

Re: The first domino?

Captain Scarlet

Depends on the setup, I managed 98% uptime at my worst, but then it was a few Windows servers and not the infrastructure I wanted (i.e Lotus Domino made it super simple to replicate everything to a backup server or a server dedicated for webmail. All could be controlled through Notes networks, which no-one I spoke to ever knew wtf it was, often setup for dialup and never updated). Also had to deal with fighting for resources on the SAN, in the end I found using the inbuilt NSF compression lowered I/O requirements and the Blade had more than enough CPU resources that actually made it faster to respond. Never did use the de duplication (I did find in testing that was a problem)

I really did like Notes 9 when released, but at the time it was bloated and wasn't Outlook and I can't make people like Notes just because I did.

Re: The first domino?

Anonymous Coward

We were working on a new project and the company had Lotus Notes, our company used Microsoft Exchange with all the trimmings.

And then the ILoveYou virus hit (which gives you some idea how long ago that was, it was even then already rubbish, and this was pre cloud, so on prem).

Our own company: about a 7 day outage (people working over the weekend included).

Company with Lotus Notes: not as much as a twitch during the original wave, and one infection about a week later when a secretary came back from holiday.

The key reason for the difference was because it was way, WAY harder to launch anything executable from Lotus Notes.

I recall another customer from those days who had two Lotus Notes setups. One they had running on a Windows setup, and the other one on Linux. The Linux guy spent most of his days drinking coffee watching the Windows people trying to keep the thing running - his ran just fine :).

Does not make sense

trevorde

"Plenty of irate customers are contacting Microsoft via Twitter to share their problems"

Is Twitter still working?

Re: Does not make sense

Paul Crawford

Is Twitter still working?

Well they still have servers up, is that enough?

Re: Does not make sense

Dan 55

Duck Duck Go was down because Azure was wobbly I assume.

Re: Does not make sense

Neil Barnes

Doesn't it get its answers from Bing?

Re: Does not make sense

Dan 55

Yes, although no idea why you were downvoted for that.

Important cloudy metric: hours of unavailability (aka downtime)

deadlockvictim

Before you even consider moving cloudwards, you need to know for many hours the service will be unavailable per year and broken down into planned & planned downtime as well as which geographical region, amongst other dimensions.

It's usually written into SLAs and the figure offered by the cloudy offerers will have to be higher than what you currently get with your on-premises hardware and their support staff.

Re: Important cloudy metric: hours of unavailability (aka downtime)

Anonymous Coward

you need to know for many hours the service will be unavailable per year

And how exactly are you going to get that data when it's all unplanned? Oh, wait, you'll ask Microsoft sales people on the golf course, of course, and naturally you even believe them..

Downtime?

localzuk

Those deriding cloud seem to be missing a rather key issue - downtime happens onsite as well. And many organisations don't have the funds or expertise to maintain the same level of service as the big cloud players do.

Schools, for example. Running an on-prem email server is very difficult, vs "use Office 365". Having a fully resilient system is expensive - vs "use Office 365"...

Re: Downtime?

NewModelArmy

Is it possible that the impact of a local server(s) failing temporarily has an impact on one business, but when Microsoft fuck up it impacts many businesses, and the total cost is vastly more than a single business issue ?

I wonder how much this costs the UK for the outage as it affects so many people.

Re: Downtime?

AnotherName

But Office(insert number less than 365 here) is not resilient. What is the real cost of local support vs. cloud subscriptions? What is the real cost of downtime to an organisation? What about somewhere big, like the NHS, with Office offline?

When the cloud was first offered as a solution, it was assumed that your data would be available from anywhere at any time and it would be held in multiple locations to make it resilient in terms of access, security and backups. Everything is fine while it works, but one screw-up seems to be able to take it all down at once.

Re: Downtime?

yakacm

Schools don't exist in isolation, or at least they don't in the UK, they are part of a local authority which are massive organisations, for instance the LA I work for, has something like 25,000 users, so not too sure why any school would have it's own email server. Prior to the cloud their email server would have been in a council computer centre somewhere. And yes there can be local outages, but there is usually a backup server, or traffic can be rerouted, whatever, my point being there are options. With the cloud all that is out of your hands. It's like the difference between driving and flying, at least when you're driving you are in control.

Re: Downtime?

localzuk

LAs are not "in control" directly of most school IT. It is delegated and not centralised.

Trusts like Somerset, that used to run an email service for them, have abandoned that service - as the cost of running it is significant compared with Office 365 or Google.

The idea that a school or even an LA has the same level of support or capability as, say, Microsoft or Google is hilarious.

Re: Downtime?

Anonymous Coward

downtime happens onsite as well

Yes, but then you can at least DO something about it, divert facilities, come up with alternatives - a whole raft of options exist (and can be planned for) that you do not have with the cloudy stuff.

Of course, they will tell you THEY have all that, but MS has yet again proven that they don't.

The difference between on prem and cloud is that when on prem goes down it's your problem and you can act, in the cloud it's EVERYBODY'S problem and you can do sod all other than wait and moan a lot. On a global scale, the impact on productivity is so much greater. Oh, and you don't have a third party holding your data hostage either.

After the already negative impact of having to use Microsoft products in the first place, going cloudy merely exacerbates it..

Re: Downtime?

localzuk

You can do ALL of that with "cloudy stuff" as well. Don't mistake most organisations putting all their eggs in one basket as the only way of using cloud services.

Rollback procedure

Sceptic Tank

Tomorrow in the morning meeting: "Ok, let's not ever install Windows on those servers again."

Hmm

yakacm

Hmm, does seem to be a fundamental flaw with the cloud model, if someone snips the cord, your cloud will float away. But what do I know, I'm just an old timer who's been in the industry since the early 80's, I take it these young'uns know what they're doing? Can't lie it seemed better when you could go in to the machine room, and add another sack of coal to old Betsie, the mainframe node, when things got a bit sketchy.

Anonymous Coward

Teams offline. Massive global upsurge in productivity reported.

The Cloud...

Anonymous Coward

Other peoples computers you have no control over.

Youth is not a time of life, it is a state of mind; it is a temper of
the will, a quality of the imagination, a vigor of the emotions, a predominance
of courage over timidity, of the appetite for adventure over love of ease.
Nobody grows old by merely living a number of years; people grow
old only by deserting their ideals. Years wrinkle the skin, but to give up
enthusiasm wrinkles the soul. Worry, doubt, self-distrust, fear, and despair
-- these are the long, long years that bow the head and turn the growing spirit
back to dust.
Whether seventy or sixteen, there is in every being's heart the love
of wonder, the sweet amazement at the stars and the starlike things and
thoughts, the undaunted challenge of events, the unfailing childlike appetite
for what next, and the joy and the game of life.
You are as young as your faith, as old as your doubt; as young as your
self-confidence, as old as your fear, as young as your hope, as old as your
despair.
So long as your heart receives messages of beauty, cheer, courage,
grandeur and power from the earth, from man, and from the Infinite, so long
you are young.
-- Samuel Ullman