Microsoft's Azure networking takes a worldwide tumble
- Reference: 1722348373
- News link: https://www.theregister.co.uk/2024/07/30/microsofts_azure_portal_outage/
- Source link:
According to the Windows giant's [1]social media orifice for all things Microsoft 365-related, "We're currently investigating access issues and degraded performance with multiple Microsoft 365 services and features."
The problems appear to have started around 1130 UTC.
[2]
A glance at the [3]Azure status dashboard indicates that all things Network Infrastructure-related are having issues on a global scale. The mega-corp's last message on the matter stated, "We are investigating reports of issues connecting to Microsoft services globally. Customers may experience timeouts connecting to Azure services. We have multiple engineering teams engaged to diagnose and resolve the issue. More details will be provided as soon as possible."
[4]
[5]
In an unfortunate turn of events, no sooner had [6]Microsoft's Azure Support mouthpiece suggested to users that they test their app's resilience with the Azure Chaos Studio, it appears the biz has kicked off its own chaos testing in production.
One UK-based Register reader noted glumly the outage "has broken most of our access to stuff."
[7]
"Can't access the portal which has stalled our development right now. I think prod is down also."
Another reader got in touch to tell us: "Initially access to the Portal wasn't possible. The attempted connection timed out. After a while, the message on the status page changed, and then I was able to connect to the portal, but only a very limited subset of my resources were visible.
"Basically, I could see Resource Groups, but none of the VMs, database servers etc. Those machines seem all to be up and working properly, but the portal isn't showing that they even exist."
[8]
It seems like a perfect excuse for a sunny afternoon in the pub — unless, of course, you actually need to get some work done while Azure staggers back to its feet.
The Register has contacted Microsoft to learn more about its latest outage.
[9]French internet cables cut in act of sabotage that caused outages across country
[10]Azure VMs ruined by CrowdStrike patchpocalypse? Microsoft has recovery tips
[11]Windows Patch Tuesday update might send a user to the BitLocker recovery screen
[12]Microsoft 365 remains 'degraded' as Azure outage resolved
The breakdown will doubtless trigger a cold sweat in admins after the events less than two weeks ago, in which an [13]Azure wobble preceded the arrival of [14]CrowdStrike's horseman of the borkpocalypse.
Reader Sam C said: "It looks like their Front Door CDN service is unavailable, for about the last 1.5 hours.. causing us all sorts of fun.. websites down, portals down (including service status portals!)"
He quipped: "As someone asked on Twitter, 'Have they just installed Crowdstrike'!"
Despite the experience of many of our readers (and so many others on various social media platforms), Microsoft's status pages continue to insist that all is well other than that pesky Network Infrastructure. Then again, if you can't access the Azure Portal, which appears to be the experience of many users, you'll likely struggle with other parts of the service.
Microsoft is scheduled to release its latest financial figures today, July 30. Apparently Azure will be a focus. Hopefully, the IT giant will manage to shoehorn how it intends to stop Azure from falling over in between all the AI-powered bragging. ®
Updated to add at 1445 UTC
Microsoft said it had implemented networking configuration changes and "have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we are continuing to monitor to ensure full recovery."
[15]
Screenshot of Microsoft's latest messaging on its Azure status page, indicating networking failures
There are still red hazard signs across the board.
Updated to add at 1509 UTC
MSFT says its telemetry has shown improvement in service availability "from approximately 14:10 UTC onwards, and we are continuing to monitor to ensure full recovery."
Updated to add at 1523 UTC
Now the UK court system has been [16]borked by Azure !
HM Courts and Tribunals Service took to Xitter to say: "We are aware of users experiencing issues accessing multiple online services. This appears to relate to a global Microsoft Azure outage."
Ahem.
"We apologise for any inconvenience caused."
Updated to add at 1800 UTC
Microsoft has narrowed the outage down to its networking infrastructure and its CDN-like [17]Azure Front Door service, which is left customers with "issues connecting to Microsoft services globally."
Can't get Minecraft, MongoDB Cloud, others to work today? Blame Azure [18]READ NEXT
"An unexpected usage spike resulted in Azure Front Door (AFD) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes," the tech goliath said, explaining that though it's fixing the issue even that is causing follow-up problems.
"We have implemented network configuration changes and have performed failovers to provide alternate network paths for relief," Microsoft explained.
"As we investigate reports of specific services and regions that are still experiencing intermittent errors, we believe that our network configuration changes have successfully mitigated the impacts of the usage spike, but that these changes are causing some side effects to certain services.
"We are updating our mitigation approach to minimize these side effects, and applying these following Safe Deployment Practices - beginning in Asia Pacific regions and then expanding in phases."
Get our [19]Tech Resources
[1] https://x.com/MSFT365Status
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zqliiyqe2isTVX76iqntkQAAAEk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://azure.status.microsoft/en-gb/status
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zqliiyqe2isTVX76iqntkQAAAEk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zqliiyqe2isTVX76iqntkQAAAEk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://x.com/AzureSupport/status/1818240131838714291
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zqliiyqe2isTVX76iqntkQAAAEk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zqliiyqe2isTVX76iqntkQAAAEk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2024/07/29/french_fiber_cables_cut/
[10] https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/
[11] https://www.theregister.com/2024/07/24/windows_update_bitlocker/
[12] https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/
[13] https://www.theregister.com/2024/07/19/microsoft_365_azure_outage_central_us/
[14] https://www.theregister.com/2024/07/19/azure_vms_ruined_by_crowdstrike/
[15] https://regmedia.co.uk/2024/07/30/image002.png
[16] https://x.com/HMCTSgovuk/status/1818293470479417702
[17] https://learn.microsoft.com/en-us/azure/frontdoor/front-door-overview
[18] https://www.theregister.com/2024/07/30/azure_outage_impact/
[19] https://whitepapers.theregister.com/
Re: MS...
If Microsoft operated an airline you'd have pay £6.99 for the journey, £150 for "upgrades" (like a seat, seat-belt). £49 for "optional security extras" (shared use of wings and a landing gear), £199 for the pilot, a further £599 for a security product to ensure the pilot isn't pissed, and you'd have to book it by calling a number in India, answered by someone who only understands you if you talk like Hugh Grant. If your flight leaves on Monday, they'll arrange to call you back, on Wednesday.
Re: MS...
Ryanair
Re: MS...
Now Now !!!
Even I would not sink that *low* !!!
Remember Boeing *kills* people ... MS is a regular inconvenience that they telegraph well ahead of time
[Something to do with 'Tuesday' so I believe.] !!!
:)
Re: MS...
Any idea if any of the multiple outages on Windows at hospitals or related medical facilities has caused death? I suspect so, even if not as spectacular as a air crash, I'm sure the outcome for many needing treatment has been downgraded.
Re: MS...
There was a guy in the Hacker News comment section who 'nearly had a heart attack' at a surprise Azure bill. Would that count?
Re: MS...
SW Boeing… Ouch. That is a very bad, but true analogy.
Re: MS...
What does that make the customers who choose to rely on it?
Re: MS...
Stuck?
Re: MS...
Insane.
As in, doing the same thing over and over and expecting different results.
Crowdstrike, perhaps. If their updates are behind other operators...?
Nope this is a MS problem, 100%.
Seemed to be more of a internal routing problem.. Or someone forgot to pointed the Azure ( sorry Entra) servers to the wrong IP or one of a million other possible errors that a company that size will undoubtedly make.
This might sound stupid but "We pay for shit to happen and it does". MS will always guarantee that side of things for us.
Or DNS
It's always DNS <\sarc>
Unless it's email, in which case it's probably still DNS. :D
SaaS
Software and assorted stoppages
Re: SaaS
Software application available sometimes
Re: SaaS
DTaaS - Down Time as a Service
Status Page: Ha, Ha, Ha!
Please can Microsoft et all either fix their status pages (I know, I'm being ridiculously unrealistic here) or just get rid of them as they always claim that everything's rosy in the Seattle garden.
It appears to be that as long as a single horse trader in Ulaanbaatar can access the system that counts as 100% uptime globally and so the status pages are worse than useless. No doubt because stating otherwise would mean that Big Tech would have to admit it isn't perfect in every possible way (and presumably SLA $$$ would be due to some organisations).
Just a thought. As AI is so wonderful and can solve all of the world's problems, could they not just point Copilot at it /s?
Re: Status Page: Ha, Ha, Ha!
< href=https://azure.status.microsoft/en-gb/status>https://azure.status.microsoft/en-gb/status
Re: Status Page: Ha, Ha, Ha!
[1]https://azure.status.microsoft/en-gb/status
FTFY
[1] https://azure.status.microsoft/en-gb/status
Re: Status Page: Ha, Ha, Ha!
The Microsoft 365 status pages (and the admin alerts they push out to Outlook) seem to always line up with actual issues, in fact if I was going to credit Microsoft with doing one thing well, it might even be that. Compared with, for example, pretty much every ISP I've ever had dealings with, whose status pages almost always deny there's anything wrong, even when you're dealing with a multi-day outage.
Re: Status Page: Ha, Ha, Ha!
Does the SLA even cover the portal (pretty sure it only covers services and resources)? Technically, they were unaffected.
Quis custodiet ipsos custodes?
Only after the problem's fixed as far as I can tell (Entra, Intune etc. are now fine and NatWest, Nationwide are both up as well).
I'm guessing that the status service itself runs on Azure - what could possibly go wrong?
Re: Quis custodiet ipsos custodes?
Sorry Stewart, I deleted my post as it should have been in reply to the post above. I didn't see that you had made a reply in between time.
Backbone borked?
Azure has what Microsoft describe as a "backbone network".
Effectively an internal network that operates within Azure's infrastructure and doesn't expose any traffic to the wider internet. Supposedly excellent for connecting all your Azure thingy things.
It's redundant in the sense of the same thing has been repeated a lot of times.
Given everything is fucked I'd suspect it's got something to do with that.
Unexpected territory access to Exchange
We have an Azure Exchange instance hosted in the US. Users are in California and, in the office, are behind a firewall that geofilters. Last week we started noticing that Outlook was trying to access Exchange using IP addresses, apparently belonging to Microsoft, that geolocated to Chile, Brazil and India. Eventually Outlook would find an IP address that wasn't blocked. Microsoft support told us on Saturday that there was an outage but didn't give me any more information.
Re: Unexpected territory access to Exchange
You've whitelisted the recommended Microsoft IP ranges corresponding to your services right?
Or at least fixed it after the "Microsoft outage" that affected the pool of customers that hadn't set up their firewall rules correctly?
I.e.
https://learn.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide
Filter for appropriate services and regions...
Re: Unexpected territory access to Exchange
Blimey - that's some spaghetti mess of domain names ! No wonder they keep forgetting to renew SSL certs ....
"access issues and degraded performance"
What, again ?
So, where are we at, Office 300 ?
Because it has never been Office 365.
Microsoft 365 now down to 359 this year? (so far)
Never forget that 365 is...
...an octal number.
Microsoft suggested to users that they test their app's resilience with the Azure Chaos Studio
Doesn't matter how resilient your app is if the computers they run are up and down like a bride's nightie.
Re: Microsoft suggested to users that they test their app's resilience with the Azure Chaos Studio
How can we start a petition to get El Reg to refer to Microsoft as the "Azure Chaos Studio" from now on?
Re: Microsoft suggested to users that they test their app's resilience with the Azure Chaos Studio
Surely Microsoft need to use their own Azure Chaos Studio on their network. Once they have it all working and nothing ever fails, then and only then can they preach to everyone else.
Currently this just looks like a story of "we have a tool, but don't use it, but you should use it" implying that in some way, it would solve THEIR problem.. Muppets.
Re: Microsoft suggested to users that they test their app's resilience with the Azure Chaos Studio
Perhaps they did, and that's the problem.
This won't matter
Just like the Azure outage which coincided with ClownStrike. Our CxOs are all-in with Azure and cloud migration. Nothing will change.
And so we get the software that we get.
Re: This won't matter
Keeps in me in work, keeps me busy, keeps paying my bills, paid my mortgage and currently fattening up my pension both with my income and the shares in the pension funds. Early retirement is just a a few years away now fingers crossed.
Re: This won't matter
Make hay while the sun shines
"Share and enjoy!" – Sirius Cybernetics Corporation
"We apologise for any inconvenience caused." – That should be Microsoft's corporate motto.
Re: "Share and enjoy!" – Sirius Cybernetics Corporation
Oh, I thought it was “bend over, we’re going to enjoy this”
Ah the delicious irony
As of last week I am 'at leisure' since I supported a non-MS product which was considered not sexy enough for a 'forward thinking progressive finance company'. Their solution was to throw out bath water and baby and go fully MS to have a tightly integrated global 'one size fits none' solution albeit at orders higher cost than the previous smorgasbord of products which each did their specific function perfectly and could talk to each other quite happily.
I hear there may have been a few issues which may have impacted every element of the day to day business and my heart goes out to all those poor buggers left scrambling to explain why they are not able to fix anything as the problems are 'in the cloud'...you know, that totally resilient, can never be broken, reach it from anywhere solution which the CxO people all believe is for 'the Greater Good'
Once it stops being so pleasantly sunny here I may drop the guys an email to see how it's going although that does require them to be able to get into O365 (insert alternative names here) to read it
Re: Ah the delicious irony
Condolences on the change in work level if it wasn't desired.
'Forward thinking progressive finance company'" sounds like a few antithetical juxtapositions back to back. And if going all in with one vendor is "Forward Thinking" I am a dancer with the Paris Opera Ballet.
“ HM Courts and Tribunals Service took to Twitter / X..”
And so lay bare the lie that G-Cloud was different and separate to public Microsoft cloud…
Less than three weeks ago
I wrote this:
"The more I read of cloud services the more I think you'd have to be an idiot to rid get rid of your on prem infrastructure in favour of it"
And got downvoted for it.
Still feel smug now?
Re: Less than three weeks ago
I get this all the time, when I ask how that cloud thing is working for them.
Boing, Boing, Boing
Well, it was working for a bit but now Microsoft content is being served slowly and RBS/NatWest is down again (no doubt Fred Goodwin is blaming the latter on an insidious invasion of pink wafer biscuits in the boardroom that would never have happened in his day).
"We have multiple engineering teams engaged to diagnose and resolve the issue."
Two teams, each pointing their fingers at the other?
You need three teams and hope that two both point at the third.
MS...
Software's Boeing.