Take the morning off because Outlook has already
- Reference: 1675768145
- News link: https://www.theregister.co.uk/2023/02/07/outlook_down/
- Source link:
According to outage tracker DownDetector, reports began coming in of users facing a 500 error and being unable to send, receive or search email through Outlook.com from about 4am UTC, peaking at 8 and 9am as Europeans reached their desks.
Microsoft confirmed the outage on its [1]service health website , saying: "We're applying targeted mitigations to a subset of affected infrastructure and validating that it has mitigated impact. We're also making traffic optimization efforts to alleviate user impact and expedite recovery."
[2]
It added that extra "Outlook.com functionality such as Calendar APIs consumed by other services such as Microsoft Teams are also affected."
[3]
[4]
At the time of writing, the blackout appears to be ongoing. As for what caused it, the Microsoft 365 Status Twitter account [5]said : "We've confirmed that a recent change is contributing to the cause of impact. We're working on potential solutions to restore availability of the service."
In plain English, Microsoft tweaked something and the house of cards came tumbling down, so they'll probably have to revert the change. It offered the reference number EX512238 to track in the admin center and otherwise directed users to watch the service health page for any updates.
[6]
Until then, you have The Register's blessing to slack off – just check with your boss that you don't have anything outstanding that can be done without the aid of Outlook.
[7]Warning: Microsoft Teams Free (classic) will be gone in 2 months
[8]Microsoft injects AI into Teams so no one will ever forget what the meeting decided
[9]Microsoft warns some Azure usage notifications – including abnormalities – are broken
[10]WAN router IP address change blamed for global Microsoft 365 outage
As ever, Microsoft's legendary approach to quality control is incredibly poorly timed because today the company is supposed to host a [11]super-duper mystery event at Redmond that is thought to include a relaunch of its search engine Bing, now with added ChatGPT from OpenAI!
Microsoft has invested heavily in OpenAI and is already spraying its wares all over its own products, [12]including Teams and [13]Azure . Let us hope that Microsoft didn't prematurely flip the AI switch on Outlook, [14]like this Twitter user suggested .
[15]Another said to forget about AI and ML, "We need better HI.. Human Inteligence
sic
to prevent global outage on a weekday."Once again, Microsoft proves that the 365 branding was a terrible choice that will come back to haunt them again and again. If we add the [16]global outage from January to this Outlook wobble, we're probably down to Microsoft 364 already and it's only the first week of February.
[17]
Maybe just go with 360, like the Xbox, and add the additional days as another paid tier. Wait, that might give someone at Microsoft ideas... ®
Get our [18]Tech Resources
[1] https://portal.office.com/servicestatus
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Y@KDsgKUh8ZqG4OYAYF-EgAAANE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y@KDsgKUh8ZqG4OYAYF-EgAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y@KDsgKUh8ZqG4OYAYF-EgAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://twitter.com/MSFT365Status/status/1622828574217576448
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Y@KDsgKUh8ZqG4OYAYF-EgAAANE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/02/07/microsoft_warns_teams_free_classic/
[8] https://www.theregister.com/2023/02/02/microsoft_teams_premium_ai/
[9] https://www.theregister.com/2023/02/02/azure_savings_plans_notifications_broken/
[10] https://www.theregister.com/2023/01/30/wan_router_ip_address_change/
[11] https://www.bloomberg.com/news/articles/2023-02-06/microsoft-msft-announces-tuesday-news-event-as-ai-bets-accelerate
[12] https://www.theregister.com/2023/02/02/microsoft_teams_premium_ai/
[13] https://www.theregister.com/2023/01/17/azure_openai_chatgpt/
[14] https://twitter.com/edwin_macharia/status/1622872868630548481
[15] https://twitter.com/spartychemist/status/1622897641796390912
[16] https://www.theregister.com/2023/01/30/wan_router_ip_address_change/
[17] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/applications&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Y@KDsgKUh8ZqG4OYAYF-EgAAANE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[18] https://whitepapers.theregister.com/
I prefer to call it Office 356.
I prefer to use something that actually works.
We had two last month, and defender deleting shortcuts
Looks like Defender was right.
Aaaaand we have the Comment of the Week, and it's not even Wednesday.
LOL :)
Microsoft proves that the 365 branding was a terrible choice
I suggest renaming it "Microsoft Occasionally".
Re: Microsoft proves that the 365 branding was a terrible choice
In a similar vein, back when I worked in (a consultancy doing work for) travel, the industry's alternative name for recently-deceased airline Flybe (including by direct employees) was Flymaybe.
Re: Microsoft proves that the 365 branding was a terrible choice
There were always a few of those kicking around:
BMI-be (Bee Em Aye-bee)
SleazyJet
Cryanair
British Hellways
Vermin Atlantic
Some funnier than others.
Re: funnier than others
Such A Bloody Experience Never Again.
Teeny Weeny Airlines.
Ratshit (a little known Aussie airline)
Re: funnier than others
Or the old Italian one:
Arrive Late In Turin, All Luggage in Athens.
Re: Microsoft proves that the 365 branding was a terrible choice
The Youth of Today. We had Better On A Camel, But Will It Arrive, Better Wait In Airport, Britain’s Worst Investment Abroad, and Panned American and The Clipped Wings of Man. (For those who don’t know… BOAC, BWIA, BWIA again, BWIA yet again, yes, they were that bad, Pan Am, and Eastern.)
United was the subject of different jokes, mostly about ahem, flying united. Delta had redneck jokes, often involving banjo music after a certain Burt Reynolds movie came out.
Re: Microsoft proves that the 365 branding was a terrible choice
Ah yes, I remember SleazyJet and Cryanair. I seem to recall that our internal names for Ryanair and Virgin however were
Re: Microsoft proves that the 365 branding was a terrible choice
Were? Still are! :-)
And yes, I'm proud to use some of these.
Re: Microsoft proves that the 365 branding was a terrible choice
And of course the always predictable Portuguese Take Another Plane
Re: Microsoft proves that the 365 branding was a terrible choice
Whoever said that it was intended as 365 base 10?
For the pessimists, how about base 8 (245 decimal ), or for the contrary, base 7 (194 decimal).
BTW re acronyms for airlines below, allegedly related to passenger "care" :
Try Walking Across
Don't Even Let Them Aboard.
Vegas?
Do bookies in Vegas take bets on the actual number associated with Office3xx? That would be a good pool to run! Now what would be a good number to pick? I see someone has already picked O356!
Re: Vegas?
and if it goes down to Office69, you get really fucked...
Re: Vegas?
At least you will then enjoy that it sucks.
I'll go and hide now :).
Re: Vegas?
"At least you will then enjoy that it sucks."
Doesn't it depend on who is sucking? Or would that be M1cr0$oft??
Re: Vegas?
There used to be a website called microsith.com
I've just looked. It's not there. Time for a ride on the Wayback Engine!
Another WAN router IP address change? Or is it something else? The wheels do seem to be coming off Microsoft's cloud ventures with alarming regularity lately.
The wheels on the cloud go bork bork bork
Bork bork bork
Bork bork bork
The wheels on the cloud go bork bork bork
3
6
5
Bullsh*t Bingo Winner?
Oh please:
"We're applying targeted mitigations to a subset of affected infrastructure and validating that it has mitigated impact. We're also making traffic optimization efforts to alleviate user impact and expedite recovery."
Is the excuse department on piece rates? Lookout chaps, ChatGPT is coming for you.
Re: Bullsh*t Bingo Winner?
"We're applying targeted mitigations to a subset of affected infrastructure and validating that it has mitigated impact. We're also making traffic optimization efforts to alleviate user impact and expedite recovery."
in other words: it's gone TIT5UP, AWOL and MIA...but WTS and it might be BSS.
As a world spanning corporation...
...it's notable that MS change management happens when it's least likely to affect the home market so rest of the world gets 12 hours or so of beta testing. Or am I I ascribing maliciousness where only ignorance is in play? Maybe the people making the decision on when to apply changes don't understand that time zones around the world cover more than the 4 or so hours most "lower 48ers" have a vague understanding of?
Re: As a world spanning corporation...
Saw it happen with AWS many moons ago too — entirety of Europe went down for many hours (and I mean everything , including all of Amazon's own websites) with complete radio silence from Amazon for the duration. Didn't appear on a single news website that I could see.
People are overall a lot cloudier these days, but it was telling as an outage like that in a US region would've been widely reported.
Re: As a world spanning corporation...
Maybe it was just you :)
Re: As a world spanning corporation...
:D unfortunately not — you could see the usual backlash on the nascent Twitter et al. but all we could do was sit and watch and pray. To be fair it was overnight but a good six-to-eight hour outage affecting every service.
Of course it turned out to be DNS. All service status pages were down too, but they barely logged any problems retrospectively once they were back up.
I think things are probably a hell of a lot better now, but even ten-minute cloud outages in the US were making headlines back then (probably about 2013 or 2014).
Re: As a world spanning corporation...
Maybe or maybe not a conspiracy. But more or less fits in with midnight US time for the change to be implement. Just about perfect for Operations Center based in IST time. Europe is still bleary eyed and in need of a coffee.
Then needs an hour or two for the failure to propagate.
And that's about 0900 CET, 0800 UK time.
Re: As a world spanning corporation...
To be fair (I know... its not really done on t'internet is it?) in this case the issue was intially reported by M$ as affecting North America only, however they changed their tune when Europe woke up.
"February 7, 2023 4:48 AM..... Scope of impact: Users primarily located in the North American region attempting to access Exchange may be unable to send, receive, or search email."
"February 7, 2023 6:58 AM..... Scope of impact: Users attempting to access Exchange may be unable to send, receive, or search email. This issue is not isolated to North America."
Above quotes taken directly from the Service Health portal.
But, in general, I agree; it does appear as though Microsoft test things on "the rest of the world" rather than on home turf.
Re: As a world spanning corporation...
I initially read that as "world spannering".
-A.
What a wonderful morning it has been!
Starting my working day with ZERO email! Slack was a bit busier than normal, but never opened Teams (as the Americans aren't yet awake) so great day all around.
Seeing this article wrecks my day as I was looking forward to being productive w/o Outlook!
Office 65
It's like 69 except you are licking your empty wallet and Nanoflaccid is right behind you.
The day will come when some sort of vulnerability will be detected in the O365 cloud, but when they'll try to patch that vuln, it'll bring the whole O365 cloud down... no matter what they do, the cloud goes down as soon as they patch it...
Now that will be an ELE.
That's already there. It's called being a Microsoft product.
In other news
FitBit has been out of commission since yesterday and, despite a brief recovery this morning, is back down again.
Re: In other news
Or maybe you're a vampire now.
Re: Vampire
So that’s why I avoid sunlight.
M365 goundhog day
Same old, same old, the service is getting too big to cope.
I hope you added this Availability issue as a verified risk when you pushed everything from "On-Prem" to Azure?
Wires and duct tape everywhere. To be expected when you build a world-aailable service based on the absence of a QA department.
QA Dept.
I thought that was us. No?
Heh heh heh
> If you sat down to work this morning and attempted to do something as routine as check your emails with Outlook, you'd be bang out of luck.
Where I work we have to be on-prem. Cloud isnt an option.
Sometimes I like to be a bit of a jerk so...
Everythings working fine here :D
Oulook was called Lookout for a good reasons
and this is just another one of them.
MS Really needs to be sued into the next century for their inability to deliver a service that customers pay for. Who will be first to file suit?
If you do then you'd better have a plan to move away from all MS products ready to go.
Proudly MS software free since 2016.
Re: Oulook was called Lookout for a good reasons
"If you do then you'd better have a plan to move away from all MS products ready to go."
Or just backtrack all MS products to the pre-cloudy versions - I'm happily running older MS products direct off my boot drive (and behind a suitable hardware firewall) and hardly ever lose any work time (mores the pity':-( ) due to some borked MS patches that have not been properly validated prior to being pushed out onto their servers for use by those who've paid through the nose for a 24/7 service.
Re: Oulook was called Lookout for a good reasons
Microsoft have 'Patch Tuesday' for you, and forced updates via Windows Update, and Defender updates that hose your shortcuts, and...
Re: Oulook was called Lookout for a good reasons
About that, who would be fool enough to install a patch, any patch, from any vendor, in the first week after release?
You just wait for the beta testers to install it, watch the fallout in the forums and, if all seems OK(ish), you bite the bullet. If the forums go nuclear you just wait for the patching patch. Rinse and repeat.
Mail servers can and should just run and run.
Sendmail is acknowleged to be have its problems but once configured tends to run and run. Other mail servers such as Exim, Postfix, Dovevot are known to be easier to configure and also just run and run. Combine these with rock solid, stable OSes, such as FreeBSD, and you can have systems that run happily run for decades without a break.
Then you have Exchange…
Think it's actually Office 363 now, as we had that outage last month.