Major ISP bungles settings, causing Microsoft 365, Azure outage
(2024/09/12)
- Reference: 1726151111
- News link: https://www.theregister.co.uk/2024/09/12/att_microsoft_365_outage/
- Source link:
updated If you're having issues logging into Azure this morning, no - it's not just you: Microsoft has confirmed an issue.
The Azure status [1]page , as of writing, shows that Microsoft is investigating connectivity issues between Microsoft services and a major ISP's network, but provides no other details.
Frantic messaging on network admin listservs and [2]Reddit alike suggests this ISP is AT&T. Reports of issues with Microsoft services started appearing around 1300 UTC, when messages began appearing on the outages listserv from AT&T customers indicating trouble connecting to Outlook and other Microsoft services. Others report the Azure portal has since come back online, but there's no indication whether service has actually been restored.
[3]
Microsoft has since posted on its cloud service health status [4]page that the issue caused trouble with other Microsoft 365 services as well, and was tied to a system change on an unnamed ISP's end.
[5]
[6]
"The ISP has reverted the change and we're now seeing signs of recovery," Microsoft noted. "We're continuing to monitor network telemetry data to ensure a full recovery."
Microsoft and AT&T have been contacted for comment. This story will be updated as we learn more.
[7]
This is the second time in several months that Azure customers have had to deal with a widespread outage. In late July, customers found themselves unable to access Azure and some Microsoft 365 services. In this case, however, the problem was coming from inside the house: Microsoft's own DDoS defenses [8]amplified an attack instead of mitigating it. Oops. ®
Updated at 1500 UTC on September 12
AT&T confirmed that it experienced “a brief disruption connecting to some Microsoft services” on its network, but doesn’t suspect foul play.
[9]
“The issue has been resolved and connections are operating normally,” a spokesperson told The Register . “We are investigating the disruption and there is no indication of foul play.”
Microsoft also confirmed the resolution of the outage. "A third-party Internet Service Provider incident that impacted a subset of their customers' ability to access our services is now fully resolved," a spokesperson told us.
Get our [10]Tech Resources
[1] https://azure.status.microsoft/en-us/status
[2] https://www.reddit.com/r/ATT/comments/1ff1ed8/microsoft_is_down_on_att_networks/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://status.cloud.microsoft/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2024/07/31/microsoft_ddos_azure/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://whitepapers.theregister.com/
The Azure status [1]page , as of writing, shows that Microsoft is investigating connectivity issues between Microsoft services and a major ISP's network, but provides no other details.
Frantic messaging on network admin listservs and [2]Reddit alike suggests this ISP is AT&T. Reports of issues with Microsoft services started appearing around 1300 UTC, when messages began appearing on the outages listserv from AT&T customers indicating trouble connecting to Outlook and other Microsoft services. Others report the Azure portal has since come back online, but there's no indication whether service has actually been restored.
[3]
Microsoft has since posted on its cloud service health status [4]page that the issue caused trouble with other Microsoft 365 services as well, and was tied to a system change on an unnamed ISP's end.
[5]
[6]
"The ISP has reverted the change and we're now seeing signs of recovery," Microsoft noted. "We're continuing to monitor network telemetry data to ensure a full recovery."
Microsoft and AT&T have been contacted for comment. This story will be updated as we learn more.
[7]
This is the second time in several months that Azure customers have had to deal with a widespread outage. In late July, customers found themselves unable to access Azure and some Microsoft 365 services. In this case, however, the problem was coming from inside the house: Microsoft's own DDoS defenses [8]amplified an attack instead of mitigating it. Oops. ®
Updated at 1500 UTC on September 12
AT&T confirmed that it experienced “a brief disruption connecting to some Microsoft services” on its network, but doesn’t suspect foul play.
[9]
“The issue has been resolved and connections are operating normally,” a spokesperson told The Register . “We are investigating the disruption and there is no indication of foul play.”
Microsoft also confirmed the resolution of the outage. "A third-party Internet Service Provider incident that impacted a subset of their customers' ability to access our services is now fully resolved," a spokesperson told us.
Get our [10]Tech Resources
[1] https://azure.status.microsoft/en-us/status
[2] https://www.reddit.com/r/ATT/comments/1ff1ed8/microsoft_is_down_on_att_networks/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://status.cloud.microsoft/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2024/07/31/microsoft_ddos_azure/
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZuMQISonb2P5fVKwFPfuGQAAAdA&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://whitepapers.theregister.com/
The weakest component
alain williams
A system is only as reliable as the weakest component. The more parts there are the more likely that one will break and bring down the whole house of cards.
There is a cost equation to evaluate, which is greater: outsourcing IT to someone's cloud; the cost of not being able to do business when the cloud is not available.
A big unknown is how often said cloud goes down.
Hopefully in the near future ...
Eclectic Man
... it will feature in the Register's ongoing 'Who, Me?' series.
Not for laughs, you understand, but so that we can all understand and learn from someone else's 'experience'.
DNS
...it's always DNS.