Microsoft breaks geolocation, locking users out of Azure and M365
(2023/03/24)
- Reference: 1679637432
- News link: https://www.theregister.co.uk/2023/03/24/microsoft_geolocation_fail_uzbekistan/
- Source link:
Microsoft has found a new and interesting way to break its cloud services: by messing up geolocation services and sending its users to Uzbekistan, which made it impossible for them to log in.
The Beast of Redmond let the world know about the mess with this oblique tweet regarding Microsoft 365:
We're investigating an issue where users with specific conditional access policies applied may be unable to access any Microsoft 365 service. We're reverting a recent change to mitigate impact. For more details please look for MO531859 in the admin center. — Microsoft 365 Status (@MSFT365Status) [1]March 23, 2023
A kind Reg reader sent more detail: a Microsoft status notice that defined the problem as "a subset of users with geolocation-based conditional access policies experienced sign-in disruptions to Azure services."
The reader's info said the mess was caused by "a recent deployment applied to an infrastructure for regulating user geolocation had inadvertently provided incorrect IP location data."
This resulted in users who had an IP-based conditional access policy experiencing the sign-in issues mentioned above.
[2]
Our reader sent us an image of the error message seen by his clients in Darwin, Australia, which explained that Microsoft would not allow access to Teams because the traffic came from Toshkent, Uzbekistan.
[3]
[4]
Chileans also appear to have been sent to Uzbekistan if this Tweet is any guide:
the problem occurs because they are changing the IP to another country. [5]pic.twitter.com/Fe5WhtWeTJ — Marcos (@marck_al) [6]March 23, 2023
And so were United Statesians:
Not cool. Sending US-based traffic to Uzbekistan is unacceptable on many levels. [7]pic.twitter.com/ZD6Wlmeaea — Brian Wilson (@brianwilson@infosec.exchange) (@brianwilson) [8]March 23, 2023
Interestingly, the UK's National Health Service, a [9]known Microsoft 365 user , [10]listed the incident in its tech support feed. Hopefully the mess didn't impact services there.
[11]Microsoft admits Azure Resource Manager failed after code change
[12]Microsoft's Copilot AI to pervade the whole 365 suite
[13]WAN router IP address change blamed for global Microsoft 365 outage
[14]Microsoft Office 365 Cloud has a secret lining
Microsoft's Twitter thread about the incident suggests it was sorted out within about nine hours.
Users disputed that and did not appreciate Microsoft's handling of the situation.
Wow started 0300 GMT and now you tell us! I reported it on at 0830 GMT from the UK and was advised then that it was more than just ASPAC region. Come on chaps, I rely on these tweets as usually faster updates than the admin centre! — Arli (@ArliSunblade) [15]March 23, 2023
Making matters worse, the IP addresses in question appear to be IPv4. Shame, Microsoft, get thee to IPv6!
Those of you with M365 admin accounts may be able to find a more detailed status report with the identifier "MO531859". If you read it, and it details the precise duration of the outage, please let us know in a comment so that when we next report an M365 outage we can accurately rename the service to reflect its actual uptime. ®
Get our [16]Tech Resources
[1] https://twitter.com/MSFT365Status/status/1638878984225329155?ref_src=twsrc%5Etfw
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZB2C18M4xQ5m4rPDOFI@DwAAAAk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZB2C18M4xQ5m4rPDOFI@DwAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZB2C18M4xQ5m4rPDOFI@DwAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://t.co/Fe5WhtWeTJ
[6] https://twitter.com/marck_al/status/1638893554998149120?ref_src=twsrc%5Etfw
[7] https://t.co/ZD6Wlmeaea
[8] https://twitter.com/brianwilson/status/1638903277793300481?ref_src=twsrc%5Etfw
[9] https://www.theregister.com/2020/06/15/microsoft_roundup/
[10] https://support.nhs.net/2023/03/microsoft-365-alert-service-degradation-microsoft-365-suite-some-users-with-conditional-access-policies-may-be-unable-to-access-any-microsoft-365-service/
[11] https://www.theregister.com/2023/03/23/microsoft_admits_azure_resource_manager/
[12] https://www.theregister.com/2023/03/16/microsoft_365_copilot_ai_teaser/
[13] https://www.theregister.com/2023/01/30/wan_router_ip_address_change/
[14] https://www.theregister.com/2023/01/31/microsoft_office_365_dod/
[15] https://twitter.com/ArliSunblade/status/1638884766840766466?ref_src=twsrc%5Etfw
[16] https://whitepapers.theregister.com/
The Beast of Redmond let the world know about the mess with this oblique tweet regarding Microsoft 365:
We're investigating an issue where users with specific conditional access policies applied may be unable to access any Microsoft 365 service. We're reverting a recent change to mitigate impact. For more details please look for MO531859 in the admin center. — Microsoft 365 Status (@MSFT365Status) [1]March 23, 2023
A kind Reg reader sent more detail: a Microsoft status notice that defined the problem as "a subset of users with geolocation-based conditional access policies experienced sign-in disruptions to Azure services."
The reader's info said the mess was caused by "a recent deployment applied to an infrastructure for regulating user geolocation had inadvertently provided incorrect IP location data."
This resulted in users who had an IP-based conditional access policy experiencing the sign-in issues mentioned above.
[2]
Our reader sent us an image of the error message seen by his clients in Darwin, Australia, which explained that Microsoft would not allow access to Teams because the traffic came from Toshkent, Uzbekistan.
[3]
[4]
Chileans also appear to have been sent to Uzbekistan if this Tweet is any guide:
the problem occurs because they are changing the IP to another country. [5]pic.twitter.com/Fe5WhtWeTJ — Marcos (@marck_al) [6]March 23, 2023
And so were United Statesians:
Not cool. Sending US-based traffic to Uzbekistan is unacceptable on many levels. [7]pic.twitter.com/ZD6Wlmeaea — Brian Wilson (@brianwilson@infosec.exchange) (@brianwilson) [8]March 23, 2023
Interestingly, the UK's National Health Service, a [9]known Microsoft 365 user , [10]listed the incident in its tech support feed. Hopefully the mess didn't impact services there.
[11]Microsoft admits Azure Resource Manager failed after code change
[12]Microsoft's Copilot AI to pervade the whole 365 suite
[13]WAN router IP address change blamed for global Microsoft 365 outage
[14]Microsoft Office 365 Cloud has a secret lining
Microsoft's Twitter thread about the incident suggests it was sorted out within about nine hours.
Users disputed that and did not appreciate Microsoft's handling of the situation.
Wow started 0300 GMT and now you tell us! I reported it on at 0830 GMT from the UK and was advised then that it was more than just ASPAC region. Come on chaps, I rely on these tweets as usually faster updates than the admin centre! — Arli (@ArliSunblade) [15]March 23, 2023
Making matters worse, the IP addresses in question appear to be IPv4. Shame, Microsoft, get thee to IPv6!
Those of you with M365 admin accounts may be able to find a more detailed status report with the identifier "MO531859". If you read it, and it details the precise duration of the outage, please let us know in a comment so that when we next report an M365 outage we can accurately rename the service to reflect its actual uptime. ®
Get our [16]Tech Resources
[1] https://twitter.com/MSFT365Status/status/1638878984225329155?ref_src=twsrc%5Etfw
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZB2C18M4xQ5m4rPDOFI@DwAAAAk&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZB2C18M4xQ5m4rPDOFI@DwAAAAk&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZB2C18M4xQ5m4rPDOFI@DwAAAAk&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://t.co/Fe5WhtWeTJ
[6] https://twitter.com/marck_al/status/1638893554998149120?ref_src=twsrc%5Etfw
[7] https://t.co/ZD6Wlmeaea
[8] https://twitter.com/brianwilson/status/1638903277793300481?ref_src=twsrc%5Etfw
[9] https://www.theregister.com/2020/06/15/microsoft_roundup/
[10] https://support.nhs.net/2023/03/microsoft-365-alert-service-degradation-microsoft-365-suite-some-users-with-conditional-access-policies-may-be-unable-to-access-any-microsoft-365-service/
[11] https://www.theregister.com/2023/03/23/microsoft_admits_azure_resource_manager/
[12] https://www.theregister.com/2023/03/16/microsoft_365_copilot_ai_teaser/
[13] https://www.theregister.com/2023/01/30/wan_router_ip_address_change/
[14] https://www.theregister.com/2023/01/31/microsoft_office_365_dod/
[15] https://twitter.com/ArliSunblade/status/1638884766840766466?ref_src=twsrc%5Etfw
[16] https://whitepapers.theregister.com/
43300
Sometimes they fix a problem without ever actually ackowledging that it even existed!
AndrueC
Across all their product lines. I reported a bug in Visual Studio (a debug window no longer reopening at its last size and position). After asking for more information (which I provided). They changed my report to a feature request. Then they closed it because it was 'out of scope with our general product direction'. Despite requests for more information there have been no further responses.
No wonder VS is in the state it is.
IPv6 still the poor relation chez Microsoft
plunet
"Making matters worse, the IP addresses in question appear to be IPv4. Shame, Microsoft, get thee to IPv6!"
That's because last time I checked Microsoft offered no geolocation on IPv6. If you don't offer it you can't break it.
Coastal cutie
I've lost count - are we down to M350 yet?
At least they're professionals
AnotherName
Just imagine how bad it would be if they were amateurs!
Microsoft are terrible at admitting problems. It's not unusual for us to log a support ticket and then shortly after Microsoft announce they've fixed a problem that started many hours ago.