'Our hosted pools are under attack by abusers': Azure DevOps enjoys a midweek TITSUP*
(2021/04/07)
- Reference: 1617813190
- News link: https://www.theregister.co.uk/2021/04/07/azure_devops/
- Source link:
Engineers around the world have been given a few hours off today, assuming they are using Microsoft's Azure DevOps Hosted Pools, as the platform buckled under a second wave of attacks.
Issues first cropped up in the early hours of the morning (UK time), afflicting "all geographies" according to Microsoft and was described delicately as "an event" by the engineers attempting to put things back in order.
[1]
Over the course of five hours Microsoft's team battled to deal with "the incident" before declaring the issue as "fully mitigated" just as UK workers fired up their workstations to kick off the day's pipeline party.
Sadly, the respite was not to last and by [2]15:06 UK (14:06 UTC) Ed Glas posted a refreshingly blunt update to the effect that all was not well. "Our hosted pools are under attack by abusers," he said. "We knocked down the first wave of abuse but they came back with another attack. We are working to block this attack."
[3]
'Anomalous surge in DNS queries' knocked Microsoft's cloud off the web last week [4]READ MORE
The exact nature of the attack is unclear at the moment, although Microsoft's [5]wobbly infrastructure allowed a host of the company's services to come crashing down on 1 April, an event [6]attributed to "an anomalous surge in DNS queries."
Uh huh.
Nick, a Register -reading tech boffin, got in touch regarding the problem, and remarked that his outfit had seen build times jump by between 20 and 50 per cent. Other users have been similarly affected and taken their woes to social media.
Experiencing the same issue. It's taking an hour or more to acquire an agent... See also: [7]https://t.co/mZ7ETgP8aQ — Richard (@badsyntax) [8]April 7, 2021
The service has been a little up and down of late. A shame, because the likes of Ubuntu, Windows and macOS virtual machines courtesy of Microsoft are convenient alternatives to self-hosting.
When they work.
A [9]glimpse at the history shows that the last week has not been a good one for Azure DevOps. As well as this morning's whoopsie, there was also "degradation" yesterday. One user, attempting to use Ubuntu, [10]grumbled that the pipelines for the UK at least "were essentially down all afternoon & evening."
The Register understands that the current issues can be laid, at least partially, at the doorstep of opportunistic crypto-mining.
Free CI/CD (including hosted agents) has been a thing at Azure DevOps since 2018, but Microsoft has spent this year dealing with abuses of its largesse, first by [11]fiddling with the free tier in public projects and, last month, [12]being more selective about the free grant in private projects.
We have contacted Microsoft for more details, and will update should an explanation be forthcoming. ®
[13]
* Tottering Infrastructure Topples Software User Programming
Get our [14]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YG4rfzdGkqdP5ktbteshjAAAAAs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://status.dev.azure.com/_event/235501110
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YG4rfzdGkqdP5ktbteshjAAAAAs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2021/04/06/in_brief_security/
[5] https://www.theregister.com/2021/04/06/in_brief_security/
[6] https://status.azure.com/en-us/status/history/
[7] https://t.co/mZ7ETgP8aQ
[8] https://twitter.com/badsyntax/status/1379776496454238211?ref_src=twsrc%5Etfw
[9] https://status.dev.azure.com/_history
[10] https://twitter.com/Pouriaaa/status/1379748636742025217
[11] https://devblogs.microsoft.com/devops/change-in-azure-pipelines-grant-for-public-projects/
[12] https://devblogs.microsoft.com/devops/change-in-azure-pipelines-grant-for-private-projects/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YG4rfzdGkqdP5ktbteshjAAAAAs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://whitepapers.theregister.com/
Issues first cropped up in the early hours of the morning (UK time), afflicting "all geographies" according to Microsoft and was described delicately as "an event" by the engineers attempting to put things back in order.
[1]
Over the course of five hours Microsoft's team battled to deal with "the incident" before declaring the issue as "fully mitigated" just as UK workers fired up their workstations to kick off the day's pipeline party.
Sadly, the respite was not to last and by [2]15:06 UK (14:06 UTC) Ed Glas posted a refreshingly blunt update to the effect that all was not well. "Our hosted pools are under attack by abusers," he said. "We knocked down the first wave of abuse but they came back with another attack. We are working to block this attack."
[3]
'Anomalous surge in DNS queries' knocked Microsoft's cloud off the web last week [4]READ MORE
The exact nature of the attack is unclear at the moment, although Microsoft's [5]wobbly infrastructure allowed a host of the company's services to come crashing down on 1 April, an event [6]attributed to "an anomalous surge in DNS queries."
Uh huh.
Nick, a Register -reading tech boffin, got in touch regarding the problem, and remarked that his outfit had seen build times jump by between 20 and 50 per cent. Other users have been similarly affected and taken their woes to social media.
Experiencing the same issue. It's taking an hour or more to acquire an agent... See also: [7]https://t.co/mZ7ETgP8aQ — Richard (@badsyntax) [8]April 7, 2021
The service has been a little up and down of late. A shame, because the likes of Ubuntu, Windows and macOS virtual machines courtesy of Microsoft are convenient alternatives to self-hosting.
When they work.
A [9]glimpse at the history shows that the last week has not been a good one for Azure DevOps. As well as this morning's whoopsie, there was also "degradation" yesterday. One user, attempting to use Ubuntu, [10]grumbled that the pipelines for the UK at least "were essentially down all afternoon & evening."
The Register understands that the current issues can be laid, at least partially, at the doorstep of opportunistic crypto-mining.
Free CI/CD (including hosted agents) has been a thing at Azure DevOps since 2018, but Microsoft has spent this year dealing with abuses of its largesse, first by [11]fiddling with the free tier in public projects and, last month, [12]being more selective about the free grant in private projects.
We have contacted Microsoft for more details, and will update should an explanation be forthcoming. ®
[13]
* Tottering Infrastructure Topples Software User Programming
Get our [14]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YG4rfzdGkqdP5ktbteshjAAAAAs&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://status.dev.azure.com/_event/235501110
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YG4rfzdGkqdP5ktbteshjAAAAAs&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://www.theregister.com/2021/04/06/in_brief_security/
[5] https://www.theregister.com/2021/04/06/in_brief_security/
[6] https://status.azure.com/en-us/status/history/
[7] https://t.co/mZ7ETgP8aQ
[8] https://twitter.com/badsyntax/status/1379776496454238211?ref_src=twsrc%5Etfw
[9] https://status.dev.azure.com/_history
[10] https://twitter.com/Pouriaaa/status/1379748636742025217
[11] https://devblogs.microsoft.com/devops/change-in-azure-pipelines-grant-for-public-projects/
[12] https://devblogs.microsoft.com/devops/change-in-azure-pipelines-grant-for-private-projects/
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YG4rfzdGkqdP5ktbteshjAAAAAs&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[14] https://whitepapers.theregister.com/
Re: Has to be said.
Anomalous Cowturd
May I propose simply turning it off...
Anonymous Coward
Well its the cloud right and you chose it. So you wanted a non-deterministic infrastructure for your enterprise right?
Basically your decision was incorrect. Enjoy.
Sounds about right
HildyJ
When the pool is abused, you drain the pool, identify the abuser, and chlorinate everything.
I never get tired of saying it
ecofeco
How that's cloud thing working out for ya?
Has to be said.
Have they tried turning it off and back on again.