Atlassian comes clean on what data-deleting script behind outage actually did
- Reference: 1649943008
- News link: https://www.theregister.co.uk/2022/04/14/atlassian_ongoing_outage/
- Source link:
The restoration of customer data is still ongoing.
Atlassian CTO Sri Viswanath wrote that approximately 45 percent of those afflicted had had service restored but repeated the [2]fortnight estimate it gave earlier this week for undoing the damage to the rest of the affected customers. As of the time of writing, the figure of customers with restored data had risen to 49 per cent.
[3]
As for what actually happened… well, strap in. And no, you aren't reading another episode in our [4]Who, Me? series of columns where readers confess to massive IT errors.
[5]
[6]
"One of our standalone apps for Jira Service Management and Jira Software, called 'Insight – Asset Management,' was fully integrated into our products as native functionality," explained Viswanath, "Because of this, we needed to deactivate the standalone legacy app on customer sites that had it installed."
Two bad things then happened. First, rather than providing the IDs of the app marked for deletion, the team making the deactivation request provided the IDs of the entire cloud site where the apps were to be deactivated.
[7]
The team doing the deactivation then took that incorrect list of IDs and ran the script that did the 'mark for deletion magic.' Except that script had another mode, one that would permanently delete data for compliance reasons.
You can probably see where this is going. "The script was executed with the wrong execution mode and the wrong list of IDs," said Viswanath, with commendable honesty. "The result was that sites for approximately 400 customers were improperly deleted."
Yikes.
[8]At last, Atlassian sees an end to its outage ... in two weeks
[9]Day 7 of the great Atlassian outage: IT giant still struggling to restore access
[10]Atlassian outage lingers, sparking data loss fears
[11]Atlassian flags Bitbucket and Confluence Data Center flaws
The good news is that there are backups, and Atlassian retains them for 30 days. The bad news is that while the company can restore all customers into a new environment or roll back individual customers that accidentally delete their own data, there is no automated system to restore "a large subset" of customers into an existing environment, meaning data has to be laboriously pieced together.
The company is moving to a more automated process to speed things up, but currently is restoring customers in batches of up 60 tenants at a time, with four to five days required end-to-end before a site can be handed back to a customer.
[12]
"We know that incidents like this can erode trust," understated Viswanath.
Viswanath's missive did not mention compensation for businesses suffering a lengthy outage other than stating he and his team were committed to "doing what we can to make this right for you."
The Register contacted the company to clarify what this includes and will update should Atlassian respond.
With many other companies not being this transparent, especially at the point while the problem is still ongoing, it's commendable to get a proper explanation. ®
Get our [13]Tech Resources
[1] https://www.atlassian.com/engineering/april-2022-outage-update
[2] https://www.theregister.com/2022/04/11/atlassian_outage_backups/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YlhFMckEZ6bE3eRW0VQfzgAAAUE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://www.theregister.com/Tag/Who,%20Me?/
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YlhFMckEZ6bE3eRW0VQfzgAAAUE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YlhFMckEZ6bE3eRW0VQfzgAAAUE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YlhFMckEZ6bE3eRW0VQfzgAAAUE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2022/04/11/atlassian_outage_backups/
[9] https://www.theregister.com/2022/04/11/atlassian_still_down/
[10] https://www.theregister.com/2022/04/08/atlassian_service_issues_linger_leaving/
[11] https://www.theregister.com/2022/03/25/atlassian_hazelcast/
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/devops&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YlhFMckEZ6bE3eRW0VQfzgAAAUE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://whitepapers.theregister.com/
Re: Oof
An "honest mistake"? Having automated deletion scripts that don't verify the heck out of things, and require some sort of special mode required to delete entire sites?
That's not a mistake, that's incompetence.
Measure twice, cut once.
Cut once
...and always cut away from yourself.
Re: Cut once
"Cut toward your chum, not toward your thumb!"
Measure twice, cut once.
great advice until you realise that the the detail of what to measure from your colleague was wrong so you've accurately cut a wrong measurement.
BT,DT.
immediately thought of
Precisely. The circuits that cannot be cut are cut automatically in response to a terrorist incident. You asked for miracles, Theo. I give you the F...B...I...
GDPR
Is there nothing it can't screw up?
To be fair, I think when I heard of all of this I was expecting something monumentally stupid. Like "rm -rf" in the wrong folder of the server and no back ups performed. This though, it's fairly honest and happens to all of us at some point.
Re: GDPR
if wasn't rm -rf. it was shred /dev/disk/customers
Every Cloud Problem has Silver Lining?
I don't know if it's related or not, but since these Atlassian problems I've not been getting my daily junk mails from various Atlassian domains.
While I appreciate the honesty...
-> The bad news is that while the company can restore all customers ... there is no automated system to restore "a large subset" of customers into an existing environment, meaning data has to be laboriously pieced together.
That doesn't strike me as very good at all. It seems more like a reconstruction from whatever is available rather than a backup/restore. Be warned: if it can happen once it can happen again. So Atlassian should design a better recovery system.
Re: While I appreciate the honesty...
And... "The company is moving to a more automated process to speed [restoration] up"
well 'automation' caused the problem in the first place, so that should go well...
Re: While I appreciate the honesty...
Don't you mean a better backup system?
The script was executed with the wrong execution mode and the wrong list of IDs
It's the wrong trousers Gromit! And they've gone wrong!
Sh*t Happens
No matter how many safeguards you build, checks you put in place or precautions you take the fuckup fairy will come calling sooner or later. The more systems you manage, the sooner she's likely to get to you - there is no escape.
That's why we have things like backup strategies and RTO's, so that when she does visit it's not a company-ending event. At least they've been honest about what happened and how long it's going to take to put it right. No marketing spin. No fluff. Just an honest 'we screwed up, sorry'. They should be commended for that at least.
Their lackluster RTO on the other hand isn't so easily forgiven....
Oof
This sounds like a series of horrible, dumb, but honest mistakes. Also, I have to commend them on their honesty and directness; most would obfuscate, but this seems like a clear request for forgiveness.
Icon for all involved, but the victims and people currently fixing the problem get first dibs.