That Salesforce outage: Global DNS downfall started by one engineer trying a quick fix
- Reference: 1621446548
- News link: https://www.theregister.co.uk/2021/05/19/salesforce_root_cause/
- Source link:
"We're not blaming one employee," said Chief Availability Officer Darryn Dieken after spending the first half hour of a Wednesday briefing on the outage doing pretty much that very thing.
To recap, on May 11 around 2100 UTC, a configuration change was applied to Salesforce's Domain Name System (DNS) servers that resulted in folks [1]unable to access the software-as-a-service titan's products. For about five hours, clients could not reliably log in, and things got so bad that even the status page was unavailable.
[2]
[3]
[4]
Salesforce has been updating its public [5]root cause analysis ever since, and Dieken said during his briefing to customers that a few more tweaks would be needed before the fix was completed.
It was during that call that the full extent of the screw-up was revealed and the engineer concerned launched buswards.
While Dieken boasted of the automation in place within Salesforce towers, some processes remain manual. One of these is related to DNS (yes, it is always DNS.) A lone engineer was tasked with making a configuration change to connect up a new Salesforce Hyperforce environment in Australia.
[6]Salesforce fell over so hard today, it took out its own server status page
[7]Microsoft sheds some light on perplexing Outlook blank email incident: Word was to blame
[8]Trend Micro hosted email service is down, inboxes still stuck in cloudy limbo
[9]OVH outlines three-point 'hyper resilience' plan after Strasbourg fire
A DNS change is not an uncommon occurrence, and the engineer also had a four-year-old script to do the job. However, while Salesforce usually "staggers" changes to reduce the blast radius of blunders, the manual nature of this change meant it was up to the engineer to roll it out slowly.
This, alas, did not happen. The engineer instead decided erroneously, according to Dieken, to shortcut the normal procedures by using a so-called Emergency Break-Fix (EBF) process. The EBF is normally used when something really bad is happening, or an emergency patch is quickly and widely needed.
[10]
Going down the EBF route meant fewer approvals and a shortened process that wasn't gradual. Hey, this was a well-used script, the engineer had worked for Salesforce for years and these changes were pretty common. What could possibly go wrong?
In classic [11]Who, Me? fashion, rather a lot.
We don't understand
"For whatever reason that we don't understand, the employee decided to do a global deployment," Dieken went on. The usual staggered approach was therefore bypassed. And a DNS change meant those servers would need restarting.
That in itself would not be a total catastrophe. Maybe a short outage, perhaps. But not the disaster that unfolded.
However, it transpired that lurking within that tried-and-trusted script was a bug. Under load, a timeout could happen that would stop other things from running. And sure enough, as the update was being rolled out across all of Salesforce's data centers, a timeout occurred. This in turn meant that certain tasks were not carried out when the servers were restarted. And that, in turn, meant that those servers did not return to operation correctly. That left customers unable to access Salesforce's products.
[12]
And then things got even worse. The Salesforce team has tools to deal with sad servers, and use what Dieken called "our emergency break glass process" to perform rollbacks and restarts.
"In this case," he went on, "we found a circular dependency where the tool that we use to get into production had a dependency on the DNS servers being active."
It is always DNS.
We found a circular dependency where the tool that we use to get into production had a dependency on the DNS servers being active
Of course, staff did eventually get into the servers to fix them up but, as customers felt all too keenly, it took quite some time to undo the mess.
As for forthcoming actions, safeguards are to be put in place to stop manual global deployments like this in future, and the process will be automated. Dieken also acknowledged that the experience had shown up a gap in Salesforce's test coverage – the script needed to be better tested, essentially. Finally, that dependency of the recovery tools on DNS needed to be dealt with.
Customers bemused that they had to get official word of the outage from social media or this very organ, rather than the [13]status page , were doubtless more bemused at the revelation that the reason for the Salesforce status site falling over was due to auto-scale not being turned on for that web property. (During the downtime, Salesforce had to use its documentation site to explain to clients what was going wrong.)
"We over-provisioned enough capacity to make sure that we could handle large spikes," explained Dieken, "but we never foresaw that we'd have this type of load."
Not to worry, though, auto-scale is now on, so should things go south again at least the status site is unlikely to be embarrassingly absent.
And the engineer who sidestepped Salesforce's carefully crafted policies and took down the platform? "We have taken action with that particular employee," said Dieken. ®
Get our [14]Tech Resources
[1] https://www.theregister.com/2021/05/12/salesforce_outage_dns_issue/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YKWKfOhZr8N663o9ekMhkQAAAMU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YKWKfOhZr8N663o9ekMhkQAAAMU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YKWKfOhZr8N663o9ekMhkQAAAMU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://help.salesforce.com/articleView?id=000358392&type=1&mode=1
[6] https://www.theregister.com/2021/05/12/salesforce_outage_dns_issue/
[7] https://www.theregister.com/2021/05/17/outlook_incident_report/
[8] https://www.theregister.com/2021/05/11/trend_email_outage/
[9] https://www.theregister.com/2021/05/06/ovh_outlines_threepoint_hyper_resiliance/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YKWKfOhZr8N663o9ekMhkQAAAMU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://www.theregister.com/Tag/who-me
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YKWKfOhZr8N663o9ekMhkQAAAMU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[13] https://status.salesforce.com/
[14] https://whitepapers.theregister.com/
Re: "We have taken action with that particular employee"
"We have taken action with that particular ex-employee"
I doubt a cock-up of that magnitude will result in keeping your job...
Re: "We have taken action with that particular employee"
Might be worth keep them onboard - they have just learned about a scripting issue which could mean that they will never have this happen again because the tech will be damn careful next time! Replace them with a new tech and the chances are that this will happen again in a year or two.
Re: "We have taken action with that particular employee"
This is definitely a case of [1]needlessly ambiguous terms .
[1] https://www.youtube.com/watch?v=U6cake3bwnY
Re: "We have taken action with that particular employee"
Deputy Heads will roll.
wth is it with always dns?
I don't get it? Been running DNS for about 25 years now. It's super rare that a problem is DNS related in my experience. I certainly have had DNS issues over the years, most often the problems tend to be bad config, bad application(includes actual apps as well as software running on devices such as OS, storage systems network devices etc), bad user. In my experience bad application wins the vast majority of times. I have worked in SaaS-style (as in in house developed core applications) since 2000.
But I have seen many people say "it's always dns", maybe DNS related issues are much more common in windows environments? I know DNS resolution can be a pain such as dealing with various levels of browser and OS caching regarding split dns where DNS names resolve to different addresses if you are inside or outside the network/vpn). I don't classify those as DNS issues though, DNS is behaving exactly as it was configured/intended to, it was the unfortunate user who happened to do an action which performed a query whose results were then cached by possibly multiple layers in the operating system before switching networks and the cache didn't get invalidated resulting in a problem.
I know there have been some higher profile DNS related outages by some cloud providers(I think MS had one not long ago) but still seems to be a tiny minority of the causes of problems.
It makes me feel like "it's always DNS" is like the folks who try to blame the network for every little problem when it's almost never the network either(speaking as someone who manages servers, storage, networking, apps, hypervisors etc so I have good visibility into most everything except in house apps).
Re: wth is it with always dns?
I think by "it's always DNS" is that it's because DNS is so fundamental that if you screw it up, bad things often happen and these bad things often take a few hours to resolve due to caching.
If a service vendor such as Salesforce screws up one of their applications, it's crap, but it won't usually take out their entire service and the testing of such components should be readily possible to reduce the chance and quickly reversible when bad things happen. As a result even Salesforce don't screw up this kind of thing that often. However, DNS changes are considerably harder to test safely and, in particular, remotely and when something goes wrong are a PITA to fix.
bad app and DNS
Here's a great example of a bad app. Java. I first came across this probably in 2004. I just downloaded the "reccomended" release for Oracle Java on linux (from java.com) which is strangely 1.8 build 291 (thought there was java 11 or 12 or 13 now? ) anyway...
peek inside the default java.security file
(adjusted formatting of the output to make it take less lines)
# The Java-level namelookup cache policy for successful lookups:
# any negative value: caching forever - any positive value: the number of seconds to cache an address for - zero: do not cache
# default value is forever (FOREVER). For security reasons, this caching is made forever when a security manager is set. When a security manager is not set, the default behavior in this implementation is to cache for 30 seconds.
# NOTE: setting this to anything other than the default value can have serious security implications. Do not set it unless you are sure you are not exposed to DNS spoofing attack.
#networkaddress.cache.ttl=-1
I don't think I need to explain how stupid that is. It caused major pain for us back in 2004(till we found the setting), and again in 2011(4 companies later, couldn't convince the payment processor to adjust this setting at the time had to resort to rotating DNS names when we had IP changes) and well it's the same default in 2021. Not sure about newer than Java 8 what the default may be. DNS spoofing attacks are a thing of course(I believe handling them in this manor is poor), but it's also possible to be under a spoofing attack when the jvm starts up resulting in a bad dns result which never gets expired per the default settings anyway.
At the end of the day it's a bad default setting. I'm fine if someone wants to for some crazy reason put this setting in themselves, but it should not be the default and in my experience not many people know that this setting even exists and are surprised to learn about it.
But again, not a DNS problem, bad application problem.
Re: bad app and DNS
Your problem is Java. :)
OW! OW! Put down the pitchforks, I'm going!
Four-year-old
and the engineer also had a four-year-old script to do the job.
FTFY!
Having been on the receiving end of some of Salesforce's products, there's probably a fair few people who would happily buy the guy a pint for that period of blessed relief.
So...
The engineer followed a protocol and it got approved. Sure they may have used the "break/fix" option, but "Going down the EBF route meant fewer approvals" still means people higher up the chain did actually approve it.
Bad call by the engineer maybe, but someone should have spotted it if it did indeed go through a formal channel.
What probably happened...
Tech: The new Hyperforce install is done.
Management: Let's go live!
Tech: It will take a few days to slow-roll the DNS changes.
Management: We need to show billing this week! Can we do the DNS changes faster?
Tech: Yes, but it is considered an emergency procedure.
Management: Is it risky?
Tech: Not really. I've done it several times.
Management: I'm giving you verbal authority to do it now!
Tech: OK.
Management: What just happened?
Tech. Oops. Something broke.
Upper Management: What just happened?
Management: THE TECH DID IT! THE TECH DID IT! IT'S ALL THE TECH'S FAULT!
Tech: Sigh.
Why was EBF used in the first place?
There is nothing to indicate that there was an actual emergency. Someone wanted to cut corners to speed things up and someone, presumably a manager, decided EBF was appropriate sans an emergency. Several people are at fault here.
Shiney Happy People
The Salesforce team has tools to deal with sad servers.
"We have taken action with that particular employee"
Presumably involving a a shovel, a roll of carpet and two 20kg bags of quicklime.