Salesforce fell over so hard today, it took out its own server status page
(2021/05/12)
- Reference: 1620785404
- News link: https://www.theregister.co.uk/2021/05/12/salesforce_outage_dns_issue/
- Source link:
Salesforce is digging itself out of a multi-hour outage right now that it has blamed on a DNS issue.
At one point today, the IT breakdown was so severe that its [1]status page was pretty much inaccessible for netizens, and staff resorted to posting updates on their [2]help and training sub-site.
"Salesforce is experiencing a major disruption due to what we believe is a DNS issue causing our service to be inaccessible," CTO Parker Harris said in a statement. "We recognize the significant impact on our customers and are actively working on resolution.
[3]
[4]
[5]
"We believe we have isolated the issue and are in the process of bringing affected services back online. Customers may continue to experience issues as we work through remediation. This remains a top priority for Salesforce."
The downtime started shortly before 2200 UTC (1500 PT), knocked pretty much the entire software-as-a-service giant offline, and right now, everything is not yet back to normal. Though products have been restored, we're told, customers can't login if they are using multi-factor authentication.
[6]Microsoft says Outlook hit by 'email visibility issues' – as in, they're blank
[7]Trend Micro hosted email service is down, inboxes still stuck in cloudy limbo
[8]British bank TSB says it will fix days-long transaction troubles tonight
[9]Imagine your data center backup generator kicks in during power outage ... and catches fire. Well, it happened
"At 2146 UTC on May 11, 2021, the Salesforce technology team became aware of an issue impacting multiple Salesforce services," the CRM goliath [10]noted on its status page.
"Customers will experience issues while navigating the Core application, Marketing Cloud, Commerce Cloud, and Experience Cloud (formerly known as Communities).
"The issue is also impacting the Salesforce Trust site as the status.salesforce.com page is only intermittently accessible."
[11]
About half an hour ago, the biz added:
The technology team continues to manually restore service. We currently don’t have an estimated time on when full recovery is expected. The team is confident that this incident was the result of an internal DNS change.
The Government Cloud environment is now out of impact. Marketing Cloud, Commerce Cloud, and Heroku services have been restored, however, customers using multifactor authentication will still be unable to log in.
Just as we were going to press, Parker [12]tweeted : "Services are significantly restored though we are still not out of impact." ®
Get our [13]Tech Resources
[1] https://status.salesforce.com/
[2] https://help.salesforce.com/home
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YJtS1@T8WQlz7@X3lFZHqwAAAII&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YJtS1@T8WQlz7@X3lFZHqwAAAII&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YJtS1@T8WQlz7@X3lFZHqwAAAII&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] http://www.theregister.com/2021/05/12/outlook_outage_patch/
[7] http://www.theregister.com/2021/05/11/trend_email_outage/
[8] http://www.theregister.com/2021/05/07/tsb_online_and_app_banking/
[9] http://www.theregister.com/2021/04/06/webnx_data_fire/
[10] https://status.salesforce.com/generalmessages/697
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YJtS1@T8WQlz7@X3lFZHqwAAAII&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://twitter.com/parkerharris/status/1392295433582546945
[13] https://whitepapers.theregister.com/
At one point today, the IT breakdown was so severe that its [1]status page was pretty much inaccessible for netizens, and staff resorted to posting updates on their [2]help and training sub-site.
"Salesforce is experiencing a major disruption due to what we believe is a DNS issue causing our service to be inaccessible," CTO Parker Harris said in a statement. "We recognize the significant impact on our customers and are actively working on resolution.
[3]
[4]
[5]
"We believe we have isolated the issue and are in the process of bringing affected services back online. Customers may continue to experience issues as we work through remediation. This remains a top priority for Salesforce."
The downtime started shortly before 2200 UTC (1500 PT), knocked pretty much the entire software-as-a-service giant offline, and right now, everything is not yet back to normal. Though products have been restored, we're told, customers can't login if they are using multi-factor authentication.
[6]Microsoft says Outlook hit by 'email visibility issues' – as in, they're blank
[7]Trend Micro hosted email service is down, inboxes still stuck in cloudy limbo
[8]British bank TSB says it will fix days-long transaction troubles tonight
[9]Imagine your data center backup generator kicks in during power outage ... and catches fire. Well, it happened
"At 2146 UTC on May 11, 2021, the Salesforce technology team became aware of an issue impacting multiple Salesforce services," the CRM goliath [10]noted on its status page.
"Customers will experience issues while navigating the Core application, Marketing Cloud, Commerce Cloud, and Experience Cloud (formerly known as Communities).
"The issue is also impacting the Salesforce Trust site as the status.salesforce.com page is only intermittently accessible."
[11]
About half an hour ago, the biz added:
The technology team continues to manually restore service. We currently don’t have an estimated time on when full recovery is expected. The team is confident that this incident was the result of an internal DNS change.
The Government Cloud environment is now out of impact. Marketing Cloud, Commerce Cloud, and Heroku services have been restored, however, customers using multifactor authentication will still be unable to log in.
Just as we were going to press, Parker [12]tweeted : "Services are significantly restored though we are still not out of impact." ®
Get our [13]Tech Resources
[1] https://status.salesforce.com/
[2] https://help.salesforce.com/home
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YJtS1@T8WQlz7@X3lFZHqwAAAII&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YJtS1@T8WQlz7@X3lFZHqwAAAII&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YJtS1@T8WQlz7@X3lFZHqwAAAII&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] http://www.theregister.com/2021/05/12/outlook_outage_patch/
[7] http://www.theregister.com/2021/05/11/trend_email_outage/
[8] http://www.theregister.com/2021/05/07/tsb_online_and_app_banking/
[9] http://www.theregister.com/2021/04/06/webnx_data_fire/
[10] https://status.salesforce.com/generalmessages/697
[11] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/saas&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YJtS1@T8WQlz7@X3lFZHqwAAAII&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[12] https://twitter.com/parkerharris/status/1392295433582546945
[13] https://whitepapers.theregister.com/
It was the engineer in the DC with the keyboard
On one of the webcasts they said that it was an engineer deploying a emergency DNS fix across the network. The 'fix' took down most of their datacenters.
Scary how a emergency fix is rolled out globally in one go...