Websites and apps are suffering or have suffered outages around the world for at least some netizens today due to connectivity issues.

Though the exact causes of the IT breakdowns are in many cases not fully known right now, there has been a sudden uptick in downtime right as Let's Encrypt, which provides free HTTPS certificates to a ton of organizations, let one of its root and intermediate certs expire.

This expiration should be invisible to software and services relying on the certificates for encryption, tamper-proof communications and whatnot, however not all systems appear to have handled the expiry well. Thus, it is assumed the expiration and at least some of today's outages are interlinked. Other downtime, such as Slack's teetering, is not tied to Let's Encrypt.

[1]

Specifically, today, September 30, an older root certificate – DST Root CA X3 – which was used to underpin HTTPS certs issued by Let's Encrypt, [2]expired as planned along with its R3 intermediate. There's more technical info on that [3]here .

[4]

[5]

Apps and websites should instead now use, or continue to use, Let's Encrypt's ISRG Root X1 certificate and its replacement R3 intermediate automatically, and this whole process should pretty much be transparent to users. Let's Encrypt warned "older devices" will need [6]a prod to catch up with and trust the latest foundational certs.

However, it appears it's more than "older devices" that have tripped up and stopped working as expected – such as refusing to establish secure connections to remote systems – and that may be due to [7]an issue with the now-expired R3 intermediate, or because your client doesn't recognize the ISRG root cert.

[8]

If you're having trouble with Git and servers using broken Let's Encrypt certs, for instance, there's a thread on fixing that [9]here .

[10]Let's Encrypt completes huge upgrade, can now rip and replace 200 million security certs in 'worst case scenario'

[11]Let's Encrypt warns about a third of Android devices will from next year stumble over sites that use its certs

[12]Lettuce Encrypt, Encrypt We Must: Hobby projects change name after Let's Encrypt fires off trademark complaints

HTTPS cert expert Scott Helme has been tracking the certificate expiry and reporting what's falling over on Twitter, likely as a result of the cert changes, [13]right here .

These include [14]Catchpoint System Services ; [15]OVH ; [16]OpenBSD ; [17]Facebook for developers; [18]Fortigate ; parts of [19]Netify ; and [20]Cisco Umbrella .

[21]Heroku also suffered a wobble with its metrics and Connect Dashboard though it was unrelated to the cert death, which was a separate issue for its customers. [22]Shopify and parts of [23]Cloudflare also encountered mysterious downtime that appears to be unrelated to Let's Encrypt.

"There are also many reports of iOS and macOS versions newer than expected seeing issues on sites serving the expired R3 intermediate," Helme said. "I've seen errors on iOS 11, 13 and 14 along with several macOS version only a few minor releases behind current."

[24]

A spokesperson for the Let's Encrypt organization told us earlier today: "We are monitoring the expiration and providing advice when we can. The internet is a complex system and any change has knock-on effects. We are not currently seeing more than what we might describe as 'expected' issues for any change like this."

Meanwhile, we've spotted that accounting and invoicing software outfit [25]Xero has been down for hours right as people try to get their figures in by the end of the month, and [26]Slack has had connectivity issues. It's not clear why Xero fell over; it doesn't appear to use Let's Encrypt and we've asked for an explanation.

For Slack, we're told it was a DNSSEC issue rather than a certificate problem. Essentially, as described [27]here , Slack made a change to its DNS records, backed out the update, this adjustment is still propagating through the internet, and as a result, looking up the site's servers may fail for some people.

"This issue was caused by our own change and not related to any third-party DNS software and services," the chat app biz noted. In order to resolve this faster, your ISP will need to flush their DNS record for slack.com."

"Less than one per cent of users" were affected, it added. We note that Slack has 12 million or more daily active users, so you do the math.

We'll keep you posted on any updates. If your app or site isn't working, consult its support desk, mailing lists, or forums for any upgrades to apply or commands to run, or wait for code to automatically pick up today's changes. There's a long [28]help thread here on the Let's Encrypt community board. ®

Get our [29]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YVYzsS@6JdVNWur3xNCrKgAAABM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

[3] https://docs.certifytheweb.com/docs/kb/kb-202109-letsencrypt/

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YVYzsS@6JdVNWur3xNCrKgAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YVYzsS@6JdVNWur3xNCrKgAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://community.letsencrypt.org/t/production-chain-changes/150739

[7] https://community.letsencrypt.org/t/r3-intermediate-certificate-has-expired-it-issued-certs-past-its-expiration-date/160797/94

[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YVYzsS@6JdVNWur3xNCrKgAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[9] https://stackoverflow.com/questions/69387175/git-for-windows-ssl-certificate-problem-certificate-has-expired

[10] https://www.theregister.com/2021/02/15/in_brief_security/

[11] https://www.theregister.com/2020/11/06/android_encryption_certs/

[12] https://www.theregister.com/2020/06/10/lets_encrypt_trademark_complaints/

[13] https://twitter.com/Scott_Helme/status/1443336755969765385

[14] https://status.catchpoint.com/incidents/f5yl89kygf12

[15] http://travaux.ovh.net/?do=details&id=53588

[16] https://twitter.com/Scott_Helme/status/1443671104183283722

[17] https://twitter.com/spazef0rze/status/1443656911950385155

[18] https://old.reddit.com/r/fortinet/comments/pylie2/all_of_a_sudden_we_are_getting_ssl_certificate/

[19] https://www.netlifystatus.com/incidents/l6rk7f1hqfd7

[20] https://status.umbrella.com/#/detail/582

[21] https://status.heroku.com/incidents/2365

[22] https://www.shopifystatus.com/incidents

[23] https://www.cloudflarestatus.com/incidents/rph158j6tsfk

[24] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YVYzsS@6JdVNWur3xNCrKgAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[25] https://status.xero.com/incidents/d24f1j2kxq5v

[26] https://status.slack.com/2021-09/06c1e17de93e7dc2

[27] https://lists.dns-oarc.net/pipermail/dns-operations/2021-September/021340.html

[28] https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190

[29] https://whitepapers.theregister.com/