Don't turn it off and on again: Expired Cisco cert cripples vEdge SD-WAN kit
(2023/05/10)
- Reference: 1683749405
- News link: https://www.theregister.co.uk/2023/05/10/cisco_sd_wan_certificate_expired/
- Source link:
An expired security certificate is threatening to wreak havoc with Cisco customers' wide-area networks. For a change, turning the equipment off and back on again will only make things worse.
In a bulletin [1]published this week, Cisco warned that customers using vEdge SD-WAN appliances could experience complete loss of service if their device is reloaded, updated, or if new templates are pushed.
The culprit: a cryptographic certificate, affecting the SD-WAN appliance's control plane, expired Tuesday, May 9. “If left unaddressed, this could impact data plane connections and result in SD-WAN downtime,” the Cisco bulletin reads.
[2]
It's understood this hardware-level certificate is stored in the devices' TPM. And bear in mind, even if you don't manually restart or update your equipment, there are timers in the devices that will, by default, start a reload that will trigger disruption as a result of the now-dead cert.
'Time bomb'
This surprise expiry could have wide sweeping implications for enterprises that rely on Cisco’s Viptela SD-WAN products for communication between their satellite offices, headquarters, and datacenters. While the scope of the snafu isn't clear, plenty of netizens have [3]reported outages as a result of the cert expiry.
"All vEdge based SD-WAN customers are sitting on a time bomb, watching the clock with sweaty palms, waiting for their companies' WAN to implode and/or figuring out how to re-architect their WAN to maintain connectivity," as one put it.
[4]
[5]
In addition to service disruptions, Cisco said organizations could experience other failures, including:
Loss of connections to vSmart and/or vManage
Port-hopping in some way impacted
Control policy changes affected, including topology changes
Interface flapping
As of publication, it appears Cisco has released a patch resolving the issue. Posting to [6]Twitter Wednesday morning, Danial Dib, a senior network architect at Cisco, shared a (gated) link to a software update to address the disruption, and said additional updates would be rolling out soon:
Fixed software starting to be released now. For example 20.6.5.2 -> [7]https://t.co/9efUYlAnqg 20.3.7.1 is another one. Others coming soon as well. [8]https://t.co/3H7oJuAN9w — Daniel Dib (@danieldibswe) [9]May 10, 2023
Based on the documentation, the patch likely amounts to certificate replacement. Unfortunately it doesn’t appear that the update will do much good for devices that have already been rendered inoperable by the expired certs. Cisco recommends customers with bricked gateways contact Cisco for assistance.
[10]Cisco: Don't use 'blind spot' – and do use 'feed two birds with one scone'
[11]Dump these insecure phone adapters because we're not fixing them, says Cisco
[12]Russian snoops just love invading unpatched Cisco gear, America and UK warn
[13]Cisco Moscow trashed offices as it quit Putin's putrid pariah state
The Register has reached out to our contacts at Cisco for comment on how the certificate was allowed to lapse, and what the IT giant is doing to help folks hit by the blunder. The networking goliath declined to comment further.
This isn’t the first time this has happened. As we reported back in 2018, a very similar issue [14]took out Cisco VPNs for customers using the manufacturer's delightfully named Application Policy Infrastructure Controller Enterprise Module (APIC-EM).
That SDN controller relied on an SSL certificate that Cisco neglected to renew, causing all manner of headaches for network administrators trying to provision connections to branch offices and hubs.
[15]
While you might think companies would keep tabs on when certificates are set to expire as to avoid these kinds of costly, not mention confidence shaking, mishaps, they aren't uncommon. A dive into El Reg 's archives reveals plenty of examples, including several that [16]borked features in Microsoft Windows. So, at least Cisco has company. ®
Get our [17]Tech Resources
[1] https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/220448-identify-vedge-certificate-expired-on-ma.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZFwUBnnz7GsdywPu4yfohAAAAEg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.reddit.com/r/Cisco/comments/13dm1ea/vedgeviptela_based_sdwan_global_outage_impacting/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFwUBnnz7GsdywPu4yfohAAAAEg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZFwUBnnz7GsdywPu4yfohAAAAEg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://twitter.com/danieldibswe/status/1656303776989433856
[7] https://t.co/9efUYlAnqg
[8] https://t.co/3H7oJuAN9w
[9] https://twitter.com/danieldibswe/status/1656303776989433856?ref_src=twsrc%5Etfw
[10] https://www.theregister.com/2023/05/10/cisco_inclusive_language/
[11] https://www.theregister.com/2023/05/05/cisco_phone_adapter_vulnerabilitty/
[12] https://www.theregister.com/2023/04/18/uk_us_apt28_cisco_routers/
[13] https://www.theregister.com/2023/04/06/cisco_destroyed_spares_in_russia/
[14] https://www.theregister.com/2018/08/07/cisco_vpn_certificate_expiry/
[15] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFwUBnnz7GsdywPu4yfohAAAAEg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[16] https://www.theregister.com/2021/11/05/windows_insider_cert_fix/
[17] https://whitepapers.theregister.com/
In a bulletin [1]published this week, Cisco warned that customers using vEdge SD-WAN appliances could experience complete loss of service if their device is reloaded, updated, or if new templates are pushed.
The culprit: a cryptographic certificate, affecting the SD-WAN appliance's control plane, expired Tuesday, May 9. “If left unaddressed, this could impact data plane connections and result in SD-WAN downtime,” the Cisco bulletin reads.
[2]
It's understood this hardware-level certificate is stored in the devices' TPM. And bear in mind, even if you don't manually restart or update your equipment, there are timers in the devices that will, by default, start a reload that will trigger disruption as a result of the now-dead cert.
'Time bomb'
This surprise expiry could have wide sweeping implications for enterprises that rely on Cisco’s Viptela SD-WAN products for communication between their satellite offices, headquarters, and datacenters. While the scope of the snafu isn't clear, plenty of netizens have [3]reported outages as a result of the cert expiry.
"All vEdge based SD-WAN customers are sitting on a time bomb, watching the clock with sweaty palms, waiting for their companies' WAN to implode and/or figuring out how to re-architect their WAN to maintain connectivity," as one put it.
[4]
[5]
In addition to service disruptions, Cisco said organizations could experience other failures, including:
Loss of connections to vSmart and/or vManage
Port-hopping in some way impacted
Control policy changes affected, including topology changes
Interface flapping
As of publication, it appears Cisco has released a patch resolving the issue. Posting to [6]Twitter Wednesday morning, Danial Dib, a senior network architect at Cisco, shared a (gated) link to a software update to address the disruption, and said additional updates would be rolling out soon:
Fixed software starting to be released now. For example 20.6.5.2 -> [7]https://t.co/9efUYlAnqg 20.3.7.1 is another one. Others coming soon as well. [8]https://t.co/3H7oJuAN9w — Daniel Dib (@danieldibswe) [9]May 10, 2023
Based on the documentation, the patch likely amounts to certificate replacement. Unfortunately it doesn’t appear that the update will do much good for devices that have already been rendered inoperable by the expired certs. Cisco recommends customers with bricked gateways contact Cisco for assistance.
[10]Cisco: Don't use 'blind spot' – and do use 'feed two birds with one scone'
[11]Dump these insecure phone adapters because we're not fixing them, says Cisco
[12]Russian snoops just love invading unpatched Cisco gear, America and UK warn
[13]Cisco Moscow trashed offices as it quit Putin's putrid pariah state
The Register has reached out to our contacts at Cisco for comment on how the certificate was allowed to lapse, and what the IT giant is doing to help folks hit by the blunder. The networking goliath declined to comment further.
This isn’t the first time this has happened. As we reported back in 2018, a very similar issue [14]took out Cisco VPNs for customers using the manufacturer's delightfully named Application Policy Infrastructure Controller Enterprise Module (APIC-EM).
That SDN controller relied on an SSL certificate that Cisco neglected to renew, causing all manner of headaches for network administrators trying to provision connections to branch offices and hubs.
[15]
While you might think companies would keep tabs on when certificates are set to expire as to avoid these kinds of costly, not mention confidence shaking, mishaps, they aren't uncommon. A dive into El Reg 's archives reveals plenty of examples, including several that [16]borked features in Microsoft Windows. So, at least Cisco has company. ®
Get our [17]Tech Resources
[1] https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/220448-identify-vedge-certificate-expired-on-ma.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZFwUBnnz7GsdywPu4yfohAAAAEg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.reddit.com/r/Cisco/comments/13dm1ea/vedgeviptela_based_sdwan_global_outage_impacting/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFwUBnnz7GsdywPu4yfohAAAAEg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZFwUBnnz7GsdywPu4yfohAAAAEg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://twitter.com/danieldibswe/status/1656303776989433856
[7] https://t.co/9efUYlAnqg
[8] https://t.co/3H7oJuAN9w
[9] https://twitter.com/danieldibswe/status/1656303776989433856?ref_src=twsrc%5Etfw
[10] https://www.theregister.com/2023/05/10/cisco_inclusive_language/
[11] https://www.theregister.com/2023/05/05/cisco_phone_adapter_vulnerabilitty/
[12] https://www.theregister.com/2023/04/18/uk_us_apt28_cisco_routers/
[13] https://www.theregister.com/2023/04/06/cisco_destroyed_spares_in_russia/
[14] https://www.theregister.com/2018/08/07/cisco_vpn_certificate_expiry/
[15] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZFwUBnnz7GsdywPu4yfohAAAAEg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[16] https://www.theregister.com/2021/11/05/windows_insider_cert_fix/
[17] https://whitepapers.theregister.com/
claimed
If you’re a device or software producer, please start rejecting certificates 3 months before expiry, then include a bypass which will get you to ACTUAL expiry with a config setting (which can’t be put in early, so disable this on startup or something).
That should give you and your customers some recourse in these events. Yes, it means writing a couple of extra lines of code, and there might be better ways, but this way is better than nothing!
Paying attention
Nice of them to notice AFTER the horse has bolted from the stable. Even when they're both the horse AND the stable.