CSCwf28118 - vEdge: Certificate issue on Viptela devices

[mdavis300]

This is a Cisco global outage and won't open customer TAC cases it one. We already have 83 remote sites effected. Where will Cisco post incident updates and instructions for the final fix? Extending timers is just band aid.

[jonathan aquino]

We currently have this problem as well, cisco TAC did not provide any solution and they are still checking for the solution. Please see below guide/mitigation steps from Cisco TAC.
Identify vEdge Certificate Expired on May 9th 2023 - Cisco

[naka@nissho-ele.co.jp]

Is rewinding the clock of all devices effective as a countermeasure?

[Dave Lewis]

Hi,

I just tried this on a test 100b device on a test overlay but unfortunately it didn't work. 

Before change:

mydevice# sh control local                                                                                                                              
personality                       vedge                                                                                                                        
root-ca-chain-status              Installed                                                                                                                    
certificate-status                Installed                                                                                                                    
certificate-validity              Not Valid -  certificate has expired

After change:

mydevice# clock set date 2023-05-01 time 08:35:00.000

my device# sh control local                                                                                                                              
personality                       vedge                                                                                                                        
root-ca-chain-status              Installed                                                                                                                    
certificate-status                Installed                                                                                                                    
certificate-validity              Valid

Which looked good but then when I cleared the control connections it didn't come back. I haven't checked the vbond logs yet but I guess it's because the cert it's presenting to vbond is still expired.

I also tried setting the date back on one of the vbonds but the vedge still didn't recover. Device bringup logs show:
Event Name : vbond-reject-vedge-connection
reason=ERR_BID_NOT_VERIFIED

Dave

[infcommssupport1]

According to TAC, this would work but you'd have to change the clocks on all your routers and all controllers. For those of us with hosted controller instances, it's not an option as the time is synced with the VM host.

[Dave Lewis]

Thanks for the confirmation, makes sense, we're hosted too.

[anandpaleja1]

Dear Cisco team,

Kindly fix this issue on a priority basis. Our work is affected due to this issue.

19 Vedge locations are impacted.

[Rich R]

@anandpaleja1 most of the people here are Cisco customers just like yourself. 

To raise with Cisco directly you should talk to your Cisco account team or TAC

[Eis3CiL8]

Affected here as well. A number of locations still have tunnels up but control connections and OMP peers are down. We do "track-omp" for VRRP so these sites lost their default gateway. Logging in via SSH and removing "track-omp" from the VRRP configuration fixed the connectivity for now. But of course it's only a matter of time before the IPSec sessions are renegotiated and the sites go offline.

Fingers crossed that Cisco will have a fix before then. 

[Dave Lewis]

Interestingly we've got some control connections that have been up less than 24 hours (i.e. since the certificate expired). I'd have expected those to have failed. Wondering if Cisco disabled cert validation or something temporarily. I also couldn't get an answer from TAC on whether the control connections have some kind of timer or periodic rekey as we have some up for over 57 hours.  

[DrewonIT]

We are experiencing this as well. We have two DataCenters connected using 1000's that I'm very concerned about. While we are not at this point, we may need to notify the FDIC of the potential operational interruption. 

[venom43212]

Software updates are now available. In our case, it's 20.6.3.2.

Identify vEdge Certificate Expired on May 9th 2023 - Cisco

[venom43212]

Software updates are now available. In our case, it's 20.6.3.2.

Identify vEdge Certificate Expired on May 9th 2023 - Cisco

[kendodso]

Please view this document: https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/220448-identify-vedge-certificate-expired-on-ma.html#toc-hId--813846323

It is being updated with all the latest information.