cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
9743
Views
2
Helpful
13
Replies

CSCwf28118 - vEdge: Certificate issue on Viptela devices

mdavis300
Level 1
Level 1

This is a Cisco global outage and won't open customer TAC cases it one. We already have 83 remote sites effected. Where will Cisco post incident updates and instructions for the final fix? Extending timers is just band aid.

13 Replies 13

jonathan aquino
Level 1
Level 1

We currently have this problem as well, cisco TAC did not provide any solution and they are still checking for the solution. Please see below guide/mitigation steps from Cisco TAC.
Identify vEdge Certificate Expired on May 9th 2023 - Cisco

Is rewinding the clock of all devices effective as a countermeasure?

Hi,

I just tried this on a test 100b device on a test overlay but unfortunately it didn't work. 

Before change:

 

mydevice# sh control local                                                                                                                              
personality                       vedge                                                                                                                        
root-ca-chain-status              Installed                                                                                                                    
certificate-status                Installed                                                                                                                    
certificate-validity              Not Valid -  certificate has expired

 

After change:

 

mydevice# clock set date 2023-05-01 time 08:35:00.000

my device# sh control local                                                                                                                              
personality                       vedge                                                                                                                        
root-ca-chain-status              Installed                                                                                                                    
certificate-status                Installed                                                                                                                    
certificate-validity              Valid

 

Which looked good but then when I cleared the control connections it didn't come back. I haven't checked the vbond logs yet but I guess it's because the cert it's presenting to vbond is still expired.

I also tried setting the date back on one of the vbonds but the vedge still didn't recover. Device bringup logs show:
Event Name : vbond-reject-vedge-connection
reason=ERR_BID_NOT_VERIFIED

Dave

According to TAC, this would work but you'd have to change the clocks on all your routers and all controllers. For those of us with hosted controller instances, it's not an option as the time is synced with the VM host.

Thanks for the confirmation, makes sense, we're hosted too.

anandpaleja1
Level 1
Level 1

Dear Cisco team,

Kindly fix this issue on a priority basis. Our work is affected due to this issue.

19 Vedge locations are impacted.

@anandpaleja1 most of the people here are Cisco customers just like yourself. 

To raise with Cisco directly you should talk to your Cisco account team or TAC

Eis3CiL8
Level 1
Level 1

Affected here as well. A number of locations still have tunnels up but control connections and OMP peers are down. We do "track-omp" for VRRP so these sites lost their default gateway. Logging in via SSH and removing "track-omp" from the VRRP configuration fixed the connectivity for now. But of course it's only a matter of time before the IPSec sessions are renegotiated and the sites go offline.

Fingers crossed that Cisco will have a fix before then. 

Dave Lewis
Level 1
Level 1

Interestingly we've got some control connections that have been up less than 24 hours (i.e. since the certificate expired). I'd have expected those to have failed. Wondering if Cisco disabled cert validation or something temporarily. I also couldn't get an answer from TAC on whether the control connections have some kind of timer or periodic rekey as we have some up for over 57 hours.  

 

 

DrewonIT
Level 1
Level 1

We are experiencing this as well. We have two DataCenters connected using 1000's that I'm very concerned about. While we are not at this point, we may need to notify the FDIC of the potential operational interruption. 

venom43212
Level 4
Level 4

Software updates are now available. In our case, it's 20.6.3.2.

Identify vEdge Certificate Expired on May 9th 2023 - Cisco

venom43212
Level 4
Level 4

Software updates are now available. In our case, it's 20.6.3.2.

Identify vEdge Certificate Expired on May 9th 2023 - Cisco

kendodso
Cisco Employee
Cisco Employee