Problem with Edge onboarding using Automated Certificate Authorization

[Hetch]

Hi,

I 'm using Cisco Catalyst SD-WAN version 20.15.1 (on prem orchestrators). I have a problem onboarding any virtual edge C8000V router when I set the WAN Edge Cloud Certificate Authorization to automated (Manager Signed), but I do not have the same problem if the WAN Edge Cloud Certificate Authorization is set to Manual (Enterprise CA - recommended).

When I try to onboard, a C8000V router using the automated (Manager Signed) authorization process, it fails. logs on the vBond shows certificate verification failure with code "ERR_CERT_VER_FAIL"

------------------

vBond:~$ cat /var/log/vsyslog

Feb 9 10:30:19 vBond VBOND_4[1626]: %Viptela-vBond-vbond_4-5-NTCE-1400002: Notification: vbond-reject-vedge-connection severity-level:major host-name:"vBond" system-ip:10.3.3.2 uuid:"C8K-9C82B27D-FC58-6716-B992-3A616D67FE6E" organization-name:"NCRATLEOS-NOC-LAB" sp-organization-name:"NCRATLEOS-NOC-LAB" reason:"ERR_CERT_VER_FAIL" generated-at:2-9-2025T8:30:19
Feb 9 10:30:19 vBond VBOND_4[1626]: %Viptela-vBond-vbond_4-5-NTCE-1400002: Notification: control-connection-auth-fail severity-level:major host-name:"vBond" system-ip:10.3.3.2 personality:vbond peer-type:vedge peer-system-ip::: local-system-ip:10.3.3.2 local-color:default reason:"ERR_CERT_VER_FAIL" generated-at:2-9-2025T8:30:19
Feb 9 10:30:21 vBond VBOND_0[1635]: %Viptela-vBond-vbond_0-5-NTCE-1400002: Notification: vbond-reject-vedge-connection severity-level:major host-name:"vBond" system-ip:10.3.3.2 uuid:"C8K-BC9FCD48-8689-6C4D-A509-43608B131407" organization-name:"NCRATLEOS-NOC-LAB" sp-organization-name:"NCRATLEOS-NOC-LAB" reason:"ERR_CERT_VER_FAIL" generated-at:2-9-2025T8:30:21
Feb 9 10:30:21 vBond VBOND_0[1635]: %Viptela-vBond-vbond_0-5-NTCE-1400002: Notification: control-connection-auth-fail severity-level:major host-name:"vBond" system-ip:10.3.3.2 personality:vbond peer-type:vedge peer-system-ip::: local-system-ip:10.3.3.2 local-color:default reason:"ERR_CERT_VER_FAIL" generated-at:2-9-2025T8:30:21
Feb 9 10:30:23 vBond VBOND_4[1626]: %Viptela-vBond-vbond_4-5-NTCE-1400002: Notification: vbond-reject-vedge-connection severity-level:major host-name:"vBond" system-ip:10.3.3.2 uuid:"ASR-6332d2ad-329a-4b26-a0f4-175965c51b78" organization-name:"NCR-NOC-CAIRO-LAB" sp-organization-name:"NCR-NOC-CAIRO-LAB" reason:"ERR_CERT_VER_FAIL" generated-at:2-9-2025T8:30:23
Feb 9 10:30:23 vBond VBOND_4[1626]: %Viptela-vBond-vbond_4-5-NTCE-1400002: Notification: control-connection-auth-fail severity-level:major host-name:"vBond" system-ip:10.3.3.2 personality:vbond peer-type:vedge peer-system-ip::: local-system-ip:10.3.3.2 local-color:default reason:"ERR_CERT_VER_FAIL" generated-at:2-9-2025T8:30:23

-------------------------

Root CA is installed on the router, and I have compared it with the Root CA installed on vManage and vBond. it looks the same (same serial num., same issuer and Organization Name. also, NTP is synchronized, and DNS server is installed properly. connectivity is Ok.

This is the output of the "show sdwan control connection-history "

-------------------

PEER PEER
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC
TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE
-----------------------------------------------------------------------------------------------------------------------------------------
vmanage dtls 10.3.3.1 11 0 192.168.74.11 12946 192.168.74.11 12946 public-internet tear_down 0
vbond dtls 0.0.0.0 0 0 192.168.74.12 12346 192.168.74.12 12346 public-internet tear_down 0
vbond dtls 0.0.0.0 0 0 192.168.74.12 12346 192.168.74.12 12346 public-internet tear_down 0
vbond dtls 0.0.0.0 0 0 192.168.74.12 12346 192.168.74.12 12346 public-internet connect 0

-----------------------

and I see only the RootCA installed on the cEdge but not the device certificate:

-------------

cEdge61#show sdwan control local-properties | include chassis-num|serial-num
chassis-num/unique-id C8K-D175064E-AB76-2C6D-4FAA-024A075D8BA2
serial-num No certificate installed
subject-serial-num N/A
enterprise-serial-num No certificate installed

--------------------

Any idea why only Automated authorization fails, but not the manual authorization? on the vManage GUI, I see CSR generated then certificate installation failed. does this mean the vManage has some issues with the CSR signing process?

Thanks in advance, 

[Jeongjun Park]

Hi, Please refer to this article.

https://community.cisco.com/t5/sd-wan-and-cloud-networking/cedge-stuck-in-state-quot-connect-quot-dconfail/td-p/4083314

[Kanan Huseynli]

Hi,

did you upload serial number list to vManage and push to all other controller?

Virtual devices (vedge-cloud/ csr1kv /cat8kv) don't have any built in serial/token information for onboarding. Below is steps for onboarding virtual router:

1) In PNP portal, you add virtual device (type/quantity). It generates, chassis-id and token

2) You upload&refresh serial list on vManage by uploading (or re-uploading) serial file downloaded from PNP portal. You also push updated list to other controllers (vbond/vsmart)

3) In vManage router list you will see chassis-id and token information (serial number and token are in the same column)

4) You take any of them (chassis-id+token matching device type) and enable it on virtual router

request platform software sd-wan vedge-cloud activation chassis-id [id] token [token] and hit enter.

5) Router goes to the vBond using chassis-id and token. They authorize each other. vBond provided vManage information. vManages auhtorizes router using chassis-id and token from the router. Only then vManage generates CSR and signs certificate for router. Now, you have the same chassis-id but Serial number (SN) of the signed vManage based certificate instead of token (it is visible in vManage gui). Router tears down control connections and re-build using chassis-id and Serial Number beginning from the vBond again. After bidirectional authentication router gets vManage/vSmart information, gets its template and localized policy from the vManage and routing and centralized policy from the vSmart.

See:

https://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/SDWAN/sdwan-wan-edge-onboarding-deploy-guide-2020nov.pdf

Procedure 2: Additional onboarding steps for vEdge Cloud platform

[Hetch · Accepted solution]

Hi @Jeongjun Park and @Kanan Huseynli Thanks for your reply.

I used also same NTP time.google.com for synchronization, and also uploaded the serial list to the controllers, but thanks a lot for the details of the activation step.

things is the automated authorization process was working normally until only few days, and I figured out that something was wrong with the ROOTCA (I cannot determine exactly what was the problem, but it is now solved after I installed a fresh vManage and installed the ROOTCA again on the vManage). now I see the automated authorization process is working.

one question @Kanan Huseynli regarding the automated authorization process. my understanding that the vManage uses the enterprise root certificate to sign the CSR request of the Edge, is that correct? and in order to sign the CSR it needs the private key. how does it obtain it?

Thanks a lot,

[Kanan Huseynli]

In automatic option vManage itself is the CA. It is totally different from Enterprise CA

[Jeongjun Park]

Good to hear !

I think this rarely happens.

[Hetch]

Thanks again @Jeongjun Park and @Kanan Huseynli. much appreciated!