Unable to onboard C8000V into SD-WAN fabric - ERR_BID_NOT_VERIFIED

[MikeKoll]

Hi All, recently I have been trying to setup a Cisco SD-WAN lab on my VMware ESXi server. I have a very basic topology consisting of vManage, vSmart, vBond and few C8000V in controller mode acting as edges. So far I have configured all the controllers and local CA (using OpenSSL on Linux VM) without any issues, all 3 controllers are up and running and they show up in the vManage dashboard. However so far I have been unable to onboard the C8000V edge routers. I performed the initial configuration on all of them, installed root CA cert, imported WAN Edge list into the vManage and started the bootstrapping process using request vedgecloud activate chassis <C8K....> token <...> command on catalysts. At this point the catalysts show up as reachable in the vManage dashboard and based on the show sdwan control connections command run on catalysts, they form a connection to vBond and vManage (not vSmart). Since I'm using local enterprise CA I generate CSRs on vManage for every catalyst, sign it on my CA and then I import it again to vManage. However the moment vManage installs the certificates on catalysts, they all become unreachable for both vManage and vBond with the errors being "ERR_BID_NOT_VERIFIED" on vManage and "BIDNTVRFD" on vBond (shown in show orchestrator connections-history command). After searching the internet for ways to fix these errors I have so far performed these steps:

- I verified... that the organization name is the same on every edge and controller and that it is the same OU as in the root certificate I used to sign certificates. 

- Every edge and controller has correctly installed same root certificate.

- Every edge and controller has valid certificates.

- The chassis numbers and serial numbers of catalysts are present on vBond and that they are correct.

- Catalysts are set to valid state on vManage under device settings.

- Time is synchronized for every device in the topology.

- There is L3 connectivity between catalysts and controllers and that all tunnel interfaces have all services allowed.

Controllers version: 20.9.2

C8000V version: 17.10.1a

Unfortunately I don't know what else I could try/where else I could look, so my last option is to ask experts on the subject and I feel like this is the best place to do it. If more information is required I will do my best to provide it.

Every suggestion is wholeheartedly appreciated.

[Lei Tian]

Does it work with the "Automated" option for "WAN Edge Cloud Certificate Authorization"?

HTH,
Lei Tian

[MikeKoll]

Hello, no unfortunately it does not, I tried that as well. With this option set, the catalysts become reachable for a few seconds after activation but then the sessions/connections to vBond and vManage are torn down and all I get are the same errors I mentioned in my original post ("ERR_BID_NOT_VERIFIED" and "BIDNTVRFD").

[Lei Tian]

The error might be related to root CA. Can you try download the master_root cert file from controller. On cedge, uninstall root-cert-chain using command 'request platform software sdwan root-cert-chain uninstall', then install the downloaded master_root cert file using command 'request platform software sdwan root-cert-chain install'

HTH,
Lei Tian

[MikeKoll]

I tried it but I got the same result. I already verified that the root CA cert is installed on every controller and cEdge and that it is the same valid cert. Because the same root cert is installed on every cEdge and controller, and controllers are able to form DTLS tunnels between each other without issues, I don't think root CA cert is at fault here because if that were the case then the controllers wouldn't be able to form tunnels between each other either. I'm thinking that maybe I'm signing the cEdge CSRs wrong on my CA ?  Not sure whether it helps but I use Linux command openssl -req -x509 -days 3650 -in c8000V-xx.csr -CA subca.crt -CAkey subca.key -set_serial xx -out c8000V-xx.crt to sign cEdge CSRs generated by vManage. 

Another thing that comes to mind is that maybe the version of Controllers and version of C8000V are not compatible ? Seems unlikely so I haven't tested other C8000V versions yet but what do you think ? I provided versions I use in the lab in the original post.

[MikeKoll · Accepted solution]

SOLVED.... The issue was controller and cEdge version mismatch. I tried C8000V version 17.6.3a and it had no trouble authenticating and forming DTLS tunnels with controllers.

I would like to know why neither cEdge nor any of the controllers were unable to provide ANY indication that the versions are simply just not compatible and instead they thrown errors at me that were unrelated to my issue and made me troubleshoot things that were in fact correct the entire time. Obviously it is my fault that I didn't try this earlier but come on.. 

Lesson learned I guess, advice for all starting networking engineers like me, check software compatibility matrix table before setting anything up...