cEdge stuck in state "connect", DCONFAIL

[rudimocnik]

Hi

I've got a cEdge (C1111-4PLTEEA)  running version 16.12.3 IOS-XE SDWAN. All controllers are running 19.2.2. I am using Cisco automated certificates on controllers and onbox certificate option for hardware.

I am struglling to make this router talk to vSmart and vManage. I've checked several things:

• clock matches with controllers
• whitelist on vManage and vSmart has the correct serial number and chassis number. Org name is also correct.
• local properties on cEdge are fine 
• certificate is installed
• root certificate is installed
• i can ping all public IPs of controllers
• color is public on all controllers and cEdge

PEER                      PEER                                                                            
PEER     PEER     PEER             SITE        DOMAIN PEER             PRIVATE  PEER             PUBLIC                                   LOCAL      REMOTE     REPEAT               
TYPE     PROTOCOL SYSTEM IP        ID          ID     PRIVATE IP       PORT     PUBLIC IP        PORT    LOCAL COLOR      STATE           ERROR      ERROR      COUNT DOWNTIME       
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vmanage  dtls     1.1.1.1          1000        0      172.29.28.10     12446    193.xx.xx.100   36060   public-internet  connect         DCONFAIL   NOERR      1     2020-05-11T15:46:09+0200
vsmart   dtls     1.1.1.3          1000        1      172.29.28.11     12446    193.xx.xx.102   9899    public-internet  connect         DCONFAIL   NOERR      1     2020-05-11T15:46:09+0200

PEER    PEER PEER            SITE       DOMAIN PEER                                    PRIV  PEER                                    PUB                                           GROUP      
TYPE    PROT SYSTEM IP       ID         ID     PRIVATE IP                              PORT  PUBLIC IP                               PORT  LOCAL COLOR     PROXY STATE UPTIME      ID         
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart  dtls 1.1.1.3         1000       1      172.29.28.11                            12446 193.xx.xx.102                          9899  public-internet No    connect            0           
vbond   dtls 0.0.0.0         0          0      193.xx.xx.101                          12346 193.xx.xx.101                          12346 public-internet -     up     0:00:10:54  0           
vmanage dtls 1.1.1.1         1000       0      172.29.28.10                            12446 193.xx.xx.100                          36060 public-internet No    connect            0 

I am not sure what am I missing. If vBond could establish connection why are vManage and vSmart not working huh?

Any ideas on how to troubleshoot this? Is there a way to do tcpdump on cEdge?

Rudi

[imortada]

All the connections between controllers look good to me.

vbond# show orchestrator connections
PEER PEER
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC ORGANIZATION
INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE NAME UPTIME
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0 vedge dtls 100.0.0.1 100 1 172.27.167.66 12426 172.27.167.66 12426 public-internet up Cisco-systems 0:18:26:49
1 vsmart dtls 1.1.1.2 100 1 172.27.167.87 12346 172.27.167.87 12346 default up Cisco-systems 0:23:45:58
1 vsmart dtls 1.1.1.2 100 1 172.27.167.87 12446 172.27.167.87 12446 default up Cisco-systems 0:23:45:54
1 vmanage dtls 1.1.1.1 100 0 172.27.167.69 12346 172.27.167.69 12346 default up Cisco-systems 0:23:45:14
1 vmanage dtls 1.1.1.1 100 0 172.27.167.69 12446 172.27.167.69 12446 default up Cisco-systems 0:23:45:15