SDWAN control Connection issue .

[mahende17feb@gmail.com]

Hi Cisco Community , 

I prepared sdwan LAB and iam facing issue with Vedge bringup . 

I checked control connection history and i get below error .

TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 mpls tear_down DISTLOC NOERR 0 2021-07-16T14:04:43+0000
vmanage dtls 1.1.1.1 1 0 192.168.1.10 12346 192.168.1.10 12346 mpls tear_down VM_TMO NOERR 4 2021-07-16T14:04:43+0000
vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 mpls tear_down VB_TMO NOERR 5 2021-07-16T13:42:11+0000 . 

Certificate is installed in all devices verified . 

please find below config of vedge . 

vedge# show running-config system
system
host-name vedge
system-ip 1.1.1.10
site-id 1
admin-tech-on-failure
no route-consistency-check
sp-organization-name MAHI-SDWAN
organization-name MAHI-SDWAN
vbond 192.168.1.13

vedge# show running-config vpn 0
vpn 0
interface ge0/0
ip address 192.168.1.100/24
ipv6 dhcp-client
tunnel-interface
encapsulation ipsec
color mpls
allow-service all
no allow-service bgp
allow-service dhcp
allow-service dns
allow-service icmp
allow-service sshd
allow-service netconf
no allow-service ntp
no allow-service ospf
allow-service stun
allow-service https
!
no shutdown
!
ip route 0.0.0.0/0 192.168.1.1

.

please find control connection details . 

vedge# show control connections
PEER PEER CONTROLLER
PEER PEER PEER SITE DOMAIN PEER PRIV PEER PUB GROUP
TYPE PROT SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR PROXY STATE UPTIME ID
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 mpls - up 0:00:03:50 0
vmanage dtls 1.1.1.1 1 0 192.168.1.10 12346 192.168.1.10 12346 mpls No up 0:00:03:50 0

i am not able to find what is issue . 

any suggestions would be thankful .

[Kanan Huseynli]

Hi,

based on your last output (sh control connections), you have UP status with vbond and vmanage.

History just shows previous status when device could not establish connection. Now, seems normal,isn't it?

HTH,

[mahende17feb@gmail.com]

Hi @Kanan Huseynli 

Totally agree it is up with Vbond and vmanage but not coming up with vsmart . iam not sure why it is not coming up with vsmart .

when we see vbond system ip it shows 0.0.0.0 ( correct me if iam wrong ) . Vbond system ip shows 0.0.0.0 from vsmart as well . 

[Kanan Huseynli]

Oh, yeah, that's true...vsmart is missing. But it is missing from histrory as well.

Do you have true control connection between vsmart and vbond , vsmart and vmanage?

Check control connection and history on vsmart as well.

Btw, it is normal you have vbond with 0.0.0.0 sys-ip.

HTH,

[Mahendervyas35821]

Hi @Kanan Huseynli 

I checked Vsmart and control connection between vsmart and vmanage is up but there is no control connection form between vmsart and vbond and vsmart  and vedge . 

Even in Vmanage Gui control plan should show up for vsmart but it is not showing . 

i checked there is no traffic block between any device as it is a lab and i dont have any firewall . 

Not sure were is a issue . 

Firmware version 20.3.3.1

[Kanan Huseynli]

Check to reach nodes via ping.

Check and share here "sh control connections-history" to understand the issue (at least) between vsmart and vbond.

At worst case, you may do packet capture by tcpdump to see vsmart sends and receives traffic from (at least) vbond in reality.

Regards,

[mahende17feb@gmail.com]

Hi ,

i checked tcpdump and packets are receiving from vbond to vsmart but not receiving any packets from vedge t o vsmart . When i try to ping vsmart from vedge it is able to reach . 

please find below outputs of vsmart show control connection history . 

PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT
INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE ERROR ERROR COUNT DOWNTIME
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0 vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 default tear_down CRTREJSER NOERR 4 2021-07-16T16:45:06+0000
0 vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 default up RXTRDWN VSCRTREV 0 2021-07-16T16:43:32+0000
0 vmanage dtls 1.1.1.1 1 0 192.168.1.10 12346 192.168.1.10 12346 default up RXTRDWN VSCRTREV 0 2021-07-16T16:43:31+0000
0 vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 default connect DCONFAIL NOERR 3 2021-07-16T16:41:27+0000
0 vmanage dtls 1.1.1.1 1 0 192.168.1.10 12346 192.168.1.10 12346 default tear_down DISTLOC NOERR 0 2021-07-16T16:39:13+0000
0 vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 default tear_down DISTLOC NOERR 0 2021-07-16T16:39:13+0000

I dont see any control connection history in vbond . 

please find control connection history of vedge . 

PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC LOCAL REMOTE REPEAT
TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR STATE ERROR ERROR COUNT DOWNTIME
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vmanage dtls 1.1.1.1 1 0 192.168.1.10 12346 192.168.1.10 12346 mpls connect DCONFAIL NOERR 6 2021-07-16T16:19:34+0000
vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 mpls tear_down DISTLOC NOERR 0 2021-07-16T16:17:56+0000
vmanage dtls 1.1.1.1 1 0 192.168.1.10 12346 192.168.1.10 12346 mpls tear_down VM_TMO NOERR 3 2021-07-16T16:17:56+0000
vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 mpls tear_down DISTLOC NOERR 0 2021-07-16T14:04:43+0000
vmanage dtls 1.1.1.1 1 0 192.168.1.10 12346 192.168.1.10 12346 mpls tear_down VM_TMO NOERR 4 2021-07-16T14:04:43+0000
vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 mpls tear_down VB_TMO NOERR 5 2021-07-16T13:42:11+0000
vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 mpls connect DCONFAIL NOERR 15 2021-07-16T13:40:36+0000
vbond dtls 0.0.0.0 0 0 192.168.1.13 12346 192.168.1.13 12346 mpls tear_down DISTLOC NOERR 0 2021-07-16T13:28:34+0000

Looking forward for your valuable reply . 

[Kanan Huseynli]

Hi,

you will have vedge-vsmart issue unless you solve vbond-vsmart connection i.e when you have successfull vbond-vsmart connection, then vbond sends vsmart info to vedge(s) and after it vedge can connect to vsmart.

Hence, first of all you should resolve problem with vbond-vsmart. Based on outputs you have certificate issue (CRTREJSER,VSCRTREV  errors) between vbond and vsmart.

https://www.cisco.com/c/en/us/support/docs/routers/sd-wan/214509-troubleshoot-control-connections.html

Ensure that vsmart and vbond can validate each other's certificate.

Ensure that time is correct. Ensure that organization name matches. You should see proper - ready certificates of vSmart in vManage gui (configuration-certification section). You should push controller certificates (just re-do) to vbond.

HTH,

[mahende17feb@gmail.com]

Hi kanan , 

I did exact same , vsmart ,vbond and vmanage all three shows up in Vmanage GUI . certificate section also looks good . 

in Vmanage Dashboard vsmart control plane is not showing thats the issue . 

attached images please see . 

I request if you can give you 15 min of time over remote for this would be helpful as i checked all things iam not able to find solution . 

[Sujai]

Based on your screenshot, all controller already up, and you can continue to bringup vedge after upload the wan edge list into vmanage.

Control status on vmanage dashboard means there are no established control plane connection between any vsmart and vedge.

Hope this helps.

[Kanan Huseynli]

Hi,

in his case, there is no control connection between vsmart and vbond, thus vbond cant push vsmart info to vedge and vedge cant establish control connection with vsmart.

Need to troubleshoot issue between vbond and vsmart. Based on outputs, it looks like certificate issue.

regards,

[Sujai]

Sorry for my bad, i have tried the same on the lab and seems i face same issue using this version of vmanage 20.3.3.1 and vbond and vsmart 20.3.3

I will try to change the version and see whether the result would be different

[Sujai]

After searching other post, i am able to bringup the vedge using both version 20.3.3.1 and 20.5.1. Just ensure clock between all controllers and vedge is same, or use ntp.

[Mahendervyas35821]

Hello @Sujai 

Sorry for Late response , as you said i tried time synchronize with NTP as well and still i have same issue , if you get time can we join some remote session . i am trying to bring this up from long time but no luck . 

you can reach me at mahender17feb@gmail.com if you feel comfortable for remote session .

[alintadimitri]

vpn 0 interface tunnel-interface control-connections—Attempt to establish a DTLS or TLS control connection for a TLOC (on vEdge routers only). This is the default behavior.

When a vEdge router has multiple tunnel interfaces and hence multiple TLOCs, the router establishes only a single control connection to the Cisco vManage. The router chooses a TLOC at random for this control connection, selecting one that is operational (that is, one whose administrative status is up). If the chosen TLOC becomes non-operational, the router chooses another one.

Read More: https://www.cisco.com/c/en/us/td/docs/routers/sdwan/command/sdwan-cr-book/config-cmd.html#wp4894365670