SD-WAN 19.2.3 - vBond create unexpected connection to vEdges

[Ruben_IT]

Hi, 

we have a SD-WAN laboratory working correctly to show to our customers

We have verified that the vBond and vEdge keep connection permanent between them.

The Cisco theory always says that the vBond connection with the vEdge are temporal.

We have seen the connection permanent in a NAT router that we have configured in the middle of connection between vbond and vedge, and we can confirm this connecion in the vBond server and the vEdge Router.

Everything is working but....

Which is the reason?, we dont understand why this connection is UP when the theory explain that this connection must goes down.

Maybe this behaviour is something about stun server?

Thanks for your help,

Best REgards,

SHOWs from devices:

vBond address:  100.0.0.3:

vEdge-1 system ip 50.1.10.1 Address:  Public IP    172.16.100.71

vEdge-2 system ip 50.1.20.1 Address:  Public IP    172.16.101.71

vEdge-3*  system ip 50.1.30.1  Address:  Public IP    172.16.102.71

CSR1Kv 

From VBOND:

vbond# show orchestrator connections

PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC ORGANIZATION INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE NAME UPTIME -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0 vedge dtls 50.1.10.1 100 1 172.16.100.71 12346 172.16.100.71 12346 biz-internet up SD-WAN POC - 4xxx 0:01:01:18

0 vedge dtls 50.1.20.1 101 1 172.16.101.71 12346 172.16.101.71 12346 biz-internet up SD-WAN POC - 4xxx 0:01:01:19

0 vedge dtls 50.1.30.1 102 1 172.16.102.71 12346 172.16.102.71 12346 biz-internet up SD-WAN POC - 4xxx 0:00:59:55 

NAT Devices:

NAT-ROUTER#sh ip nat translations
Pro Inside global Inside local Outside local Outside global
icmp 172.106.103.2:4465 172.16.103.71:4465 100.0.0.3:4465 100.0.0.3:4465
icmp 172.106.103.2:4583 172.16.103.71:4583 100.0.0.3:4583 100.0.0.3:4583
icmp 172.106.103.2:4701 172.16.103.71:4701 100.0.0.3:4701 100.0.0.3:4701
icmp 172.106.103.2:4823 172.16.103.71:4823 100.0.0.3:4823 100.0.0.3:4823
icmp 172.106.103.2:4947 172.16.103.71:4947 100.0.0.3:4947 100.0.0.3:4947
icmp 172.106.103.2:5053 172.16.103.71:5053 100.0.0.3:5053 100.0.0.3:5053
icmp 172.106.103.2:5195 172.16.103.71:5195 100.0.0.3:5195 100.0.0.3:5195
udp 172.106.103.2:12346 172.16.103.71:12346 100.0.0.3:12346 100.0.0.3:12346

From one vEdge Router:

vEdge_Router# show control connections

PEER PEER CONTROLLER PEER PEER PEER SITE DOMAIN PEER PRIV PEER PUB GROUP TYPE PROT SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT LOCAL COLOR PROXY STATE UPTIME ID ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

vsmart dtls 50.1.1.3 10 1 192.168.1.72 12346 192.168.1.72 12346 biz-internet No up 0:01:10:55 0

vbond dtls 0.0.0.0 0 0 100.0.0.3 12346 100.0.0.3 12346 biz-internet - up 0:01:10:56 0

vmanage dtls 50.1.1.2 10 0 192.168.1.71 12346 192.168.1.71 ..................

[ashok_boin]

Hi,

I believe there is continuous kind of keepalive packets between vBond and vEdge devices as we do have parameters "VB_TMO" (vBond peer timeout) as part of resolving connection issues.

So, as far as the handshake between vBond and vEdge is successful, this connection should be in UP condition only.

SD-WAN Routers: Troubleshoot Control Connections - Cisco Community

Regards...

Ashok.

[ruben.rexa]

THE FIRST -- Thanks for your reply.

But...

i dont think so....

I have read a lot of web pages, and all the sites always says the same, the connection is not permanent.

You can read in the Cisco Design Guide the same info:

https://www.cisco.com/c/en/us/td/docs/solutions/CVD/SDWAN/cisco-sdwan-design-guide.html

"

Control Connections

Cisco SD-WAN vManage and vSmart controllers initially contact and authenticate to the vBond controller, forming persistent DTLS connections, and then subsequently establish and maintain persistent DTLS/TLS connections with each other. WAN Edge devices onboard in a similar manner, but drop the transient vBond connection and maintain DTLS/TLS connections with the vManage and vSmart controllers. 

"

they call to the connection like "transient" connection

And in my lab, the connection is stablished the same time like vmanage or vSmart and never is lost

I think that there is any configuration reason or other software problems.

Again, Thanks for you reply,

[ashok_boin]

Hmm, you are right. They should be transient connections b/w vbond & vedge. I think some configuration is preventing default behavior. Do you have any debug options available to try with restarting process may be with "clear dns cache"?

[Ruben_IT]

Thanks for your reply again.

Hi,

i have reload all the devices several times and always get up this connection.

And the laboratory is working correctly, all the user traffic goes the Wan edge to Wan edge without problems. 

It is only with the vBond control connections.

I have read explanation with similar problems:

https://community.cisco.com/t5/sd-wan-and-cloud-networking/encapsulation-ipsec-on-vbond/td-p/4451149

or 

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvq04498

But it is not the same problem,

Best Regards,

[ashok_boin]

In this case, it seems a bug in Cisco software. Are you sure you can't debug the connection flows after rebooting?

[svemulap@cisco.com · Accepted solution]

hi Ruben -

There is one case where control connection from vEdge to vBond is persistent, when equilibrium is not reached.
i.e., by default max-control-connections==2 and if you have only ONE vSmart in the Overlay, vEdge will keep asking
vBond for the other (additional) vSmart. You can configure max-connections==1, to change it from the default.

Sample output: (set it to 1)


vedge(config)# vpn 0 interface ge0/0 tunnel-interface max-control-connections ?
Description: Maximum control connections for this TLOC (default is same as maximum OMP sessions)
Possible completions:
<0..100>
vedge(config)#

HTH

[Ruben_IT]

HI!!,

Thanks for your support

i have configured by default this value in TLOC INTERNET wan connection and i have configured manually to 0 in TLOC MPLS wan interface connection.

i will try to modify the default value 2,  to new value of 1.

I will take some time because now i have not access to the labo.

i tell you something as soon as possible.

Again, Thanks for your help,

[Ruben_IT]

AND YES!!!!!!

This was the problem, i didnt see this posibility in any site.

THANKS!!!!

configured max-connection 1 on INTERNET TLOC (only control connection interface) and FIXED:

The permanent connection from vedge to vbond has been cleared.

vbond2# show orchestrator connections
PEER PEER
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC
ORGANIZATION
INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE
NAME UPTIME
-------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------
0 vsmart dtls 50.1.1.3 10 1 192.168.1.72 12346 192.168.1.72 12346 default up
SD-WAN POC - AAAAA 0:00:27:59
0 vsmart dtls 50.1.1.3 10 1 192.168.1.72 12446 192.168.1.72 12446 default up
SD-WAN POC - AAAAA 0:00:27:59
0 vmanage dtls 50.1.1.2 10 0 192.168.1.71 12346 192.168.1.71 12346 default up
SD-WAN POC - AAAAA 0:00:27:48
0 vmanage dtls 50.1.1.2 10 0 192.168.1.71 12446 192.168.1.71 12446 default up
SD-WAN POC - AAAAA 0:00:27:49
0 vmanage dtls 50.1.1.2 10 0 192.168.1.71 12546 192.168.1.71 12546 default up
SD-WAN POC - AAAAA 0:00:27:51
0 vmanage dtls 50.1.1.2 10 0 192.168.1.71 12646 192.168.1.71 12646 default up
SD-WAN POC - AAAAA 0:00:27:51
0 vmanage dtls 50.1.1.2 10 0 192.168.1.71 12746 192.168.1.71 12746 default up
SD-WAN POC - AAAAA 0:00:27:50
0 vmanage dtls 50.1.1.2 10 0 192.168.1.71 12846 192.168.1.71 12846 default up
SD-WAN POC - AAAAA 0:00:27:49
0 vmanage dtls 50.1.1.2 10 0 192.168.1.71 12946 192.168.1.71 12946 default up
SD-WAN POC - AAAAA 0:00:27:51
0 vmanage dtls 50.1.1.2 10 0 192.168.1.71 13046 192.168.1.71 13046 default up
SD-WAN POC - AAAAA 0:00:27:52

vbond2# show orchestrator connections-history
Legend for Errors
ACSRREJ - Challenge rejected by peer. NOVMCFG - No cfg in vmanage for device.
..................................

PEER PEER PEER
PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC
REPEAT
INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE
LOCAL/REMOTE COUNT DOWNTIME
-------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------
0 vedge dtls 50.1.30.1 102 1 172.16.102.71 12346 172.16.102.71 12346 biz-internet up
RXTRDWN/DISCVBD 1 2021-12-16T21:25:05+0000
0 vedge dtls 50.1.10.1 100 1 172.16.100.71 12346 172.106.100.2 12346 biz-internet up
RXTRDWN/DISCVBD 1 2021-12-16T21:21:42+0000
0 vedge dtls 50.1.20.1 101 1 172.16.101.71 12346 172.16.101.71 12346 biz-internet up
RXTRDWN/DISCVBD 1 2021-12-16T21:21:39+0000
0 vedge dtls 50.1.40.1 103 1 172.16.103.71 12346 172.106.103.2 12346 biz-internet up
RXTRDWN/DISCVBD 0 2021-12-16T21:19:23+0000

AGAIN,

THANKS FOR YOUR HELP,

[BryanHefner2568]

Looks like one of your control connections is having an issue. Use “show control connections history”. Should give you an error code on the connection that is failing. 

---

[Ruben_IT]

THANKs for your Reply,

Hi,

I cant execute the command now, tomorrow i will test.

But, i can see the connection UP in all the vEdge (show orchestration command)

    (the laboratory only has 3 vEdge ( 2-vEdge Cloud,   1-CSR1Kv )

vbond# show orchestrator connections

PEER PEER PEER PEER PEER SITE DOMAIN PEER PRIVATE PEER PUBLIC ORGANIZATION INSTANCE TYPE PROTOCOL SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT REMOTE COLOR STATE NAME UPTIME -------------------------------------------

0 vedge dtls 50.1.10.1 100 1 172.16.100.71 12346 172.16.100.71 12346 biz-internet up SD-WAN POC - 4xxx 0:01:01:18

0 vedge dtls 50.1.20.1 101 1 172.16.101.71 12346 172.16.101.71 12346 biz-internet up SD-WAN POC - 4xxx 0:01:01:19

0 vedge dtls 50.1.30.1 102 1 172.16.102.71 12346 172.16.102.71 12346 biz-internet up SD-WAN POC - 4xxx 0:00:59:55 

And like i said before, the vEdge connect with the vSmart and vManage and send traffic.

If we have had problem with the control connection with the vBond, i suppose that the connection with the vManage and vSmart dont happens. it is necesary connect correctly with the vBond to progress in the connections with vManage and vSmart

Best Regards,

[Ruben_IT]

Hi again,

I have got the show command, and like i suppose, there is no problem with the Controller Connection, because the connection progress to the vManage and vSmart.

(i attach the show command)

 

Thanks for your help

[BryanHefner2568]

From your earlier post, you were seeing the vbond connections on the vEdge device.  Can you run the "show control connections-history" command from the edge device that you are getting the constant vbond connection attempts?  I have not used the viptela hardware, but I have built a few large SD-WAN builds for some customers.  I have seen ISP issues that cause the control connections to the edge routers to constantly go up and down, which would explain why the vbond connection keeps popping up.

[Ruben_IT]

Hi again,

I´m sorry by the time  taken in answer.

i have attached the command, and YES, this error VM-TMO (TIME-OUT) is something that we can understand like the error

NOW, in the lab, we are lossing ping from vEdge to the vBond and vSmart if we send the traffic from VPN0 without associate to the transport Interface.

The reason is because we have two transport aisolated, and we have had a ip route 0.0.0.0/0 to reach the other ends

This configuration must be works, BUT, it is failing. 

We think that there is a bad behaviour with the traffic.when this traffic goes to the  other end by VPN0 without specify source.

We have opened another " ask " in cisco community. reporting this problem.

https://community.cisco.com/t5/sd-wan-and-cloud-networking/sdwan-loss-ping-vpn-0-with-2-wan-separate-transport-in-vmanage/td-p/4502812SDWAN-LOSS PING VPN 0 with 2 wan separate transport in vManage troubl. - Cisco Community

Thanks for your help,

Best regards,

[Ruben_IT]

Hi again, thanks for your time!!!

i have checked the command reference and there is no debug command to vbond orchestration controller traffic

https://www.cisco.com/c/en/us/td/docs/routers/sdwan/command/sdwan-cr-book/operational-cmd.html#wp6767394710

Cisco SD-WAN Command Reference - Operational Commands [Cisco SD-WAN] - Cisco

I have seen :

   "debug transport events"

But i think that this is no the correct command to check this problem.

Do you have any idea?

Best REgards,