Cisco SD-WAN - Control Connections (DTLS) are up, but BFDs are down

[Fabot]

Dear Cisco Community,

The DLTS control connections are up, but BFD sessions are down, with exeception of just 1 BFD session. Follow the outputs and topology in attachment.

[MHM Cisco World]

Bfd is only for data tunnel not for dtls control connection.

MHM

[MHM Cisco World]

Cisco doc.

It does not run on the control plane (DTLS or TLS) tunnels that Cisco SD-WAN Controllers establish with all Cisco devices in the network.

https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/ha-scaling/ios-xe-17/high-availability-book-xe/m-high-availability-and-scaling.html

MHM

[Fabot]

Dear @MHM Cisco World,

Yeah... Indeed they are different, but they are also dependent.
Before a BFD session can be built, a DTLS control connection should be stablished first.

In my scenario, just one BFD session is up. I would like to solve this. Can you help, please?

[MHM Cisco World]

So let agree one point here dtls is different than data tunnel' bfd not run over dtls.

Now all dtls is up and work that great.

The issue why one is up and three data tunnel is down?

See the color' only the match color is UP so I think you use color restricted.

Other three not work tunnel the two end color is different and with restricted the data never be UP.

Other note :- the color must not restricted and the two end is reachable to each other.

MHM

[Fabot]

They are not restricted. take a look at the config.

cEdge-ADM-1# show run vpn 0
vpn 0
name Transport
router
ospf
router-id 10.10.10.1
timers spf 200 1000 10000
area 0
interface ge0/2
exit
interface ge0/3
exit
exit
!
!
interface ge0/2
description "Color MPLS"
ip address 172.16.100.1/24
tunnel-interface
encapsulation ipsec
color mpls
allow-service all
no allow-service bgp
no allow-service dhcp
allow-service dns
allow-service icmp
allow-service sshd
allow-service netconf
no allow-service ntp
allow-service ospf
no allow-service stun
allow-service https
!
no shutdown
!
interface ge0/3
description "Color Public-Internet"
ip address 172.16.200.1/24
tunnel-interface
encapsulation ipsec
color public-internet
allow-service all
no allow-service bgp
no allow-service dhcp
allow-service dns
allow-service icmp
allow-service sshd
allow-service netconf
no allow-service ntp
allow-service ospf
no allow-service stun
allow-service https
!
no shutdown
!
ip route 0.0.0.0/0 192.168.18.254
!
cEdge-ADM-1#

==================================

cEdge-PTO-1# show run vpn 0
vpn 0
name Transport
router
ospf
router-id 20.20.20.1
timers spf 200 1000 10000
area 0
interface ge0/2
exit
interface ge0/3
exit
exit
!
!
interface ge0/2
description "Color MPLS"
ip address 172.16.100.2/24
tunnel-interface
encapsulation ipsec
color mpls
allow-service all
no allow-service bgp
no allow-service dhcp
allow-service dns
allow-service icmp
allow-service sshd
allow-service netconf
no allow-service ntp
allow-service ospf
no allow-service stun
allow-service https
!
no shutdown
!
interface ge0/3
description "Color Public-Internet"
ip address 172.16.200.2/24
tunnel-interface
encapsulation ipsec
color public-internet
allow-service all
no allow-service bgp
no allow-service dhcp
allow-service dns
allow-service icmp
allow-service sshd
allow-service netconf
no allow-service ntp
allow-service ospf
no allow-service stun
allow-service https
!
no shutdown
!
ip route 0.0.0.0/0 192.168.18.254
!

[MHM Cisco World · Accepted solution]

Two transport interface in two different edge route use defualt route via same next-hop

Can you check this point 

MHM

[Kanan Huseynli]

You have an issue in MPLS cloud, because even point-to-point reachibility in the same network - broadcast domains does not work. Begin with this.

Do ping from 172.16.100.1 to 172.16.100.2 and check. Even if it fails check for arp entry. If it is not created, you have an issue in underlay.  Behind MPLS cloud what do you use in background? L2 IOU switch? If yes, check mac table that switch learns mac-s or not.

[Fabot]

Dear Community,

I double-checked the IP reachability end-to-end, and it is working pretty fine.

However, the problem still persists. 

Basically, vBond is allowing just one DLTS control connection per vEdge. By the way, if I shutdown the interface interface/tunnel that is connecting to vBond to try a fail over, in order to let the other tunnel to stablish connection, vBond doesn't allow. In other words, vBond is allowing connection over a specific Color per vEdge.

The local error I get from the "show control connections-history" is "DISTLOC". According to Cisco, it is related to changes the Color or IP configuration. Indeed, I have changed the IP addresses from both edges interfaces.
From the vBond, using "show orchestrator connections-history" I found the following errors: RXTRDWN, ORPTMO, and DHSTMO.
On this context, how may I clear the records to use the new IP addressing that were just configured on the vedges?
Is therey another solution?

Follows important outputs in attachment.

[Kanan Huseynli]

I didn't get your underlay routing, why do you have "ip route 0.0.0.0/0 192.168.18.254" under VPN0?

You need to have two default route, one per underlay transport for controllers. Basically, in SD-WAN interfaces uses only next-hop via itself, not through another interface. Most probably, you have "gateway" in backbone for 172.16.100.0/24 and  172.16.200.0/24 subnets. Use those gateway and it should work.

You need two equal-cost default routes, one per interface.

[MHM Cisco World]

I already mention him this point.

The underlay must be reachable to make data tunnel (bfd) up.

@Fabot you ping but did you use source when ping from edge to edge?

MHM

[MHM Cisco World]

one more note 
you can have more than default route, the VPN 0 select the route depend on the transport interface 
MHM