TLOC "Inv,U"; how to troubleshoot/resolve

[maxnpj]

I'm trying to allow access from a group of sites to another group of sites using one site in one of the groups as the hub. Here's the setup:

Region A has the data centers

Region B; BFD sessions to other region B sites and to region A DC's

Region C; BFD sessions to other region C sites and to region A DC's

I want to allow Region B sites to be able to access Region C sites by "pivoting" through one particular site in region C (a hub site). So in order to do that I added this to the policy applied to region B sites:
sequence 10
match route
site -id 21 <Random region C site-id>
!
action accept
set
tloc-list Site-ID-20-TLOCs (Site-ID 20 is in region C and will be the hub site)

Once that policy was in place I then go to a site in Region B and run "sh omp route <site-id 21 prefix>" I see the following: all of the routes are  "Inv, U"

Region B site# sh omp route <site-id 21 prefix>

PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 <site-id 21 prefix> 1.1.1.5 384837 1003 Inv,U installed <Site-ID 20 -V01 sys ip> public-internet ipsec -
                                 1.1.1.5 384838 1003 Inv,U installed <Site-ID 20 -V01 sys ip> private1 ipsec -
                                 1.1.1.5 384839 1003 Inv,U installed <Site-ID 20 -V02 sys ip> public-internet ipsec -
                                 1.1.1.5 384840 1003 Inv,U installed <Site-ID 20 -V02 sys ip> private1 ipsec -

If I remove that part of the policy and then go to the same site in Region B and run "sh omp route <site-id 21 prefix>" I see the following:

Region B site# sh omp route <site-id 21 prefix>
% No entries found.
Region B site#

So...where am I going wrong here? There are no BFD sessions set up between sites in regions B & C, but I didn't think I needed the BFD sessions to route traffic through a hub.

Thanks in advance. Rookie in training here........

[Octavian Szolga]

Hi,

To quick answer you question, think of it as this:

In SDWAN, If router X has to talk to router Y, two things have to happen:

1. TLOCs have to be exchanged; because TLOCs were exchanged, BFD sessions come UP

2. (omp/service side) routes have to be exchanged; routes have to resolve to a TLOC just like in BGP a route has to have a valid next-hop;

So, in your case, your policy is changing the TLOC for some routes belonging to region C when advertised to region B, but the new TLOC (of the region C hub router) is not advertised to region B, actually causing all region B routers to invalidate the route, because nobody knows how to reach that TLOC.

Long story short, you have to advertise the TLOCs of router hub from region C to region B in order to provide connectivity between B and C regions using this router.

When advertising the TLOCs of region C hub router, you basically end up having BFD (IPsec) sessions between all region B routers and region C hub router. Bottom line, you just 'promoted' region C hub router to something like an A region DC router.

I don't know the specifics of your topology, but if you don't want a full mesh in your SDWAN deployment, you can adjust your policy so that region C talks to region B on a hub to hub level.

Basically, traffic would go: regular region B router > region B hub router > region C hub router > region C regular router.

Best regards,

Octavian

[maxnpj]

Octavian;

First off...thank you for the answer and the detail. It's very helpful and educational. Here is a follow-on question, there's a lot here so feel free to ignore this post, we're getting deep into the topology.

In response to this comment:
Basically, traffic would go: regular region B router > region B hub router > region C hub router > region C regular router.

(I'm going to use your descriptions here to keep us on the same page)
Currently, I have various control policies that allow the following BFD sessions as follows:
"regular region B" sites <BFD>"region B hub"<BFD>"region C hub"<BFD>"regular region C" sites
So...like you said, traffic should follow: regular region B router > region B hub router > region C hub router > region C regular router.

I then go to the "regular region B" control-policy and add the following:
sequence 60
match route
site-id Region C site-1
site-id Region C site 2
!
action accept
set
tloc-list Region_B_HUB_TLOCs

So..if I translate that to English as I think it's written:
match routes of region C site-1 and region C site-2;
upon match; accept those routes and set the TLOC to "Region_B_HUB_TLOCs"; and by setting the TLOCs I'm really setting the next-hop as the region B hub

I then go to the "regular region C" control-policy and add the following:
sequence 60
match route
site-id Region B site-1
!
action accept
set
tloc-list Region_C_HUB_TLOCs

Looks good so far.....but a traceroute from "regular region B" site to "regular region C" site will only get as far as the "region B" hub. The "region B" hub knows the OMP route to the "regular region C" site but it is "Inv, U", and I believe that's because the TLOC next hop is the system IP of the "regular region C" site, and the "region B" hub does NOT have BFD sessions to the "regular region C" site.

I know there are many parts of my production centralized policy I'm not explaining here, but I was thinking the same thing you mentioned; router > hub router > hub router > router and that's where I'm trying to get to.

[Kanan Huseynli]

Hi,

you don't have BFD session i.e tunnel between site-B router (any or one of them) and site-C router (which is hub). Then, how router B may send traffic over non-existing tunnel? No way.

You should definitely have tunnel (and bfd over it) between 2 sites in order to send data traffic from one to another (and vice versa).

Btw, how do you restrict one group from another? Using policy or restric/tunnel group commands?

HTH,

[maxnpj]

Kanan;

We use a combination of the things you mentioned. The centralized policy matches on routes and TLOCs from one region to another. We also use "restrict" on the tunnel interfaces. Our setup is (basically) Data centers in region A; remote sites in region A, as well as remote sites in regions B / C / D / E. All remote sites have BFD sessions to to DC in region A; and then each region has certain sites that have BFD sessions to each other. 

We use a "tiered' structure to identify sites. So then a Tier 1 remote site has BFD sessions to other Tier 1 sites as well as region A DC; tier 2 and tier 3 remotes sites will have BFD sessions to region A DC but not each other. That pattern is the same among regions. Each of the regions also has a "regional data center". So further, a region's Tier 1 sites have BFD sessions to each other; the regional DC (RDC) and the region A DC. 

One of the reasons for this post is the trouble I'm having getting from a remote site in one region to a remote site in another region by using the RDC as the "transit" point....as Octavian mentioned in a previous post "regular region B router > region B hub router > region C hub router > region C regular router."

[Kanan Huseynli]

Hi,

enable "service TE" on DC routers (or hubs) in VPN0 template.

Regards,