Lisp Pub/Sub and iBGP

[KevinR99]

Hi

I have a test SDA setup and I've selected Lisp Pub/Sub as the method.  Should I expect to see iBGP between my two Borders?  I don't recall configuring it but it's there. 

I'm also using the Extranet feature and handing off to the external network on one link from each Border.  During some tests I'm getting issues with routes from the external devices and I'm thinking it might be an iBGP issue.

Thanks, Kev.

[jedolphi]

Hello, you will see Catalyst Center automated IBGP with VPNv4/6 AF enabled between BN and each CP for BGP route transport over SD-Access Fabric Site, explained in slides 37-40 in the following presentation: https://www.ciscolive.com/on-demand/on-demand-library.html?#/session/1707505512189001p6lp , and this should not impact Extranet routing, they should work seamlessly in parallel. If you want to t/shoot over slow pather (this forum) then please share details of the Extranet problem and I'll try to help here, or open TAC case. Best regards, Jerome

[KevinR99]

Thanks Jerome.  I enjoyed your sessions at CL recently.

The scenario I have is we have 2 Anywhere Borders handing off to a test network on one link each.  The test network has our DNAC and ISE in the same subnet.  So the test network advertises that subnet to both Borders over eBGP.  However, I have had to install static routes on the Borders to that subnet and when I remove the statics instead of a BGP route appearing a Lisp route to null0 appears and blackholes the return traffic to DNAC.

The Extranet policy isn’t really doing anything just now because I’ve not populated any subnets in the only VN I have as yet configured to share the Infra which is the provider.  It is the Infra only that I hand off to the external test network.

I’m due to do some more testing tomorrow.  It’s probably something fundamental I’ve missed.

Thanks for your input, Kev.

[jedolphi]

Hi Kevin, to reconfirm, are they Anywhere borders (default to each VN + import routes to LISP) or External borders (default route to each VN and do NOT import to LISP)? If Pub/Sub External then you need to have a default route in RIB for each VRF. If Anywhere you need both a default route in RIB for each VRF and BGP routes in RIB that you want to be imported into LISP. Static routes don't make sense to me as LISP does not import static routes in SDA.  Please consider opening a TAC case or contacting your Cisco SME. You could also review Michel's t/shooting presentation: https://www.ciscolive.com/on-demand/on-demand-library.html?#/session/1707505568614001pI1a . If none of that helps and you're desperate send me a DM and I'll arrange something to help.

Also I might mention that as a general rule we encourage External border, where either default route is present (BN is flagged as up per VRF in LISP), or default route is missing (BN is flagged as down per VRF in LISP). Importing routes can add complexity to LISP database and traffic forwarding paths without adding any functional value to the network, usually.

I'm glad the CL presentations added some value! Cheers, Jerome

[KevinR99]

Thanks again for your reply Jerome.

Perhaps I gave the wrong impression about the static routes.  Without the static routes I lose connectivity to my external subnet that hosts DNAC and ISE.  The static routes are a workaround to keep me connected to DNAC.  If I don’t have the static routes I should be receiving the DNAC subnet from 2 external BGP peers.  This will be the working solution if I can get it working.  However, with this setup the external subnet is seen on the Borders as a lisp route to null0.  If I recall correctly, I’ll check later, the null route had an AD of 10 which isn’t even the Lisp default AD.  So as a quick fix/workaround I installed static routes to override the null route and keep DNAC connected to my Borders and the rest of my fabric.  Otherwise I can’t push out any potential changes/fix to my Borders.

Regarding the Anywhere Border choice I selected this because I felt it gave me more flexibility in the future in that I could potentially have a single border connected to some other devices/subnets. Perhaps this is a wrong assumption but since DNAC does not allow us to change the Border type without removing it from the fabric and re-provisioning I wanted to choose the most flexible option from the start.  At this point I could remove the Borders and reprovision them es External borders not importing routes to Lisp.  However, I feel the option I have chosen should work.

I should perhaps add that currently the Borders do not have a default route.  This will come from a live network once we test and then integrate with that live, non-SDA, network.

If required I will raise a TAC case but at this stage I’m happier to work on the problem to try to find a solution without involving TAC.  Unless this is unexpected behaviour and needs TAC to look at it.

Many thanks again, Kev.

[jedolphi]

OK, thanks for explaining Kevin. For maximum flexibility I agree Anywhere Border might make sense, however please be mindful to only send it minimal BGP routes. Ideally you'd send it only 0/0 unless there's a specific need for other BGP routes to be imported in to LISP. In general, the simpler we keep the fabric routing the better.

In Pub/Sub you MUST have default route in RIB on the External Border Node, otherwise the EBN is effectively down from the perspective of LISP. Since you are using Extranet Policy you need 0/0 in the Provider VN on the EBN (or in your case Anywhere Border). Before you add 0/0, please execute this CLI on the site CP: show lisp remote-locator-set default-etrs . If you see a priority of 255 that means the BNs are telling LISP "do no use me for a default route, I have no external connectivity". Once you add 0/0 to Provider VN the priority shown in that CLI should change from 255 to 10 and traffic should start flowing,

Best regards, Jerome

[KevinR99]

Thanks Jerome.

I understand from the point of view of a consumer VN that without a 0/0 the Border is down.  However, I have no Edges at the moment.  We are simply testing the Border functionality before integrating it with the legacy network where the 0/0 will come from.  So all I need at the moment is connectivity from my Borders to external devices that host the DNAC subnet so that I can make changes to the Borders.  These external devices send some subnets to my Borders. What I don’t understand is why are null0 routes being created when the routes are valid whether or not I have a 0/0.  Are you saying that without a 0/0 on the Borders they will create these Null0 routes I see associated with any subnet coming from my external devices?  If so would I be correct in saying that with the same setup, i.e. no 0/0 at the moment, then even if I were to change my Borders to External instead of Anywhere then I would still see the same issue?  

Are there any plans to allow us to change the Border type in DNAC without having to remove the Border from the Fabric and re-provision it ?  The tick boxes associated with the Border type are greyed out once the Border functionality is designated.

Kev

[jedolphi]

Hi Kevin, have you by chance configured an overlapping route scenario which is not supported with Extranet? In other words you are advertising overlapping routes through EBGP into both the Provider VN and the Subscriber VN? With Extranet, in 99% of scenarios you only need Layer 3 Handoff in the Provider VN, not the Subscriber VNs.

If above guidance doesn't resolve please send me from both BN "show ip bgp all" and "show ip route vrf *", and from both CP "show lisp site". Thanks, Jerome

[KevinR99]

Jerome

There is no other hand off except one Infra handoff on each Border.  I have seen that issue you describe before where a consumer and a provider VN have handoffs but that is not the case here.  I am going to pick this up with one of our senior engineers tomorrow and do some more troubleshooting. I will update any findings.

Kev.

[jedolphi]

Hi Kevin. The only way I can reproduce the scenario you're describing is with overlapping routing. If I do exactly as discussed in this thread then below is what I see on my Anywhere BN. 10.67.33.0/24 is my external management subnet hosting Catalyst Center, ISE, DNS, DHCP, etc. When there is Extranet Policy, the Provider VN Null0 summary routes for Subscriber VN subnets are expected and normal. I'm  happy to solve this with you in DM if you like, please share the show commands I requested. Or raise a TAC case. Once we reach a conclusion we can update the discussion here for other people searching/reading later. Best regards, Jerome

FS1_BNCP1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 7 subnets, 3 masks
C 10.0.0.0/31 is directly connected, Vlan10
L 10.0.0.1/32 is directly connected, Vlan10
B 10.3.0.0/24 [200/0], 00:02:09, Null0 <<<IOT_VN Anycast Gateway
C 10.3.0.1/32 is directly connected, Loopback6002
B 10.4.7.0/24 [200/0], 00:02:09, Null0 <<<CORP_VN Anycast Gateway
C 10.4.7.1/32 is directly connected, Loopback6001
B 10.67.33.0/24 [20/0] via 10.0.0.0, 00:07:43
192.0.2.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.0.2.0/31 is directly connected, GigabitEthernet1/0/3
L 192.0.2.0/32 is directly connected, GigabitEthernet1/0/3
192.168.8.0/32 is subnetted, 2 subnets
C 192.168.8.59 is directly connected, Loopback0
O 192.168.8.60 [110/2] via 192.0.2.1, 00:07:43, GigabitEthernet1/0/3
FS1_BNCP1#

[KevinR99]

Hi Jerome

Just to update this issue.  I resolved the problem by removing my Borders from the fabric and re-adding as External only Borders.  I then recreated my Extranet policy and all my routing is k now.  Unfortunately I have no idea what caused the initial issue but it prompted me to change from Anywhere to External Borders.  This was an as yet nn-production fabric so it was feasible to remove and re-add the Borders.

Thanks again for your input Jerome.  Much appreciated.