Routing issue with Cisco DNA

[Kamran Mustafayev]

Hi,

I have a DNA fabric network, consist of border routers and fusion, edge connected to borders:

Edge1 --------- Border1-------Fusion1

Edge1 --------- Border2-------Fusion2

Borders have iBGP between them, Fusion1 and Fusion2 have iBGP as well,

Behind edge there is subnet lets say 10.90.0.0/22 and I have routes to the devices in via LISP and BGP (between borders):

B1:

*> 10.90.0.0/22 0.0.0.0 32768 i - this are received from peer B2 with aggregate address summary-only command* i 172.16.12.174 0 100 0 i
s> 10.90.0.20/32 0.0.0.0 10 32768 ? - this are redistributed to BGP from LISP
s> 10.90.0.21/32 0.0.0.0 10 32768 ? - this are redistributed to BGP from LISP
s> 10.90.3.254/32 0.0.0.0 0 32768 i - this are redistributed to BGP from LISP

B2:

*> 10.90.0.0/22 0.0.0.0 32768 i - this are received from peer B2 with aggregate address summary-only command* i 172.16.12.173 0 100 0 i
s> 10.90.0.20/32 0.0.0.0 10 32768 ? - this are redistributed to BGP from LISP
s> 10.90.0.21/32 0.0.0.0 10 32768 ? - this are redistributed to BGP from LISP
s> 10.90.3.254/32 0.0.0.0 0 32768 i - this are redistributed to BGP from LISP

Problem is, I have Edge switch connected to both of this borders, when lets say link from Edge to B1 is down, LISP route goes down and logically it should sent traffic through B2, but the problem is that BGP best path points to 0.0.0.0 and being blackholed, second path in BGP routing table which is peer IP (in this case 172.16.12.174) never used, this happens because "aggregate address summary only command" injectw 0.0.0.0 into BGP and choosing it as the best in BGP routing table, and beside that there is redistributed from LISP 3 routes with metric 10

I'm trying to understand if its kind of LISP/BGP routing loop, its DNA network and all this aggregates are pushed by DNA, I wonder if we need to add some additional configuration beside automated by DNA to avoid this?

Let me know if you need additional details

[Torbjørn]

Are border 1 and border 2 connected directly to eachother, or are they both only connected to your edge? The summary routes are generated by DNA center and shouldn't cause any such routing issues in normal circumstances.

[Kamran Mustafayev]

Yes, they are connected directly to each other, there is port-channel and peering done on interface VLAN between them

[andy!doesnt!like!uucp]

what do u mean under "Behind edge there is subnet lets say 10.90.0.0/22"? 
whatever "routes" behind ENs will be visible in bgp on BNs as LISP redistributed " s>"
can u drop here output of "show ip cef vrf <target_vn> 10.90.0.0/22 de" from BN?

[Kamran Mustafayev]

It is redistributed with suppressed s>:

Border1:

* i 10.90.0.0/22 172.16.12.174 0 100 0 i
*> 0.0.0.0 32768 i
s> 10.90.0.20/32 0.0.0.0 10 32768 ?
s> 10.90.0.21/32 0.0.0.0 10 32768 ?
s> 10.90.0.23/32 0.0.0.0 10 32768 ?
s> 10.90.3.254/32 0.0.0.0 0 32768 i

Border2:

*> 10.90.0.0/22 0.0.0.0 32768 i
* i 172.16.12.173 0 100 0 i
s> 10.90.0.20/32 0.0.0.0 10 32768 ?
s> 10.90.0.21/32 0.0.0.0 10 32768 ?
s> 10.90.0.23/32 0.0.0.0 10 32768 ?
s> 10.90.3.254/32 0.0.0.0 0 32768 i

--------------------

BORDER_NODE1#show ip cef 10.90.0.0/22 de
10.90.0.0/22, epoch 2, flags [cover dependents, rib only nolabel, check lisp eligibility]
LISP remote EID: 279 packets 160128 bytes fwd action signal, cfg as EID space
LISP source path list
attached to LISP0.4097
Covered dependent prefixes: 3
notify cover updated: 3
1 IPL source [no flags]
attached to LISP0.4097
BORDER_NODE1#

BORDER_NODE2#show ip cef 10.90.0.0/22 de
10.90.0.0/22, epoch 2, flags [cover dependents, rib only nolabel, check lisp eligibility]
LISP remote EID: 327 packets 187200 bytes fwd action signal, cfg as EID space
LISP source path list
attached to LISP0.4097
Covered dependent prefixes: 3
notify cover updated: 3
1 IPL source [no flags]
attached to LISP0.4097

This are my AP subnets, its getting redistributed into BGP by DNA with "redistributed lisp 10", then borders sending this routes as aggregated summary-only 10.90.0.0/22 towards fusion routers, BUT there is also iBGP configured between the borders, so whenever my lisp routes goes down (lets say link towards edge from Border 2 went down) I should not be able to receive those routes via lisp and I won't be able to redistributed them into BGP, so I need iBGP between borders to be used in this case, so I can reach that lisp subnets via iBGP to Border1, but this fails cause aggregate summary only created by DNA creates 0.0.0.0 as the next hop for 10.90.0.0/22 via iBGP and it just blackholing this traffic

[andy!doesnt!like!uucp]

"so whenever my lisp routes goes down (lets say link towards edge from Border 2 went down) I should not be able to receive those routes via lisp and I won't be able to redistributed them into BGP, so I need iBGP between borders to be used in this case, so I can reach that lisp subnets via iBGP to Border1, but this fails cause aggregate summary only created by DNA creates 0.0.0.0 as the next hop for 10.90.0.0/22 via iBGP and it just blackholing this traffic"
it's wrong conclusion. in fact both your BNs "learn" IPV4 eids of your APs from ENs. & thus unless all BNs go down FNs will learn aggregated subnet from live BN. 

[Kamran Mustafayev]

Yes BN's learns them from EN routers, but if my external path to fusion goes down from Border1 + link from border2 to the same edge node goes down, it should redirect traffic to Border2 via iBGP path, should be Edge node > Border 1 > ibgp > Border2 > Fusion 2 > outside, but the problem is that aggregate summary only command on borders generates 0.0.0.0 as the next hop i.e blackholing traffic because it sees /32 subnets from 10.90.0.0/22 that are redistributed from lisp and see them as the best path if lisp routes goes down, so if link to EN goes down it looses lisp routes and trying to use iBGP routes, but iBGP route to neighbor border blackholing traffic:

* i 10.90.0.0/22 172.16.12.174 0 100 0 i - should use this one
*> 0.0.0.0 32768 i    ------------ blackholed traffic

[Torbjørn]

Can you post the output of "show run | s lisp"? This shouldn't happen as your border should be learning the same LISP EID mappings from your other border/control-plane node.

[Kamran Mustafayev]

Border1:

BORDER_NODE1#show run | s lisp
router lisp
locator-table default
locator-set WLC
172.16.12.178
exit-locator-set
!
locator-set rloc_74d3b092-ceea-4dc2-ad76-f53905188508
IPv4-interface Loopback0 priority 10 weight 10
auto-discover-rlocs
exit-locator-set
!
locator default-set rloc_74d3b092-ceea-4dc2-ad76-f53905188508
service ipv4
encapsulation vxlan
itr map-resolver 172.17.170.13
itr map-resolver 172.17.170.14
etr map-server 172.17.170.13 key 
etr map-server 172.17.170.13 proxy-reply
etr map-server 172.17.170.14 key 
etr map-server 172.17.170.14 proxy-reply
etr
sgt distribution
sgt
no map-cache away-eids send-map-request
proxy-etr
proxy-itr 172.17.170.13
map-server
map-resolver
exit-service-ipv4
!
service ethernet
map-cache-limit 32768
itr map-resolver 172.17.170.13
itr map-resolver 172.17.170.14
itr
etr map-server 172.17.170.13 key
etr map-server 172.17.170.13 proxy-reply
etr map-server 172.17.170.14 key
etr map-server 172.17.170.14 proxy-reply
etr
map-server
map-resolver
exit-service-ethernet
!
instance-id 4097
remote-rloc-probe on-route-change
service ipv4
eid-table default
map-cache 10.90.0.0/22 map-request
route-export site-registrations
distance site-registrations 250
map-cache site-registration
exit-service-ipv4
!
exit-instance-id
!
instance-id 4099
remote-rloc-probe on-route-change
service ipv4
eid-table vrf Campus_VN
database-mapping 10.112.32.4/30 locator-set rloc_74d3b092-ceea-4dc2-ad76-f53905188508
route-export site-registrations
distance site-registrations 250
map-cache site-registration
exit-service-ipv4
!
exit-instance-id
!
map-server session passive-open WLC
site site_uci
description map-server configured from Cisco DNA-Center
authentication-key 7 
eid-record instance-id 4097 10.90.0.0/22 accept-more-specifics
eid-record instance-id 4099 10.121.0.0/24 accept-more-specifics
eid-record instance-id 4099 10.121.1.0/24 accept-more-specifics
eid-record instance-id 4099 10.122.0.0/20 accept-more-specifics
eid-record instance-id 4099 10.124.0.0/16 accept-more-specifics
eid-record instance-id 4099 10.125.0.0/19 accept-more-specifics
eid-record instance-id 4099 10.126.0.0/22 accept-more-specifics
eid-record instance-id 4099 10.126.16.0/20 accept-more-specifics
eid-record instance-id 8191 any-mac
eid-record instance-id 8201 any-mac
eid-record instance-id 8203 any-mac
eid-record instance-id 8207 any-mac
eid-record instance-id 8209 any-mac
eid-record instance-id 8210 any-mac
eid-record instance-id 8211 any-mac
eid-record instance-id 8212 any-mac
exit-site==================================================

Border2:

BORDER_NODE2#show run | sec lisp
router lisp
locator-table default
locator-set WLC
172.16.12.178
exit-locator-set
!
locator-set rloc_9fe7384c-1128-459e-b6a9-5c705d7167b
IPv4-interface Loopback0 priority 10 weight 10
auto-discover-rlocs
exit-locator-set
!
locator default-set rloc_9fe7384c-1128-459e-b69-5c7052d7167b
service ipv4
encapsulation vxlan
itr map-resolver 172.17.170.13
itr map-resolver 172.17.170.14
etr map-server 172.17.170.13 key
etr map-server 172.17.170.13 proxy-reply
etr map-server 172.17.170.14 key
etr map-server 172.17.170.14 proxy-reply
etr
sgt distribution
sgt
no map-cache away-eids send-map-request
proxy-etr
proxy-itr 172.17.170.14
map-server
map-resolver
exit-service-ipv4
!
service ethernet
map-cache-limit 32768
itr map-resolver 172.17.170.13
itr map-resolver 172.17.170.14
itr
etr map-server 172.17.170.13 key 
etr map-server 172.17.170.13 proxy-reply
etr map-server 172.17.170.14 key 
etr map-server 172.17.170.14 proxy-reply
etr
map-server
map-resolver
exit-service-ethernet
!
instance-id 4097
remote-rloc-probe on-route-change
service ipv4
eid-table default
map-cache 10.90.0.0/22 map-request
route-export site-registrations
distance site-registrations 250
map-cache site-registration
exit-service-ipv4
!
exit-instance-id
!
instance-id 4099
remote-rloc-probe on-route-change
service ipv4
eid-table vrf Campus_VN
database-mapping 10.112.32.0/30 locator-set rloc_9fe7384c-1128-459e-b69-5c7052d7167b
route-export site-registrations
distance site-registrations 250
map-cache site-registration
exit-service-ipv4
!
exit-instance-id
!
map-server session passive-open WLC
site site_uci
description map-server configured from Cisco DNA-Center
authentication-key
eid-record instance-id 4097 10.90.0.0/22 accept-more-specifics
eid-record instance-id 4099 10.121.0.0/24 accept-more-specifics
eid-record instance-id 4099 10.121.1.0/24 accept-more-specifics
eid-record instance-id 4099 10.122.0.0/20 accept-more-specifics
eid-record instance-id 4099 10.124.0.0/16 accept-more-specifics
eid-record instance-id 4099 10.125.0.0/19 accept-more-specifics
eid-record instance-id 4099 10.126.0.0/22 accept-more-specifics
eid-record instance-id 4099 10.126.16.0/20 accept-more-specifics
eid-record instance-id 8191 any-mac
eid-record instance-id 8201 any-mac
eid-record instance-id 8203 any-mac
eid-record instance-id 8207 any-mac
eid-record instance-id 8209 any-mac
eid-record instance-id 8210 any-mac
eid-record instance-id 8211 any-mac
eid-record instance-id 8212 any-mac
exit-site

=======================================================================

[andy!doesnt!like!uucp]

in the scenario where u lose links BN1-FN1 & ENx-BN2 simultaneously, BN2 will still keep your APs EIDs registration with LISP via path ENx-BN1-BN2. Meaning it will always know where to send traffic destined to APs unless Underlay path between ENx & BN2 is lost. 
btw it's good practice to interconnect BNs with FNs in cross manner additionally.

[Kamran Mustafayev]

Understood, it seems I'm only testing it with two edge switches and getting impression of something being wrong, but as you mentioned if I will have plenty of edges connected traffic will still find a way to the destination downstream

[Kamran Mustafayev]

btw, what do you think of ISIS link between borders of course with lan automation, will it bring redundancy on underlay level, just wanted to know your opinion if we use it with cross cons

[andy!doesnt!like!uucp]

it's basically what we have in fabric sites with 2+ BN|CPs. Now imagine situation that u need 3rd BN (for L2-handoff f.e.) & number of ENs. how will u connect it to underlay assuming u dont want to introduce distribution layer? it's a rhetoric Q :0) 

[Kamran Mustafayev]

Got, it might be the reason why im failing, I don't have lan automation ISIS link between my Borders (CP's), will try to bring it up and re-check

[Torbjørn]

It can be useful to inspect your LISP EID table to see the state of LISP mappings with "show lisp site"