LISP not routing VN traffic beyond fabric

[Joshua Marks]

Hello,

We have recently deployed an SD-Access network with two defined VNs: users_vn and devices_vn.

After provisioning the fabric roles in DNA Center we are testing end device connectivity on the users_vn. Users can authenticate and are allocated an IP address using DHCP. Devices within the fabric can ping each other, but all external-bound user traffic is failing at the border. Switches and devices in the GRT/INFRA_VN (APs, switches) can talk outside the fabric, however devices in the defined VNs cannot. Likewise, endpoints in the fabric cannot be contacted and traffic destined to the fabric is dropped at the border. Note that no traffic is reaching the Fusion firewall. Looking in the Border Router LISP instance database for the users_vn, we can see no entries for clients, but when we run show lisp site on the same device we can see the endpoints in question. The endpoints are also seen in the routing table as LISP routes. This feels like irregular behaviour and we can't be sure why the VN traffic is not routing through the border. MTU on all switches is 9100. Output from the border and edge device below (endpoints in question are 10.159.255.253 and 10.159.255.254):

BORDER#show lisp instance-id 4099 ipv4 database
LISP ETR IPv4 Mapping Database for EID-table vrf users_vn (IID 4099), LSBs: 0x1
Entries total 3, no-route 0, inactive 0

10.10.16.152/29, route-import, inherited from default locator-set rloc_52a23614-4a4a-4b5d-8db1-7b0fc4f718d9, auto-discover-rlocs
Locator Pri/Wgt Source State
10.3.254.255 10/10 cfg-intf site-self, reachable
10.128.0.0/11, route-import, inherited from default locator-set rloc_52a23614-4a4a-4b5d-8db1-7b0fc4f718d9, auto-discover-rlocs
Locator Pri/Wgt Source State
10.3.254.255 10/10 cfg-intf site-self, reachable
10.159.231.1/32, locator-set rloc_52a23614-4a4a-4b5d-8db1-7b0fc4f718d9, auto-discover-rlocs
Locator Pri/Wgt Source State
10.3.254.255 10/10 cfg-intf site-self, reachable

BORDER#show lisp site
LISP Site Registration Information
* = Some locators are down or unreachable
# = Some registrations are sourced by reliable transport

Site Name Last Up Who Last Inst EID Prefix
Register Registered ID
site_uci never no -- 4097 0.0.0.0/0
never no -- 4097 10.3.245.0/24
never no -- 4097 10.3.246.0/23
15:59:05 yes# 10.3.254.253:30531 4097 10.3.247.247/32
15:59:05 yes# 10.3.254.253:30531 4097 10.3.247.248/32
15:58:53 yes# 10.3.254.254:20727 4097 10.3.247.249/32
15:57:50 yes# 10.3.254.254:20727 4097 10.3.247.250/32
never no -- 4099 0.0.0.0/0
15:58:26 yes# 10.3.254.255:33699 4099 10.10.16.152/29
15:58:26 yes# 10.3.254.255:33699 4099 10.128.0.0/11
never no -- 4099 10.159.231.0/24
15:59:27 yes# 10.3.254.255:33699 4099 10.159.231.1/32
15:59:07 yes# 10.3.254.254:20727 4099 10.159.231.2/32
15:59:08 yes# 10.3.254.252:33077 4099 10.159.231.3/32
15:59:05 yes# 10.3.254.253:30531 4099 10.159.231.5/32
15:59:08 yes# 10.3.254.251:13886 4099 10.159.231.6/32
never no -- 4099 10.159.252.0/22
15:59:05 yes# 10.3.254.253:30531 4099 10.159.255.253/32 <----------(Endpoint in question)
15:58:57 yes# 10.3.254.252:33077 4099 10.159.255.254/32 <----------(Endpoint in question)

BORDER#show ip route vrf users_vn lisp

Routing Table: users_vn
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 10.101.1.34 to network 0.0.0.0

10.0.0.0/8 is variably subnetted, 24 subnets, 6 masks
l 10.159.231.2/32 [250/1], 16:01:37, Null0
l 10.159.231.3/32 [250/1], 16:01:38, Null0
l 10.159.231.5/32 [250/1], 16:01:36, Null0
l 10.159.231.6/32 [250/1], 16:01:38, Null0
l 10.159.255.253/32 [250/1], 16:01:36, Null0 <----------(Endpoint in question)
l 10.159.255.254/32 [250/1], 16:01:27, Null0 <----------(Endpoint in question)

EDGE#show lisp instance-id 4099 ipv4 map-cache 8.8.8.8
LISP IPv4 Mapping Cache for EID-table vrf users_vn (IID 4099), 11 entries

8.0.0.0/7, uptime: 16:08:18, expires: 00:13:18, via map-reply, forward-native
Sources: map-reply
State: forward-native, last modified: 16:08:18, map-source: 10.3.254.255 <----------(Border Loopback)
Active, Packets out: 3085(1776960 bytes) (~ 00:00:08 ago)
Encapsulating to proxy ETR

EDGE#show lisp instance-id 4099 ipv4 database
LISP ETR IPv4 Mapping Database for EID-table vrf users_vn (IID 4099), LSBs: 0x1
Entries total 5, no-route 0, inactive 3

10.159.231.5/32, locator-set rloc_3115b9ad-e5c3-4107-af56-d70bd25546ea
Locator Pri/Wgt Source State
10.3.254.253 10/10 cfg-intf site-self, reachable
10.159.251.253/32, Inactive, expires: 07:07:30
10.159.251.254/32, Inactive, expires: 05:11:35
10.159.255.253/32, dynamic-eid 10_159_252_0-users_vn-IPV4, inherited from default locator-set rloc_3115b9ad-e5c3-4107-af56-d70bd25546ea <----------(Endpoint in question)
Locator Pri/Wgt Source State
10.3.254.253 10/10 cfg-intf site-self, reachable
10.159.255.254/32, Inactive, expires: 07:40:22 <----------(Endpoint in question)

[JL421-Retired]

Out of curiosity, what version of IOS are you running, and do you have multicast enabled for your VN? We had a very similar sounding issue that was caused by CSCvo42353.

[Joshua Marks]

IOS-XE 16.12.4 on everything. We have Head-End Replication Multicast configured for both of the VNs with the Border as the RP. Did you implement the workaround mentioned in CSCvo42353? 

[Joshua Marks · Accepted solution]

After hours of going in circles, we have fixed out SD-Access external connectivity issue. I'll post it here because there isn't a lot about LISP troubleshooting specific to SD-Access.

I noticed on my edge switches that the summary route for the VN was in the map-cache:

EDGE#sh lisp instance-id 4099 ipv4 map-cache
LISP IPv4 Mapping Cache for EID-table vrf users_vn (IID 4099), 17 entries

0.0.0.0/0, uptime: 00:58:08, expires: never, via static-send-map-request
Encapsulating to proxy ETR
10.128.0.0/11, uptime: 00:18:02, expires: 23:42:25, via map-reply, complete
Locator Uptime State Pri/Wgt Encap-IID
10.3.254.255 00:18:02 up 10/10 -

This had me wondering how the summary route ended up in the map-cache, so I looked at the import database on the border and noticed that is was importing prefixes into LISP from BGP (as a good internal/anywhere border is supposed to):

BORDER#show lisp instance-id 4100 ipv4 route-import database
LISP IPv4 imported routes for EID-table vrf users_vn (IID 4099)
Config: 1, Entries: 2 (limit 5000)
Prefix Uptime Source RLOC-set Cache/DB State
10.10.16.152/29 21:48:17 bgp 65516 rloc_52a23 installed
10.128.0.0/11 21:48:17 bgp 65516 rloc_52a23 installed <----------------(This is the site summary route)

I removed the aggregate-address command from the BGP address-family which removed this prefix from the border LISP import database, then cleared the map-cache entry from each edge device with this command:

clear lisp instance-id 4099 ipv4 map-cache 10.128.0.0/11

I think having the VN summary route in the LISP map cache was causing return traffic to be blackholed in the fabric somewhere. We haven't had time to do a pcap, but I believe the edge devices may have been continually forwarding the return traffic back to the border, which would cause the traffic to be returned to the destination RLOC, and back to the border etc. Removing the summary route from the border and purging the prefix from all map caches fixed our external connectivity issue.

I noticed in the config that DNAC pushes to the border that it creates a DENY route map for each VN which contains all the IP pools belonging to the fabric. It then imports all routes from each BGP address family except the prefixes belonging to the fabric:

show run | s route-map
route-import database bgp 65516 route-map DENY-users_vn locator-set rloc_52a23614-4a4a-4b5d-8db1-7b0fc4f718d9
route-import database bgp 65516 route-map DENY-devices_vn locator-set rloc_52a23614-4a4a-4b5d-8db1-7b0fc4f718d9

Bit of a gotcha if you assume you can use route aggregation on your border without making changes to the LISP configuration.

[Jonathan Cuthbert]

---

This had me wondering how the summary route ended up in the map-cache, so I looked at the import database on the border and noticed that is was importing prefixes into LISP from BGP (as a good internal/anywhere border is supposed to):

---

Can I ask why you are using an anywhere border here?
We have seen that for most deployments (95%++), that external-only is generally the appropriate type.  

---

I removed the aggregate-address command from the BGP address-family which removed this prefix from the border LISP import database

---

To confirm, you manually removed configuration provisioned by Cisco DNA Center as part of your SD-Access workflow? This immediately puts you into unsupported territory.  Please don't manually intervene with any of the SD-Access provisioned configuration unless otherwise directed by TAC.  While altering this configuration may have (appeared to have) solved one problem, it is actually going to create another. It's just not clear where yet based on the information currently shared in this post. 

---

I noticed in the config that DNAC pushes to the border that it creates a DENY route map for each VN which contains all the IP pools belonging to the fabric. It then imports all routes from each BGP address family except the prefixes belonging to the fabric

---

The aggregate address command is used to, among other things, advertise the fabric prefixes to the external world. 

Do you have dual border nodes?  Did you create an manual BGP between them?

[Joshua Marks]

>Can I ask why you are using an anywhere border here?
We have seen that for most deployments (95%++), that external-only is generally the appropriate type.  

This line in the SDA CVD is what has led us to believe an external border with a default route would not work: 

WLC reachability—Connectivity to the WLC should be treated like reachability to the loopback addresses.  A default route in the underlay cannot be used by the APs to reach the WLCs.  A specific route (non-default route) to the WLC IP address must exist in the Global Routing Table at each switch where the APs are physically connected.  This can be a host route (/32) or summarized route. 

I may have misinterpreted this, but the need to import more specific prefixes (i.e. WLC) as well as a default route is the reason we have an anywhere border. It seems to me like external border would be the way to go in future.

>To confirm, you manually removed configuration provisioned by Cisco DNA Center as part of your SD-Access workflow? This immediately puts you into unsupported territory.  Please don't manually intervene with any of the SD-Access provisioned configuration unless otherwise directed by TAC.  While altering this configuration may have (appeared to have) solved one problem, it is actually going to create another. It's just not clear where yet based on the information currently shared in this post.  

No - we removed the aggregate-address for the whole VN (a /11 summary) which was configured via template to advertise one prefix for the entire VN rather than one per host pool. We've since noticed that DNAC creates an aggregate-address for each host pool anyway, so aggregating the whole on the border does not do much. We did this to be efficient with routing to the non-SDA parts of the network (this is a PoC network).

>Do you have dual border nodes?  Did you create an manual BGP between them?

We have a single border node which is why we looked at using an anywhere border.

Thanks for your comments, these have been quite helpful.