topic Re: Cisco Catalyst Center SDA Transit Problem in Software-Defined Access (SD-Access)

Cisco Catalyst Center SDA Transit Problem

Marco Seiffert — Tue, 27 Feb 2024 12:02:20 GMT

Hi guys,

today I have a little problem with SDA-transit and I am not sure, if I am doing it wrong or DNAC (or CC). I have a customer and we want to migrate to a SDA fabric solution. The customer has different remote locations where we can use higher MTU connections to the main fabric, so the design idea was to use the SDA-transit to connect these remote sites to the main site. So every remote location should be it's own fabric and we want them all to communicate over the SDA-transit to preserve SGT's etc. At the main fabric there will be an IP-transit, that ultimately decapsulates any VXLAN headers to transfer packets into the rest of the non-fabric-world.

At the moment we're setting up a testing remote location and I am not able to get the SDA-transit to work, although there is not much, that you really can configure in the DNAC GUI. Border nodes can reach each other in the underlay, transit control plane nodes are up, but DNAC Assurance complains about missing Internet connectivity in the configured virtual network, because there is no default route in the corresponding vrf on the switch.

But if I understand the concept of SDA transit correctly, than that is by design? If traffic hits the switch and there is no route, the switch asks lisp where to send the packet. In case of SDA transit, it has to ask the transit control plane node and this one gives him the border switch with the configured IP transit, as a central exit point for any unknown traffic. Or do I miss something here? I am pretty new to lisp and I know a few troubleshooting commands but I am not always sure, which output to expect in this scenario. I added a little sketch as visual support.

Regards and thanks in advance to any hint...

Marco

Re: Cisco Catalyst Center SDA Transit Problem

jedolphi — Wed, 28 Feb 2024 07:25:09 GMT

Hi Marco, is it Pub/Sub SDA Transit? If yes then please review the following presentation and then ask any followup questions: https://www.ciscolive.com/on-demand/on-demand-library.html?#/session/1707505512189001p6lp . Best regards, Jerome

Re: Cisco Catalyst Center SDA Transit Problem

Torbjørn — Wed, 28 Feb 2024 09:08:58 GMT

Can you send the output of "show lisp site" from a SDA transit attached border?

Do you have a site that functions as a en egress site for non-fabric traffic? (Can be checked under SDA Border configuration for the SD transit, there should be an option "This site provides internet access to other sites through SD-Access.")

Re: Cisco Catalyst Center SDA Transit Problem

p11l — Fri, 01 Mar 2024 12:46:36 GMT

Hello together,

iam the customer who is Marco work for.

Some outpus from Border DNA-Transit(remote fabric), CP and Border-exit-DNA(9500-VSS) below.

Addresses

BN DNA-Transit: x.x.129.2

CP: x.x.0.1 and x.x.0.2

BN-exit-DNA with checked option "site provides internet access to other sites through SD-Access": x.x.129.1

I checked with "sh lisp instance-id 4100 ipv4 statistics | sec Map-Register" on BN and CP an see a line like this on BN DNA-Transit:
Map-Register records in/out: 62/207
Not valid site eid prefix: 11
and this on CP:
Map-Register records in/out: 58/0
Not valid site eid prefix: 0
Border-exit-DNA:
Map-Register records in/out: 140/231
Not valid site eid prefix: 9

Additional the failure messages on DNA GUI are:
IPv4 internet service on Fabric Border 'Border-exit-DNA' for Virtual Network 'VN' is unavailable on Transit Control Plane 'CP' since default route is lost.
IPv4 internet service on Fabric Border 'Border-exit-DNA' in Fabric Site 'exit-DNA' for Virtual Network 'VN' is unavailable on local Control Plane 'Border-exit-DNA' since default route is lost.

Thanks a lot for answers

Re: Cisco Catalyst Center SDA Transit Problem

jedolphi — Sat, 02 Mar 2024 03:04:08 GMT

Hello. You could open a TAC case if you want faster resolution of the problem, or if you are happy to troubleshoot over a forum then so am I.

These Assurance messages mean there is no default route in RIB on the Border Node for each flagged VRF. For the BN that connects to IP-Based Transit, can someone please confirm there 0.0.0.0/0 in BN RIB for each SDA VRF?

On the Control Plane Node in the Fabric Site related to the Assurance messages, please execute the following CLI and share the results: show lisp remote-locator-set default-etrs

Also please consider reviewing the suggested presentation, it explains how default-etr works, https://www.ciscolive.com/on-demand/on-demand-library.html?xd_co_f=YzUxYjE5NjEtNzI0OS00YWE1LWFmY2QtMTY0OWZiNWM1YzI2&#/session/1707505512189001p6lp

Best regards, Jerome

Re: Cisco Catalyst Center SDA Transit Problem

p11l — Wed, 13 Mar 2024 09:10:32 GMT

Hello Jerome,

Thanks for your help.

... For the BN that connects to IP-Based Transit, can someone please confirm there 0.0.0.0/0 in BN RIB for each SDA VRF?...

#sh ip route vrf VN

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

output from show lisp remote-locator-set default-etrs:

LISP remote-locator-set default-etr-locator-set-ipv4 Information

RLOC Pri/Wgt/Metric Inst Domain-ID/MH-ID ETR SI/ID
x.x.129.1 10/10 /0 4100 2212154865/52721 Default P /-

Re: Cisco Catalyst Center SDA Transit Problem

jedolphi — Wed, 13 Mar 2024 23:43:33 GMT

@p11l , the "show ip route vrf VN" output tells me there is no default route in RIB on the BN connected to IP Transit, thus Assurance is correctly flagging Internet access as down on this BN.

In addition, above it is stated that "BN DNA-Transit: x.x.129.2", but the output of "show lisp remote-locator-set default-etrs" is saying .x.129.1 has Internet access, NOT x.x.129.2.

Re: Cisco Catalyst Center SDA Transit Problem

p11l — Thu, 14 Mar 2024 14:35:52 GMT

I opened a case at our dealer, i will write the solution here if i get it

Re: Cisco Catalyst Center SDA Transit Problem

p11l — Thu, 14 Mar 2024 13:30:45 GMT

correct - 129.1 has Internet access, 129.2 is our remote location which is configured with SDA transit to the main site who as 129.1

Re: Cisco Catalyst Center SDA Transit Problem

p11l — Thu, 14 Mar 2024 15:39:03 GMT

I did some t-shoot.

On our remote location, connected via SDA Transit:

#show lisp instance-id 8212 ethernet server

=================================================
Output for router lisp 0 instance-id 8212
=================================================
LISP Site Registration Information
* = Some locators are down or unreachable
# = Some registrations are sourced by reliable transport

Site Name Last Up Who Last Inst EID Prefix
Register Registered ID
site_uci never no -- 8212 any-mac
04:28:30 yes# x.x.129.2:48123 8212 e070.ead6.f549/48

#sh lisp instance-id 8212 ethernet
Instance ID: 8212
Router-lisp ID: 0
Locator table: default
EID table: Vlan 2048
Ingress Tunnel Router (ITR): enabled
Egress Tunnel Router (ETR): enabled
Proxy-ITR Router (PITR): disabled
Proxy-ETR Router (PETR): disabled
NAT-traversal Router (NAT-RTR): disabled
Mobility First-Hop Router: disabled
Map Server (MS): enabled
Map Resolver (MR): enabled
Mr-use-petr: disabled
First-Packet pETR: disabled
Multiple IP per MAC support: disabled
Delegated Database Tree (DDT): disabled
Multicast Flood Access-Tunnel: disabled
Publication-Subscription: enabled
Publisher(s): *** NOT FOUND ***
Site Registration Limit: 0
Map-Request source: derived from EID destination
ITR Map-Resolver(s): x.x.129.2
ETR Map-Server(s): x.x.129.2 (never)
xTR-ID: 0x478BE804-0x9D2B8D1E-0x45D25727-0x43186701
site-ID: unspecified
ITR local RLOC (last resort): x.x.129.2
ITR Solicit Map Request (SMR): accept and process
Max SMRs per map-cache entry: 8 more specifics
Multiple SMR suppression time: 2 secs
ETR accept mapping data: disabled, verify disabled
ETR map-cache TTL: 1d00h
Locator Status Algorithms:
RLOC-probe algorithm: disabled
RLOC-probe on route change: N/A (periodic probing disabled)
RLOC-probe on member change: disabled
LSB reports: process
IPv4 RLOC minimum mask length: /1
IPv6 RLOC minimum mask length: /0
Map-cache:
Static mappings configured: 0
Map-cache size/limit: 0/32768
Imported route count/limit: 0/5000
Map-cache activity check period: 60 secs
Map-cache signal suppress: disabled
Conservative-allocation: disabled
Map-cache FIB updates: established
Persistent map-cache: disabled
Map-cache activity-tracking: enabled
Global Top Source locator configuration:
Loopback0 (x.x.129.2)
Database:
Total database mapping size: 3
static database size/limit: 0/32768
dynamic database size/limit: 3/32768
route-import database size/limit: 0/5000
import-site-reg database size/limit: 0/32768
dummy database size/limit: 0/32768
import-publication database size/limit: 0/32768
import-publication-cfg-prop database siz0
proxy database size: 0

#show lisp instance-id 4100 ipv4 database
LISP ETR IPv4 Mapping Database for LISP 0 EID-table vrf VN (IID 4100), LSBs: 0x1
Entries total 12, no-route 0, inactive 0, do-not-register 5

0.0.0.0/0, locator-set DEFAULT_ETR_LOCATOR, default-ETR
Uptime: 2w2d, Last-change: 2w2d
Domain-ID: local
Metric: 0
Service-Insertion: N/A
Locator Pri/Wgt Source State
x.x.129.2 10/10 cfg-intf site-self, reachable
x.x.48.0/24, locator-set rloc_df37203b-8903-4339-a23b-deac4f9a4b18, auto-discover-rlocs, proxy
Uptime: 05:15:31, Last-change: 05:15:31
Domain-ID: local
Service-Insertion: N/A
Locator Pri/Wgt Source State
x.x.129.2 10/10 cfg-intf site-self, reachable
x.x.48.1/32, dynamic-eid VN-IPV4, do not register, inherited from default locator-set rloc_df37203b-8903-4339-a23b-deac4f9a4b18, auto-discover-rlocs
Uptime: 2w2d, Last-change: 2w2d
Domain-ID: local
Service-Insertion: N/A
Locator Pri/Wgt Source State
x.x.129.2 10/10 cfg-intf site-self, reachable
x.x.48.20/32, dynamic-eid VN-IPV4, inherited from default locator-set rloc_df37203b-8903-4339-a23b-deac4f9a4b18, auto-discover-rlocs
Uptime: 04:29:48, Last-change: 04:29:48
Domain-ID: local
Service-Insertion: N/A
Locator Pri/Wgt Source State
x.x.129.2 10/10 cfg-intf site-self, reachable

On our SDA Transit ControlPlane, connected to main (c9500) and remote (c9300) site:

#sh lisp instance-id 8212 ethernet server
LISP Site Registration Information

#sh lisp instance-id 8212 ethernet
% Could not find EID table instance ID 8212 in LISP 0.

#sh lisp instance-id 4100 ipv4 database
% No local database entries configured.

Is that helpful for further t-shoot?

Tahnks!

Re: Cisco Catalyst Center SDA Transit Problem

jedolphi — Sat, 16 Mar 2024 22:47:07 GMT

Hi,
Just to reconfirm please:
A-We are troubleshooting Assurance reporting that Internet is down on 129.2
B-129.2 is connected to Pub/Sub SDA Transit and has no local Internet service
C-129.1 is connected to Pub/Sub SDA Transit and has default route in RIB + SDA Transit Internet sharing enabled

If yes, then on 129.2 please:
1- In Assurance, when you click on the issue, you should be able to navigate to "Suggested Actions" for troubleshooting steps, screenshot below from my lab. You could follow these steps.
2- On 129.2 you could issue the CLI show lisp instance-id * ipv4 map-cache 0.0.0.0/0

Regards, Jerome

Re: Cisco Catalyst Center SDA Transit Problem

p11l — Mon, 18 Mar 2024 14:53:47 GMT

Hello Jerome,

1- I ran the "suggested actions" - all look good.

#show lisp session all
Sessions for VRF default, total: 5, established: 4
Peer                           State      Up/Down        In/Out    Users
x.x.0.1:4342                Up         2d19h          41/15     4 OUR Transit ControlPlane 1
x.x.0.2:4342                Up         2d19h          37/11     4 OUR Transit ControlPlane 2
x.x.129.2                   Listening  never           0/0      0 local device
x.x.129.2:4342              Up         3d14h          64/37     14 
x.x.129.2:40069             Up         3d14h          37/64     10

#show lisp instance-id 4100 ipv4 publisher
LISP Publisher Information
Publisher                               State                          Session                       PubSub State        
x.x.0.1                              Reachable                      Up                            Established         
x.x.0.2                              Reachable                      Up                            Established         
x.x.129.2                            Reachable                      Up                            Established

#show lisp remote-locator-set default-etrs
Codes:
ETR = ETR Type (Default = Default-ETR, Service = Service-ETR)
SI  = Service Insertion Type
ID  = Service Insertion ID
-   = No service insertion config type defined
DS  = Default-ETR Firewall Service Insertion
SS  = Service-ETR Firewall Service Insertion
P   = Primary/Direct in use, Backup not available
PB  = Primary/Direct in use, Backup available
B   = Backup in use, Primary/Direct not available
BP  = Backup in use, Primary/Direct available
MT  = Multisite Service Insertion
 * = This locator has multiple service EID configured.

LISP remote-locator-set default-etr-locator-set-ipv4 Information

 RLOC          Pri/Wgt/Metric     Inst       Domain-ID/MH-ID  ETR       SI/ID          
 x.x.129.2   10/10 /0          4100       495636582/53350  Default   P /-

#sh lisp instance-id 4100 ipv4 database 0.0.0.0/0
LISP ETR IPv4 Mapping Database for LISP 0 EID-table vrf VN (IID 4100), LSBs: 0x1
Entries total 1, no-route 0, inactive 0, do-not-register 5

0.0.0.0/0, locator-set DEFAULT_ETR_LOCATOR, default-ETR
  Uptime: 2d21h, Last-change: 2d21h
  Domain-ID: local
  Metric: 0
  Service-Insertion: N/A
  Locator       Pri/Wgt  Source     State
  x.x.129.2   10/10   cfg-intf   site-self, reachable
  Map-server       Uptime         ACK  Domain-ID 
  x.x.129.2     2d21h          Yes  495636582

2- output from command:

sh lisp instance-id * ipv4 map-cache 0.0.0.0/0

=================================================
Output for router lisp 0 instance-id 4100
=================================================
% EID 0.0.0.0/0 not found in cache.

=================================================
Output for router lisp 0 instance-id 8212
=================================================
% EID table not enabled for IPv4.

Tahnks for your help!

Re: Cisco Catalyst Center SDA Transit Problem

jedolphi — Tue, 19 Mar 2024 07:13:29 GMT

OK thanks, this suggests no LISP mapping for 0/0 in user VRF on 129.2:

================================================= Output for router lisp 0 instance-id 4100 ================================================= % EID 0.0.0.0/0 not found in cache.

Something doesn't make sense here. Earlier we said 129.2 has no local internet (thus no 0/0 in user VRF RIB), but then we see this from 129.2 which says there is a local internet service as indicated by the "P":

#show lisp remote-locator-set default-etrs
Codes:
ETR = ETR Type (Default = Default-ETR, Service = Service-ETR)
SI  = Service Insertion Type
ID  = Service Insertion ID
-   = No service insertion config type defined
DS  = Default-ETR Firewall Service Insertion
SS  = Service-ETR Firewall Service Insertion
P   = Primary/Direct in use, Backup not available
PB  = Primary/Direct in use, Backup available
B   = Backup in use, Primary/Direct not available
BP  = Backup in use, Primary/Direct available
MT  = Multisite Service Insertion
 * = This locator has multiple service EID configured.

LISP remote-locator-set default-etr-locator-set-ipv4 Information

 RLOC          Pri/Wgt/Metric     Inst       Domain-ID/MH-ID  ETR       SI/ID          
 x.x.129.2   10/10 /0          4100       495636582/53350  Default   P /-

It's almost like someone has created a static route for 0/0 on 129.2 in the user VRF.

I think we need a TAC case or we need to take this offline, with diagrams, show runs, show vers, etc.

FYI this is what we should see on a BN that has no local internet access and is using remote internet over SD-Access Transit. Note the "B" which indicates we are seeking backup Internet over SD-Access Transit as opposed to a "P" which suggests there is local internet:

C-FIAB#show lisp remote-locator-set default-etrs Codes: ETR = ETR Type (Default = Default-ETR, Service = Service-ETR) SI = Service Insertion Type ID = Service Insertion ID - = No service insertion config type defined DS = Default-ETR Firewall Service Insertion SS = Service-ETR Firewall Service Insertion P = Primary/Direct in use, Backup not available PB = Primary/Direct in use, Backup available B = Backup in use, Primary/Direct not available BP = Backup in use, Primary/Direct available MT = Multisite Service Insertion * = This locator has multiple service EID configured. LISP remote-locator-set default-etr-locator-set-ipv4 Information RLOC Pri/Wgt/Metric Inst Domain-ID/MH-ID ETR SI/ID 192.168.8.53 10/10 /- 4099 3104824114/36304 Default B /- 192.168.8.53 10/10 /- 4101 3104824114/36304 Default B /- C-FIAB#

Re: Cisco Catalyst Center SDA Transit Problem

p11l — Fri, 22 Mar 2024 12:24:13 GMT

Hello Jerome,

our local dealer resolved the problem.

Between our sites we do static routing. From our Site 129.2 (which is connected via SDA Transit to 129.1, Main Site, with IP-Transit to external world) we have a default route 0.0.0.0 0.0.0.0 x.x.128.1

128.1 is on the same device which has the loopback address 129.1 (Main Site).

128.1 is the ip address of the interface who is connected to Site wich has the loopback 129.2.

The Device 129.2 has a physical address 128.2 which is connected do Main site 128.1.

On the Site with 129.2 we ca do a ping wich is successful to 129.1

The 129.2 Site has a default route to 128.1

The IP bindings via lisp from Site 129.2 to the TransitCP are all fine for default VN. But not for user created VN.

I do a few monitor sessions and saw that a ping from a user createt VN (Endpoint, PC) never leave the Edge Switch who the device is connected to.

I did a ping from another PC in the same VN which is connected via Main Site to Site connected via SD Transit. I saw packets came into Edge switch but never get back.

Solution:

The Edge Switch with loopback 129.2 need a dedicated host route to neighbour 129.1 like this:

#ip route x.x.129.1 255.255.255.255 x.x.128.1

[Remember, 128.1 has loopback0 with 129.1]

Thats all, the Client can communicate with each other and can leave via IP-Transit from Main Site the SDA Cloud.

Thanks for your help and have a nice weekend!

Output changed as expected:

#show lisp remote-locator-set default-etrs
Codes:
ETR = ETR Type (Default = Default-ETR, Service = Service-ETR)
SI = Service Insertion Type
ID = Service Insertion ID
- = No service insertion config type defined
DS = Default-ETR Firewall Service Insertion
SS = Service-ETR Firewall Service Insertion
P = Primary/Direct in use, Backup not available
PB = Primary/Direct in use, Backup available
B = Backup in use, Primary/Direct not available
BP = Backup in use, Primary/Direct available
MT = Multisite Service Insertion
* = This locator has multiple service EID configured.

LISP remote-locator-set default-etr-locator-set-ipv4 Information

RLOC Pri/Wgt/Metric Inst Domain-ID/MH-ID ETR SI/ID
x.x.129.2 10/10 /- 4100 2212154865/53350 Default B /-

#sh lisp instance-id 4100 ipv4 database 0.0.0.0/0
LISP ETR IPv4 Mapping Database for LISP 0 EID-table vrf VN (IID 4100), LSBs: 0x1
Entries total 1, no-route 1, inactive 0, do-not-register 5

0.0.0.0/0, locator-set DEFAULT_ETR_LOCATOR *** NO ROUTE TO EID PREFIX ***, default-ETR
Uptime: 20:12:16, Last-change: 03:29:53
Domain-ID: 2212154865
Metric: -
Service-Insertion: N/A
Locator Pri/Wgt Source State
x.x.129.2 10/10 cfg-intf site-self, reachable
Map-server Uptime ACK Domain-ID
x.x.129.2 03:29:53 Yes 495636582

#sh lisp instance-id * ipv4 map-cache 0.0.0.0/0

=================================================
Output for router lisp 0 instance-id 4100
=================================================
LISP IPv4 Mapping Cache for LISP 0 EID-table vrf VN (IID 4100), 1 entries

0.0.0.0/0, uptime: 00:00:18, expires: 00:00:42, via pub-sub, unknown-eid-forward, remote-to-site
Sources: pub-sub
State: unknown-eid-forward, last modified: 03:30:24, map-source: local
Exempt, Packets out: 89(67824 bytes), counters are not accurate (~ 00:18:13 ago)
Configured as EID address space
PETR Uptime State Pri/Wgt Encap-IID Domain-ID/MH-ID Metric
x.x.129.1 03:30:24 up 10/10 - 2212154865/52721 0

Re: Cisco Catalyst Center SDA Transit Problem

jedolphi — Sun, 24 Mar 2024 23:06:10 GMT

Well done! And thanks for the update. In BRKENS-2816 I list the underlay routing requirements within a fabric site and between fabric sites using SD-Access Transit, in short /32 is always best for fabric node reachabilitiy: https://www.ciscolive.com/on-demand/on-demand-library.html?xd_co_f=OTJlOTE5YzktNzYyNy00YWM1LTg1ZTMtZTA3MGNkNjA1NmU4&#/session/1707505512189001p6lp