ACI routing issues

[mannycho]

Hello

i added a new leaf to a fabric, with no endpoints connected and I am unable to ping some external networks from new leaf. I check the routing table and there are routes to the destination I am unable to reach via the border leaf nodes. Existing leaf switches that are not border nodes have the same routes as new leaf switch I added and are able to reach the external network. 

Any ideas what’s preventing the new leaf switch from reaching some networks and how do I troubleshoot it? What is the trace route command to find the path on the VRF?

My understanding is that the border leaf inject external routes into BGP and the spines as RR send the networks to all leafs via BGP. A solution will be appreciated.

Thank you

[RedNectar]

hi @mannycho ,

Oh boy.  This is GREAT question, because it is such a good example of the thing least understood by ACI practitioners, especially those who are new to the idea.

The key to understanding your problem (BTW - you don't have a problem with ACI, your problem is a lack of understanding) is understanding the dynamic nature of ACI.

---

i added a new leaf to a fabric, with no endpoints connected

OK. Lets start there.  If a leaf has no endpoints connected you should not be expecting ANY routes other than those seen on VRF overlay-1

This is the whole idea of ACI. Routes and policies are instantiated (pushed) on the leaf switches on an "as needed" basis.  A new leaf switch has absolutely NO NEED to know any routes, because it will never need to deliver a packet.

and I am unable to ping some external networks from new leaf.

Nor should you be able to. Great news

I check the routing table and there are routes to the destination I am unable to reach via the border leaf nodes.

Then you are not telling us something. There should be NO routes if there are no endpoints - perhaps you have a L3Out connected on that leaf that you are hiding from us?  I think you better tell us the WHOLE story

Existing leaf switches that are not border nodes have the same routes as new leaf switch I added and are able to reach the external network. 

Again, I'm now certain that you have not given us a complete picture. Do you have a L3Out connected to this new leaf?

Any ideas what’s preventing the new leaf switch from reaching some networks and how do I troubleshoot it?

Give us ALL the information, including the output of the commands you are using (may sure you paste the output in preformatted style - see my tips pater.)

What is the trace route command to find the path on the VRF?

You may find both the iping and itraceroute   commands useful. These commands must be issued from a leaf switch. Here's a couple of examples.  Like you I have no endpoints either, but this should give you the idea.  Notice that both commands need the VRF specified, but withiping you use -V but with itraceroute  there is a vrf option.

Leaf1201# iping -V Tenant18:Production_VRF 10.118.12.11
PING 10.118.12.11 (10.118.12.11) from 10.118.12.1: 56 data bytes
...snip...
Leaf1201# itraceroute 10.118.10.10 vrf Tenant18:Production_VRF
Tenant traceroute to 10.118.10.10, tenant VRF Tenant18:Production_VRF, source encap vlan-1386, from [10.118.12.1], payload 56 bytes

My understanding is that the border leaf inject external routes into BGP and the spines as RR send the networks to all leafs via BGP. A solution will be appreciated.

Correct

RedNectar's Forum Tips:

• Click the camera icon then paste pictures in the grey area
• After clicking Preformatted, click HTML before pasting to preserve spacing
  

[mannycho]

Thanks Chris,

Love your response, you write like an ACI inspector, and I stand guilty of being a practitioner To answer your questions.

1. The new leaf has no endpoints connected and only fabric ports uplink. I checked from fabric inventory and all ports are down except fabric ports.

2. I need to clarify this been a new leaf, it is but with configuration. EPGs have static ports are mapped to new leaf with Deployment immediacy set to immediate. Could this be the reason the routes are seen on the new leaf?

3. The new leaf has no L3OUT connections, the L3OUT are on other leaf switches that are in production and have endpoints connected. I confirmed this from the L3OUT checking the node profiles.

4. I was using the show ip route vrf tenant:vrf to see the routes on the new leafs. I was using the iping -V tenant:vrf x.x.x.x to ping an IP address on the external network. I am able to ping some external IP address and others I cannot ping from new leaf.

I swear this is the truth on the life of my black cat that moved into our lives in 2020 and two years late moved next door for better food

[RedNectar · Accepted solution]

Hi @mannycho ,

My bad. I assumed "no endpoints connected" meant "no endpoints configured".

Let me first put to bed the idea that Deployment immediacy has anything to do with it. The point at which the configuration gets instantiated on (pushed to) the leaf is the moment you do the static mapping. Which explains why you see the routes on the leaf.

Deployment immediacy only determines when the policies are committed to TCAM - and you have to do some seriously ugly commands to figure that out, and will have no effect on the normal operation, especially for a switch that has no endpoints attached so clearly no TCAM issues.

---

It seems you are concerned with external routes - so I"m going to concentrate just on those for a moment.

So I'll go back to your original statement

"I am unable to ping some external networks from new leaf"

Does that mean you CAN ping some, but not others?  If that is the case, I'd be checking the source IP that is being used to do the iping - you can use the -S source.IP option with iping to be sure you are using an interface that the remote EP has a complete return path.

For fun I created this:

The static mapping on the FEX was configured, but the endpoint WebServer was left powered off.

On Leaf 1202 I could see the route to the external 10.110.10.0/24 subnet, plus the Loopback addresses and the addresses of the link between Leaf1201 and the external router.

Leaf1202# show ip route vrf Tenant10:Production_VRF
IP Route Table for VRF "Tenant10:Production_VRF"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF 

1.1.1.10/32, ubest/mbest: 1/0
    *via 10.1.184.66%overlay-1, [200/5], 00:06:59, bgp-65001, internal, tag 65001
10.110.0.201/32, ubest/mbest: 1/0
    *via 10.1.184.66%overlay-1, [1/0], 00:06:59, bgp-65001, internal, tag 65001
10.110.1.0/24, ubest/mbest: 1/0
    *via 10.1.184.66%overlay-1, [200/0], 00:06:59, bgp-65001, internal, tag 65001
10.110.10.0/24, ubest/mbest: 1/0
    *via 10.1.184.66%overlay-1, [200/44], 00:06:59, bgp-65001, internal, tag 65001
10.110.12.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.1.64.64%overlay-1, [1/0], 00:07:00, static
10.110.12.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.110.12.1, vlan19, [0/0], 00:07:00, local, local

Note the path to the external subnet is known, and was learned on this leaf via BGP (as you so correctly pointed out in your first post).

Note also, that the SVI interface (10.110.12.1/32) that will be used to send the ipings also exists 

But to be sure there is a complete path, lets check the external router

aciextrtr# show ip route 10.110.120.0 vrf OSPF-110
IP Route Table for VRF "OSPF-110"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF 

10.110.12.0/24, ubest/mbest: 1/0
    *via 10.110.1.201, Eth1/10.1501, [110/20], 00:00:03, ospf-fab1, type-2

And sure enough, the external router has a path back to the SVI address of 10.110.12.1/32

So now test iping

Leaf1202# iping -V Tenant10:Production_VRF -S 10.110.12.1 10.110.10.10
PING 10.110.10.10 (10.110.10.10) from 10.110.12.1: 56 data bytes
64 bytes from 10.110.10.10: icmp_seq=0 ttl=62 time=0.624 ms
64 bytes from 10.110.10.10: icmp_seq=1 ttl=62 time=0.353 ms
64 bytes from 10.110.10.10: icmp_seq=2 ttl=62 time=0.264 ms
64 bytes from 10.110.10.10: icmp_seq=3 ttl=62 time=0.399 ms
64 bytes from 10.110.10.10: icmp_seq=4 ttl=62 time=0.314 ms

--- 10.110.10.10 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.264/0.39/0.624 ms

 So as you can see, I've been unable to re-create your problem!

What's different about your setup to mine?

[mannycho]

Hello Chris,

I will do the same sourcing the ping from an IP address on the new leaf and confirming that the external router has path back to that IP address. Yes I am able to ping some external networks and others I cannot. I will check the routing. Thank you