ACI Issue - Faulty route in ACI fabric cause connectivity issue

[Nilay Patel]

Problem: Some time Ping/Communications stops working for certain destination(EP or IP) from some source(its random). Source or destination can be out-side or in-side of fabric(true for any direction)

Looking for proper solution, 

Situation as e.g.

• We updated the ACI code multiple time some issue solution.
• Current fabric version Version: 3.0(2n)

This IP(192.168.10.78) located outside of ACI fabric - we have seen same issue with internal ACI EP

Leaf-23# show endpoint ip 192.168.10.78 detail

Legend:

s - arp O - peer-attached a - local-aged S - static

V - vpc-attached p - peer-aged M - span L - local

B - bounce H - vtep

+-----------------------------------+---------------+-----------------+--------------+-------------+------------------------------+

VLAN/ Encap MAC Address MAC Info/ Interface Endpoint Group

Domain VLAN IP Address IP Info Info

+-----------------------------------+---------------+-----------------+--------------+-------------+------------------------------+

TENANT_K:VRF_K 192.168.10.78 tunnel17

Leaf-23# show int tunnel 17

Tunnel17 is up

MTU 9000 bytes, BW 0 Kbit

Transport protocol is in VRF "overlay-1"

Tunnel protocol/transport is ivxlan

Tunnel source 192.168.104.84/32 (lo0)

Tunnel destination 192.168.192.67

Last clearing of "show interface" counters never

Tx

0 packets output, 1 minute output rate 0 packets/sec

Rx

0 packets input, 1 minute input rate 0 packets/sec

###

• You can see this tunnel destination going to Leaf 25-26 instated of boarder leaf
• From Leaf 23, tunnel entry pointing incorrectly towards Leafs 25-26.

Leaf-23# show isis dteps vrf overlay-1

IS-IS Dynamic Tunnel End Point (DTEP) database:

DTEP-Address Role Encapsulation Type

192.168.192.67 LEAF N/A PHYSICAL (@@@@ Its Virtual IP of VPC pair Leaf25-26)

On Leaf 25-26 No EP found

Leaf-25# show endpoint ip 192.168.10.78

Legend:

s - arp O - peer-attached a - local-aged S - static

V - vpc-attached p - peer-aged M - span L - local

B - bounce H - vtep

+-----------------------------------+---------------+-----------------+--------------+-------------+

VLAN/ Encap MAC Address MAC Info/ Interface

Domain VLAN IP Address IP Info

+-----------------------------------+---------------+-----------------+--------------+-------------+

ROB-Leaf-105# show time

Only temporary solution: 

Leaf-23# clear system internal epm endpoint key vrf TENANT_K:VRF_K ip 192.168.10.78

[micgarc2]

I would open a TAC case so we can take a look this but looks like this learn shouldn't be learned anywhere. If this is actually an external EP the BL(s) nor compute leaf(s) should learn that as an EP. We aren't suppose to learn external IP addresses period. Are these GEN2 or GEN1 switches? 

If you take a look at this document you can see in what scenarios this can happen and also features to prevent it. Do you have Limit IP learning to subnet enabled on the internal EPs BD? Or Global Enforce Subnet Check?

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html

Thanks,

Michael G

[Nilay Patel]

Thanks Michael,

I opened multiple cases for this issue. Believe or not some time creating big issue in Data canter even one or two IP had issue. We have to clear the entry manually. It’s gen 2 switches. Like I mentioned it doesn’t matter, we faced the issue with destination and source IP-EP being inside the fabric. I don’t mind to open the call multiple time. I will try it one more time. 

I will reply about limiting EP setting. I am sure we have setup. 

[alieson]

Hi Nilay,

Can you see any mac flapping under the EPG level ?

Go to destination EPG--operational -- then check the ip and mac address , are they flapping between different ports of the fabric ?

[alieson]

And check if any (for example ESXI chassis) connected to leaf 23 is using ip address 192.168.10.78 causing duplication of ip address. since there is no reasonable to learn ip address from specific leaf till packet has received with header contains this ip address as a source ip.!