Endpoint intermittent reachability issue

ChandanG · ‎06-16-2022

Why an endpoint has intermittent reachability issue from access Leaf-Pair (where source IP is attached), while at the same time, the same endpoint is continuously reachable from Border leaf-pair?
Source IP is attached to VPC interface in LEAF-117 & LEAF-118.
Destination IP is attached to VPC interface in LEAF-103 & LEA-104.
Border Leaf pair - LEAF-101 & LEAF-102.
Affected destination endpoint - 10.186.90.15

Below logs capture at the same time.

===============================================================
Destination endpoint is continuously pinging from Border leaf switch - LEAF-101
===============================================================
leaf101# iping -V Production:Production-VRF 10.186.90.15 -S 10.184.53.1
PING 10.186.90.15 (10.186.90.15) from 10.184.53.1: 56 data bytes
64 bytes from 10.186.90.15: icmp_seq=0 ttl=128 time=0.713 ms
64 bytes from 10.186.90.15: icmp_seq=1 ttl=128 time=0.408 ms
64 bytes from 10.186.90.15: icmp_seq=2 ttl=128 time=0.497 ms
64 bytes from 10.186.90.15: icmp_seq=3 ttl=128 time=0.411 ms
64 bytes from 10.186.90.15: icmp_seq=4 ttl=128 time=0.497 ms

leaf101# show system internal epm endpoint ip 10.186.90.15

MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 10.186.90.15 ::: IP# 0 flags : ::: l3-sw-hit: No
Vlan id : 0 ::: Vlan vnid : 0 ::: VRF name : Production:Production-VRF
BD vnid : 0 ::: VRF vnid : 2686976
Phy If : 0 ::: Tunnel If : 0x18010009
Interface : Tunnel9
Flags : 0x80004400 ::: sclass : 32792 ::: Ref count : 3
EP Create Timestamp : 05/28/2022 18:38:46.869607
EP Update Timestamp : 06/16/2022 19:34:16.571792
EP Flags : IP|sclass|timer|

::::

leaf101# show clock
19:36:45.283662 BST Thu Jun 16 2022
leaf101#

=======================================================================================
Destination endpoint has intermittent ping issue from Access Leaf Switches (where source IP - 10.184.53.31 is attached)
=======================================================================================
leaf117# show clockiping -V Production:Production-VRF 10.186.90.15 -S 10.184.53.1 -c 100
PING 10.186.90.15 (10.186.90.15) from 10.184.53.1: 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
---output omitted---
Request 10 timed out
Request 11 timed out
Request 12 timed out
Request 13 timed out
Request 14 timed out
^C
--- 10.186.90.15 ping statistics ---
16 packets transmitted, 0 packets received, 100.00% packet loss
leaf117#
leaf117# show system internal epm endpoint ip 10.186.90.15

leaf117# show clock
19:36:21.983926 BST Thu Jun 16 2022
leaf117#

leaf117# iping -V Production:Production-VRF 10.186.90.15 -S 10.184.53.1 -c 100
PING 10.186.90.15 (10.186.90.15) from 10.184.53.1: 56 data bytes
64 bytes from 10.186.90.15: icmp_seq=0 ttl=128 time=0.589 ms
64 bytes from 10.186.90.15: icmp_seq=1 ttl=128 time=0.418 ms
64 bytes from 10.186.90.15: icmp_seq=2 ttl=128 time=0.396 ms
64 bytes from 10.186.90.15: icmp_seq=3 ttl=128 time=0.506 ms
64 bytes from 10.186.90.15: icmp_seq=4 ttl=128 time=0.512 ms
64 bytes from 10.186.90.15: icmp_seq=5 ttl=128 time=0.5 ms
64 bytes from 10.186.90.15: icmp_seq=6 ttl=128 time=0.571 ms
64 bytes from 10.186.90.15: icmp_seq=7 ttl=128 time=0.429 ms
64 bytes from 10.186.90.15: icmp_seq=8 ttl=128 time=0.54 ms
64 bytes from 10.186.90.15: icmp_seq=9 ttl=128 time=0.456 ms
^C
--- 10.186.90.15 ping statistics ---
10 packets transmitted, 10 packets received, 0.00% packet loss
round-trip min/avg/max = 0.396/0.491/0.589 ms
leaf117#
leaf117# show system internal epm endpoint ip 10.186.90.15

MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 10.186.90.15 ::: IP# 0 flags : ::: l3-sw-hit: No
Vlan id : 0 ::: Vlan vnid : 0 ::: VRF name : Production:Production-VRF
BD vnid : 0 ::: VRF vnid : 2686976
Phy If : 0 ::: Tunnel If : 0x1801000c
Interface : Tunnel12
Flags : 0x80004400 ::: sclass : 32792 ::: Ref count : 3
EP Create Timestamp : 06/16/2022 19:43:35.232424
EP Update Timestamp : 06/16/2022 19:49:22.415795
EP Flags : IP|sclass|timer|

::::

leaf117#

================================================================================
Endpoint learning status at Leaf Switches where destination endpoint is attached
================================================================================
leaf103# show system internal epm endpoint ip 10.186.90.15

MAC : 82e3.9220.00a3 ::: Num IPs : 3
IP# 0 : 10.186.90.21 ::: IP# 0 flags : ::: l3-sw-hit: No
IP# 1 : 10.186.90.20 ::: IP# 1 flags : ::: l3-sw-hit: No
IP# 2 : 10.186.90.15 ::: IP# 2 flags : ::: l3-sw-hit: No
Vlan id : 159 ::: Vlan vnid : 22010 ::: VRF name : Production:Production-VRF
BD vnid : 15531941 ::: VRF vnid : 2686976
Phy If : 0x16000005 ::: Tunnel If : 0
Interface : port-channel6
Flags : 0x80004c05 ::: sclass : 32792 ::: Ref count : 7
EP Create Timestamp : 09/03/2021 23:42:25.897680
EP Update Timestamp : 06/16/2022 19:38:03.940231
EP Flags : local|vPC|IP|MAC|sclass|timer|

::::

leaf103# show clock
19:36:24.751502 BST Thu Jun 16 2022
leaf103#

leaf103# show clocksystem internal epm endpoint ip 10.186.90.15

MAC : 82e3.9220.00a3 ::: Num IPs : 4
IP# 0 : 10.186.90.21 ::: IP# 0 flags : ::: l3-sw-hit: No
IP# 1 : 10.186.90.20 ::: IP# 1 flags : ::: l3-sw-hit: No
IP# 2 : 10.186.90.15 ::: IP# 2 flags : ::: l3-sw-hit: No
IP# 3 : 10.186.90.37 ::: IP# 3 flags : ::: l3-sw-hit: Yes ::: flags2 : inh-sclass|tentative|del-bounce|ptp|
Vlan id : 159 ::: Vlan vnid : 22010 ::: VRF name : Production:Production-VRF
BD vnid : 15531941 ::: VRF vnid : 2686976
Phy If : 0x16000005 ::: Tunnel If : 0
Interface : port-channel6
Flags : 0x80004c05 ::: sclass : 32792 ::: Ref count : 8
EP Create Timestamp : 09/03/2021 23:42:25.897680
EP Update Timestamp : 06/16/2022 19:41:13.839135
EP Flags : local|vPC|IP|MAC|sclass|timer|

::::

leaf103# show system internal epm endpoint ip 10.186.90.15

MAC : 82e3.9220.00a2 ::: Num IPs : 3
IP# 0 : 10.186.90.15 ::: IP# 0 flags : ::: l3-sw-hit: No
IP# 1 : 10.186.90.14 ::: IP# 1 flags : ::: l3-sw-hit: No
IP# 2 : 10.186.90.37 ::: IP# 2 flags : ::: l3-sw-hit: Yes ::: flags2 : inh-sclass|tentative|del-bounce|ptp|
Vlan id : 159 ::: Vlan vnid : 22010 ::: VRF name : Production:Production-VRF
BD vnid : 15531941 ::: VRF vnid : 2686976
Phy If : 0x16000005 ::: Tunnel If : 0
Interface : port-channel6
Flags : 0x80004c05 ::: sclass : 32792 ::: Ref count : 7
EP Create Timestamp : 09/03/2021 23:42:27.492602
EP Update Timestamp : 06/16/2022 19:41:38.540251
EP Flags : local|vPC|IP|MAC|sclass|timer|

::::

========================================
COOP status in SPINE during issue:
========================================
spine901# show coop internal info ip-db key 2686976 10.186.90.15

IP address : 10.186.90.15
Vrf : 2686976
Flags : 0
EP bd vnid : 15531941
EP mac : 82:E3:92:20:00:A3
Publisher Id : 10.199.16.71
Record timestamp : 06 16 2022 19:29:35 943008714
Publish timestamp : 06 16 2022 19:29:35 943459579
Seq No: 0
Remote publish timestamp: 01 01 1970 01:00:00 0
URIB Tunnel Info
Num tunnels : 0

spine901# show clock
19:36:34.333573 BST Thu Jun 16 2022

=================================================
COOP status in SPINE, when there is no issue:
=================================================
spine901# show coop internal info ip-db key 2686976 10.186.90.15

IP address : 10.186.90.15
Vrf : 2686976
Flags : 0
EP bd vnid : 15531941
EP mac : 82:E3:92:20:00:A2
Publisher Id : 10.199.16.71
Record timestamp : 06 16 2022 19:42:31 875096469
Publish timestamp : 06 16 2022 19:42:31 875251309
Seq No: 0
Remote publish timestamp: 01 01 1970 01:00:00 0
URIB Tunnel Info
Num tunnels : 1
Tunnel address : 10.199.200.97
Tunnel ref count : 1

spine901#

Sergiu.Daniluk · ‎06-16-2022

Hi @ChandanG

To me it looks like the IP address is flapping between two different MACs:

leaf103# show system internal epm endpoint ip 10.186.90.15

MAC : 82e3.9220.00a3 ::: Num IPs : 4
IP# 2 : 10.186.90.15 ::: IP# 2 flags : ::: l3-sw-hit: No

-------------

leaf103# show system internal epm endpoint ip 10.186.90.15

MAC : 82e3.9220.00a2 ::: Num IPs : 3
IP# 0 : 10.186.90.15 ::: IP# 0 flags : ::: l3-sw-hit: No

I would suggest to fix this first and see if the issues persists after.

Cheers,

Sergiu

ChandanG · ‎06-16-2022

Thanks @Sergiu.Daniluk for checking.

Yes, you are right and this has been highlighted to affected customer. They are using "Switch Independent - dynamic load balancing" teaming at the server end (HyperV cluster). But their counter question to us, is why at the same time, we are not observing ping drops issue, if traffic is sourced from outside of the ACI fabric.

We also observed that we are getting continuous ping from border Leaf switches, but not from the access leaf switches where the client IPs are attached.

Sergiu.Daniluk · ‎06-17-2022

Can you check if you have "Disable remote EP learning" activated for border leafs (System > System Settings > Fabric Wide Setting).

If yes, could be because of this, the ping from BL works w/o interruption.

Stay safe,

Sergiu

ChandanG · ‎06-17-2022

Hi
"Disable Remote Endpoint Learning" is not enabled/checked in our setup.
Still border leaf is not loosing the endpoint and can continuously reach the destination. Not sure why it is behaving like this. Any other possible root cause?
[cid:image001.png@01D8829E.029A7430]

ChandanG · ‎06-17-2022