Solved: Re: how does ACI iping work when I use pervasive gateway as source IP

Banyan2022 · ‎12-19-2022

May I know how does iping work when I use a pervasive gateway IP as source IP? I wonder how ACI know which leaf is the correct source node when forwarding ICMP reply.

For example, if a user have connection issue between endpoint A and endpoint B, I would use "iping -V test:test 192.168.223.20 -S 192.168.123.1" to test network between subnets.

endpoint A (192.168.123.10) on Leaf node 101 - EGP: VLAN123, BD: VLAN123, SVI: 192.168.123.1/24, Vrf: test:test

endpoint B (192.168.223.20) on Leaf node 131 - EGP: VLAN234, BD: VLAN234, SVI: 192.168.223.1/24, Vrf: test:test

Sergiu.Daniluk · ‎12-19-2022

Oh iping... indeed a very nicely crafted tool. The trick is how the leaf generates the data portion of ICMP request. To be more precise, the leaf which initiates the ping will add in the ICMP data portion it's own TEP IP address.

Regardless from which leaf you initiate the iping, the ICMP reply packet will always have the same DMAC and DIP of the anycast gateway, configured on all leafs. This means that local leaf (where server is connected) will always send the ICMP reply to the CPU and it will look into it's data portion:

if in data portion it finds it's own TEP IP, meaning you initiated the iping from local leaf, then you will see that PING reply with TTL, time etc.
if in data portion it finds a different TEP IP, it will send an ICMP Reply port unreachable to the corresponding leaf.

Let's take an example:

I have two leafs:

Leaf1201, TEP IP 10.0.104.64/32 (in hex: 0a.00.68.40)
Leaf1202, TEP IP 20.0.104.65/32 (in hex: 14.00.68.41)

1. When I ping from local leaf, you receive the ICMP reply and display the message:

Leaf1201# iping  10.0.0.45 -V TN:VRF -c 1
PING 10.0.0.45 (10.0.0.45) from 10.0.0.250: 56 data bytes
64 bytes from 10.0.0.45: icmp_seq=0 ttl=64 time=0.709 ms

data portion of ICMP request/reply, captured on the server:
tcpdump on local leaf:

09:21:20.469446 IP 10.0.0.250 > 10.0.0.45: ICMP echo request, id 18945, seq 0, length 64
07:08:23.964990 IP 10.0.0.45 > 10.0.0.250: ICMP echo reply, id 18945, seq 0, length 64

2. When I ping from remote leaf, we see the ICMP reply in remote leaf, but also we see it being received on local leaf in tcpdump:

Leaf1202# iping  10.0.0.45 -V TN:VRF -c 1
PING 10.0.0.45 (10.0.0.45) from 10.0.0.250: 56 data bytes
64 bytes from 10.0.0.45: icmp_seq=0 ttl=64 time=0.777 ms

ping initiated on remote leaf:

09:32:09.827562 IP 10.0.0.250 > 10.0.0.45: ICMP echo request, id 41217, seq 0, length 64

data portion of ICMP request/reply captured on the server:

icmp reply received on local leaf, then udp port unreachable sent to remote leaf:

07:08:25.142999 IP 10.0.0.45 > 10.0.0.250: ICMP echo reply, id 41217, seq 0, length 64
07:08:25.143050 IP Leaf1202 > Leaf1201: ICMP Leaf1202 udp port 65125 unreachable, length 120

port unreachable received on remote leaf and message displayed:

09:32:09.828265 IP Leaf1202 > Leaf1201: ICMP Leaf1202 udp port 65125 unreachable, length 120

Leaf1202# iping  10.0.0.45 -V TN:VRF -c 1
PING 10.0.0.45 (10.0.0.45) from 10.0.0.250: 56 data bytes
64 bytes from 10.0.0.45: icmp_seq=0 ttl=64 time=0.984 ms

I hope this was informative.

Take care,

Sergiu

View solution in original post

RedNectar · ‎12-19-2022

Hi @Banyan2022 ,

You raise a VERY interesting question - now I can give you some answers - but won't be able to explain exactly HOW until I get time to set up a SPAN and capture some packets to see what is actually going on. I've traced out the steps below to the point of where I believe you question is leading - hopefully I'll get time to get back to this and finish the answer (unless my friend @Sergiu.Daniluk can finish it for me )

So to answer your question

I wonder how ACI know which leaf is the correct source node when forwarding ICMP reply

The answer is, ACI will forward the reply to the leaf from which you sent the ping - but that is not the whole story.

For instance, in your example, if the SVI 192.168.123.1/24 DID NOT EXIST on the leaf from which you were issuing the iping, (say leaf101) you would get an error message along the lines of

Leaf101# iping -V test:test 192.168.223.20 -S 192.168.123.1
% Invalid source address
ping: %192.168.123.1 is an invalid Source. Use IP or full interface name without spaces (e.g. Ethernet1/1)

So your first line of troubleshooting is to make sure that the IP addresses you are using as source addresses exist on the leaf that you are issuing the iping from.

However, there is still a little conundrum. If we assume that the IP addresses 192.168.123.1 exists on both your Leaf101 and Leaf102, and endpoint 192.168.223.20 is attached to leaf 102, and you issue the iping from Leaf101

Leaf101# iping -V test:test 192.168.223.20 -S 192.168.123.1

This is what will happen.

Leaf101 will create an ICMP echo packet with SIP=192.168.123.1 and DIP=192.168.223.20
It will use it's local station table to see that endpoint 192.168.223.20 is located at TEP=Leaf101
It will create a VXLAN encapsulated packet with source VTEP=Leaf102 and destination VTEP=Leaf101 and send it to Leaf101
Leaf101 will get the VXLAN packet and decapsulate it, and see the encapsulated ICMP packet
Leaf101 will see that the destination IP is locally attached and send the ICMP ping packet to the MAC of 192.168.223.20 (it may have to ARP for this MAC first - if it does, it will use SIP=192.168.223.1)
The endpoint will get the ICMP ping and create an ICMP echo reply, SIP=192.168.223.20 DIP=192.168.123.1 and sent this to the MAC address of it's default gateway (192.168.223.1)

NOW: THIS IS WHERE THINGS GET TRICKY - AND WE GET TO THE CRUX OF YOUR QUESTION (I think)
Leaf101 will get the ICMP reply - and see its OWN IP address as the destination!

And this is as far as I can answer for the moment. One of the thoughts that crossed my mind was that PROBABLY subnets 192.168.123.0/24 and 192.168.223.0/24 are in different EPGs and have different VLAN encapsulations, but that is not necessarily true - the process would still work if they were in the same EPG and same VLAN encap.

Hopefully more to follow - but ATM I have lots of prep for a big family day next Sunday, so may not get back to this for a while!

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

Sergiu.Daniluk · ‎12-19-2022