ICMP redirect and ACI forwarding

[brian.holmes]

We are running 3.2 and seeing some strange forwarding that appears to be triggered from an external device sending ICMP redirect messages.

What does ACI do with ICMP Redirect messages?  Does it adjust the Coop or Endpoint forwarding as a reaction to them?

[Sergiu.Daniluk]

Hi @brian.holmes 

As far as I know, I think the leaf will simply ignore the received ICMP redirect msg.

Can you give more details about the behavior you are seeing, including a topology, source, destination, behavior before and after ICMP redirect is received, the more info the better.

Cheers,

Sergiu

[brian.holmes]

We are having periodic connectivity issues with an EPG.   When the problem occurs we see a host send out a ICMP redirect message then all of a sudden we see see traffic with the src MAC address of that host, and src IP of several other hosts which are off-net from this EPG. 

[Sergiu.Daniluk]

Hi @brian.holmes 

Still not enough details to understand the behavior. Let me guide you with some questions:

1. Where do you have the host gateway configured? Is it the ACI BD or is it an outside gateway?

2. Do you have routing enabled on the BD?

3. Where is the ICMP redirect sent (what's the DIP)?

4. What's the redirect gateway IP address?

5. What's the IP of the host/BD/affected flow?

Note1: if the IP addresses are public, change them to something else. If they are private, then well it's up to you if you want to share them as they are or not. If not, change the first two octets and it should be fine.

Note2: the reason I am asking for more details is because I do not want to make assumptions. You should make your conclusions based on evidence (Evidence Based Troubleshooting).

Cheers,

Sergiu

[brian.holmes]

1. Where do you have the host gateway configured? Is it the ACI BD or is it an outside gateway?

ACI BD

2. Do you have routing enabled on the BD?

Yes

3. Where is the ICMP redirect sent (what's the DIP)?

Still trying to get a good capture

4. What's the redirect gateway IP address?

Still trying to get a good capture

5. What's the IP of the host/BD/affected flow?

Devices on the other side of the L3Out cannot ping devices in the EPG periodically.  Comes and goes...sometime lasting like 10 min or so

We have two Cisco Routers sitting in the EPG used for Out-Of-Band management

When they are connected to the EPG, we get a bunch of messages like this

For Tenant PROD, application EPG EPG-xxx, ACI has detected multiple MACs using the same IP address xx.xx.xx.180. MACs: Context: 2752514. fvCEps: uni/tn-PROD_SHDC/ap-TIER_4_AP/epg-EPG-490/cep-08:94:EF:xx.xx.xx; uni/tn-PROD_SHDC/ap-TIER_4_AP/epg-EPG-490/cep-44:03:A7:xx.xx.xx;.

we have confirmed there is no duplicate IP address between the devices.  We see some ICMP redirect messages in the packet traces we have been able to get.

Periodically we cannot ping devices from outside the L3Out for like 10 minutes...then it comes back..very hard to catch.so far.

When we disconnect these Cisco Routers from the EPG, the dup IP messages stop and we have not seen the connectivity issue.

[brian.holmes · Accepted solution]

EPG-XXX is multinetted

EP-IBM-HMC sends out IP Subnet Broadcast messages on subnet A

EP-TSRXXX( a cisco ios router) - on subnet B picks up the broadcast and forward to spine-proxy/ default gateway

- when it forwards the broadcast, TSRXXX changes the src mac to itself but keeps the src IP as IBM HMC So now the Spine-Proxy is going to have bad info...thinking the IP for HMC should be sent to the MAC address of TSRXXX

to stop this we ended up adding a secondary IP on TSRXXX on subnet A...this stops TSRXXX from forwarding the broadcasts to the spine-proxy because it now has an IP on both subnets