Deny ICMP reverse path check from x.x.x.x to x.x.x.x on interface OOB

[pdub206]

Since about two weeks ago, we have been receiving reverse path notifications on our ASA for an ip address that doesn't exist on that interface.  I will call the culprit host 10.2.3.4 and the interface it lives on the mgmt interface (10.2.3.0/24), while the reverse path notices are coming in on the oob interface (172.16.0.0/24). 

"Deny ICMP reverse path check from 10.2.3.4 to 10.2.3.4 on interface oob"

The above is the error we are seeing.  Now I understand uRPF and how it functions -- if there isn't a path back to the destination network from that interface, it considers it not legitimate and drops it.  The oob interface has no ARP entry for the 10.2.3.4 host and I can't ping the culprit host sourced from it, so it appears that it isn't actually there or coming from the oob interface.  I can ping 10.2.3.4 from the mgmt interface and I have verified that it is indeed on the correct vlan, so I don't believe the traffic is coming from it (also why would it be trying to ping itself?).

My first thought is there is some spoofing going on, or one of the devices on that interface is owned.  My questions for you all are:

1. How can I identify the location of the spoofed traffic, if it is indeed spoofed? Would a SPAN session w/ Wireshark help to determine where it is coming from?

2. Let's say the oob network traverses several routers into a much larger network which is layer 3 isolated in a vrf.  Assuming that the culprit host is in that network, how likely would it be that the other hosts are owned as well?

Thank you for any input, it's driving me nuts seeing this error and I don't quite understand why it's happening.

[Julio Carvajal]

Hello,

Can you check the NAT table for this host IP address ( or even it's subnets) I have seen this problems due to a NAT misconfiguration

Regards.

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2-CCNP, JNCIS-SEC
For inmediate assistance hire us at http://i-networks.us

[pdub206]

Hi Julio,

The ip that is showing up as the source/destination is actually a public ip, not a private one.  What is odd is that it is showing up on a private interface.  I hope that clarifies some things.

I did look at the nat translations, and it appears this firewall was set up without any sort of nat whatsoever.  "sh run | inc nat" yields no nat configuration, and neither does "show nat detail".

[Julio Carvajal]

Hello,

So if you do show run nat you do not get anything?

Question:

-Why are the packets being forwarded on the ASA through the  oob  interface if they do not exist there?

[pdub206]

Nope, nothing comes from that.

The management device is on the management nameif interface, and polls various other devices in different nameif interfaces.  The issue appears to be the return ICMP packet not having its source/destination IP's rewritten. 

I've got a TAC case open and noticed that the same software is being run on all three switches that exhibit this problem.  I'm assuming that that is the issue as we have several other switches of the same model running different software without any issues.

[Julio Carvajal]

Hello,

Lets say the host at MGMT is 1.1.1.1 and the one in OOB is 2.2.2.2

Okey so the packet goes from the MGMT interface to the OOB, and when the packet comes back it looks like ICMP echo reply from 2.2.2.2 to 2.2.2.2

Is that right

[pdub206]

The packet indicates ICMP source is 1.1.1.1 and destination is 1.1.1.1 coming in on the OOB interface.  The remote switch host at 2.2.2.2 is not even mentioned at layer 3.

[pdub206]

Hi Julio,

After doing a wireshark capture w/ SPAN on the switch that connects the interface we are seeing these uRPF log messages on, I was able to see several of the ICMP frames with identical source/destination.

The frame source MAC address changes, however.  It appears to be coming from several of our switches! So at this point I am thinking that the following situation is happening:

ICMP packet destined to a remote switch is being forwarded through the ASA, and upon reaching destination, the switch is not re-writing the packet correctly, leaving the source/destination IP's intact.  Since the ASA sees the source as from the original source coming in on an interface it couldn't possibly come in on, it triggers the uRPF check and drops the packet.

I'm going to open a TAC case to see if there may be a software bug in the switches this is happening with -- they are all the same model.