PIX replying with wrong SRC MAC?

slug420 · ‎01-01-2006

Having a problem here on a 506 6.3(4) that the user says popped up out of nowhere one morning with no changes having been made to the config. There is a 506 a layer 2 dumb switch, and then 1 internal server hanging off this switch. the PIX can ping the server but the server cannot ping the switch.

lots of troubleshooting was done but most all of those steps only answered questions that are also answered by the following which is the result of a capture for all traffic between the inside IP of the pix and the IP of the server:

18:07:47.322065 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR: icmp: echo reply (ttl 255, id 10920)

18:07:48.322325 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR > 12.12.3.161: icmp: echo request (ttl 255, id 25094)

18:07:48.322371 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR: icmp: echo reply (ttl 255, id 10921)

18:07:49.322676 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR > 12.12.3.161: icmp: echo request (ttl 255, id 25095)

18:07:49.322737 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR: icmp: echo reply (ttl 255, id 10922)

18:07:50.323011 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR > 12.12.3.161: icmp: echo request (ttl 255, id 25096)

18:07:50.323072 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR: icmp: echo reply (ttl 255, id 10923)

18:08:40.136314 0003.e300.2851 0800.20c5.93f6 0x0800 135: 12.12.3.161.514 > THOR.514: [udp sum ok] udp 93 (ttl 255, id 10924)

18:09:56.289932 0003.e300.2851 0800.20c5.93f6 0x0800 74: 12.12.3.161 > THOR: icmp: echo request (ttl 255, id 10925)

18:09:56.290405 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR > 12.12.3.161: icmp: echo reply (DF) (ttl 255, id 25097)

18:09:57.288101 0003.e300.2851 0800.20c5.93f6 0x0800 74: 12.12.3.161 > THOR: icmp: echo request (ttl 255, id 10926)

18:09:57.288406 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR > 12.12.3.161: icmp: echo reply (DF) (ttl 255, id 25098)

18:09:58.288116 0003.e300.2851 0800.20c5.93f6 0x0800 74: 12.12.3.161 > THOR: icmp: echo request (ttl 255, id 10927)

18:09:58.288406 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR > 12.12.3.161: icmp: echo reply (DF) (ttl 255, id 25099)

18:09:59.288452 0003.e300.2851 0800.20c5.93f6 0x0800 135: 12.12.3.161.514 > THOR.514: [udp sum ok] udp 93 (ttl 255, id 10928)

as you can see, the top portion is where it was NOT working and th eproblem is the PIX is replying to these ARP requests with an ARP packet that has the server's mac as both the src and dst. On the bottom you can see a ping from the PIX to the server (which works) where the Macs are as they should be. This same behavior may be occuring on the outside interface but I cannot tell because I do not have access to theupstream router to initiate a ping.

Any ideas what could cause this?

(no input/output errors or collisions on interfaces, duplex settings are fine, debug packet and debug icmp trace show packets coming and going, when connected with just a hub in the middle the server can even see (in a tcpdump) the ICMP echo replies but the pings still fail (bogus SRC addresson the reply))

thanks

arunsing · ‎01-01-2006

Hi,

The pix is proxy arping for this ip address which is enable by default. You could try

sysopt noproxyarp inside

for details please go through the command reference

http://www.cisco.com/en/US/products/sw/secursw/ps2120/products_command_reference_chapter09186a00801cd841.html#wp1026942

Thanks

Arun

slug420 · ‎01-01-2006

I was aware of the noproxyarp command (thought of it but never actually tried it while t-shooting) but it didnt seem like that would be the problem.

Host is on the inside, IP of .162 pinging inside interface of firewall .161

Process would be ARP request from .162 asking who has .161, response from .161. Then the ICMP Echo request is sent, arrives at the firewall.

Firewall sends out an arp request "who has .162?" Which is where proxyarping would come into play if anywhere since .162 is a host behind the firewall with a static built in the firewall (static (inside, outside) x.x.x.x x.x.x.x netmask 255.255.255.255). So if proxyarping was the problem it would now allegedly use proxy arp to respond to its own ARP request (which doesnt sound plausible to begin with) in which case the problem would be that the src and dst macs would both be those of the firewall. That is not the case, the source and destination MACs are both those of the server....how could proxy arping change what the firewall uses as its own source mac in its replies?

sh arp in the firewall shows the server's IP associated with its correct mac

also, if proxyarp was the problem wouldnt it affect the packets similarly when the pix pinged the server? This problem does not exist when the pix pings the server, only when the server pings the pix (see logs above).

Can you explain how proxy arp is causing this?

slug420 · ‎01-03-2006

bump?

mheusinger · ‎01-03-2006

Hi,

as far as I can see there are no ARP packets in the output given. Looks just like a normal ping to me.

Can you please post the part which is NOT working?

Regards

Martin

slug420 · ‎01-03-2006

Sorry, the PING is what is not working (and all other connectivity to "THOR" through the firewall.

I misspoke when I said "ARP", I was trying to express that the problem with ICMP appears to be a layer 2 problem since the fw is responding with the destination mac as the src and dst mac in the packet.

slug420 · ‎01-03-2006

18:07:47.322065 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR: icmp: echo reply (ttl 255, id 10920)

18:07:48.322325 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR > 12.12.3.161: icmp: echo request (ttl 255, id 25094)

18:07:48.322371 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR: icmp: echo reply (ttl 255, id 10921)

In above Notice SRC MAC: 0800.20c5.93f6 and DST MAC 0800.20c5.93f6

18:07:49.322676 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR > 12.12.3.161: icmp: echo request (ttl 255, id 25095)

In above notice SRC MAC: 0800.20c5.93f6 abd DST MAC 0003.e300.2851.

how can a packet have the same src and dst mac address? If nothing else wouldnt that cause problems with the switch's cam table since it would see the incoming packet and say "oh my, that host is now on this interface (referring the the FW interface the ICMP reply just came in from)"

slug420 · ‎01-03-2006

any other ideas?

shouldnt the pix be incapable of generating these packets? What is a circumstance where it would be ok to have a packet with the same SRC and DST mac? How can it even generate this traffic?

mheusinger · ‎01-04-2006

Ok, now I got the problem.

This is really a puzzle.

I agree that those packets should not occur. Are you sure that these are the packets really sent? Could be an error in the debug output.

Can you check the CAM table in the switch? It should then show the server MAC at the PIX port, because the switch would learn it by the src MAC sent in the PIX reply. This would also explain why the ICMP reply doesn´t reach the server - the switch would not forward it to the server port.

But assuming the debug output shows what really happens, what I would do:

a) famous reload of PIX... (means being clueless, but optimistic)

b) IOS upgrade ... (standard TAC answer)

c) open a TAC case, because this is definately a software bug.

Hope this helps

Martin

slug420 · ‎01-05-2006

Reloaded it numerous times, did a write erase and reconfigured it also.

IOS upgrade is being tried as we speak (I think he was going to try that yesterday but it was already at 6.3(4) so im not sure what we will be able to gain)

has anyone else ever run into this?

slug420 · ‎01-09-2006

bump

slug420 · ‎01-10-2006

pix os upgrade to 6.3(5) didnt fix it either....this is officially unsolved.