Gratuitous arp getting dropped on PIX 6.3.3

[cmchong77]

We have a cluster of devices behind the pix, sharing the same virtual IP. When there's a failover in the cluster, a gratuitous arp is sent with the new MAC for the device taking over the virtual IP. As packets need to be routed through the pix to the cluster- the pix needs to update it's arp cache. This is not happening after we upgraded to 6.3.3.

We're getting the following debug message on the pix:

145: arp-in: Dropping response at outside from unsolicited non-adjacent 160.96.XX

.XX 0002.b3c8.d0d2 for 255.255.255.255 ffff.ffff.ffff

When we were running 6.2.2- this used to work and the pix's arp cache was updated whenever a failover happened. Is there a workaround without having to downgrade?

Regards

Cheok Mun

Singapore

[jsivulka]

Did you try clearing the ARP? You could try this as a temperory fix atleast. You could refer to CSCed38053 ARP cache on neighbors may get corrupt during partial outage.

[cmchong77]

Hi,

Thanks for your response.

yes- clearing the ARP works. We do device failover tests every other day and in the middle of the night. We have an ops person doing it, but this procedure is prone to human error. The best thing would be the ARP table gets updated when the pix receives the gratuitous ARP.

We have tried 6.3.4 and the results are the same (problem exists). I could put a router in between the pix and cluster, but that would introduce another point of failure,

Regards

CM

[jmia]

Try adding:

sysopt noproxyarp inside

And see if this helps.

[cmchong77]

Tried it with the inside, and tried it with the outside- still the same...

Regards

CM

[shamilton-wilkes]

Did you ever resolve this?

I'm hitting the same thing under FWSM 2.2.1 code, and it worked fine under 1.1.3 and below.

[cmchong77]

It was due to the PIX's tightened security and our ARPs were not in the correct format. Our servers have since been patched to correct the ARP format and the failover works now, even with PIX 6.3.3-

According to Cisco the gratuitous arp that is sent by the BB (server) is not in the correct format (according to some RFCs).

Have a look at this arp decode (right at the bottom):

------------ ETHER Header ------------

ETHER: Destination: Broadcast (FF-FF-FF-FF-FF-FF)

ETHER: Source: 00-02-B3-C8-D0-D2

ETHER: Protocol: ARP

ETHER: FCS: 51A57C17

------------ ARP Header ------------

ARP: Hardware address format = 1 (Ethernet)

ARP: Protocol address format = 2048 (IP)

ARP: Hardware address length = 6

ARP: Protocol address length = 4

ARP: Operation = ARP Reply

ARP: Responder hardware address = 00-02-B3-C8-D0-D2

ARP: Responder protocol address = 160.96.88.88

ARP: Target hardware address = Broadcast

ARP: Target protocol address = All IP Stations (255.255.255.255)

[shamilton-wilkes]

Thanks there was a bug for this, which is marked resolved on FWSM 2.3.1, next change window I'll give that a try.

Hopefully my ARPs are the right format, I don't fancy my chances of getting Veritas to change anything...

[cmchong77]

Here's the ref given by a Cisco engineer to prove his point (anyway, we asked our developer who kindly fixed the grat ARP format in about 2 months)-

------

Hi CM,

You misread my email, I quote it here again.

sip - IP of itself,

smac - MAC of itself,

dip - IP of itself,

dmac - broadcast mac (ffff.ffff.ffff)

I am in 100% agreement that the DMAC is broadcast, but I am trying

to point out that the DIP is NOT 255.255.255.255 BUT SIP itself.

ref:

http://www.geocities.com/SiliconValley/Vista/8672/network/arp.html#A28

... In the ARP request packet, the source IP address and

destination IP address are filled with the same source IP address itself.

The

destination MAC address is the Ethernet broadcast address

(FF:FF:FF:FF:FF:FF).

[rparacchi]

I have the same problem with the fwsm 2.2.1 in High Availability in two catalyst 6500 box that have a CheckPoint cluster ( High availability)as secondary bastion firewall.

When the CP make a failover switch it broadcast his gratuitos arp but the fwsm do not update his arp table !

The configuration is in production and it become critical for the customer.

Any update /workaround/fix ?

TIA

Roberto

[cmchong77]

Well, we had a problem with the format of the grat ARP sent by our servers (which didn't work with the 6.3.3 ver PIX). Our application developers have since corrected the format and the failover works now.

Pls see the previous emails for more details of the problem (summary- in the grat ARP, the dest IP must be the source IP and the dest MAC is broadcast, etc).

Regards

CM

[shamilton-wilkes]

Yes, it's fixed in FWSM 2.3.1

Simon