ARP overwritten - IE3000 - 4000 switches

[bradleyordner]

Hi,

I have a funny issue that I need to bounce off some people to try and work out where else to troubleshoot. I have a large switching network and the default gateway of these switches is a Checkpoint firewall. Every now and again the default gateway arp is overwritten with another mac address for about 30 seconds and then changes back to the Checkpoint firewall. 

I have traced the ARP to a Network Video Recorder and escalated to them. 

The funny thing is, only 3 switches are doing this out of 10. They are different models and IOS. 

Debugs show the following, but i need more information, i am thinking a wireshark for sure. 

Arp changes - 

Dec 22 19:29:02.221: IP ARP ADJ: queueing to install adjacency for 10.29.83.254 on Vlan700
Dec 22 19:29:02.221: ARP DYNAMIC: set up subblock for 10.29.83.254 in tableid 0
Dec 22 19:29:02.221: IPARP_TRACK: ARP tableid(0) 10.29.83.254 Vlan700 mode(1) modified.
Dec 22 19:29:02.221: IP ARP ADJ: process adjacency event INSTALL for 10.29.83.254 on Vlan700, with conn id 0
Dec 22 19:29:02.221: IP ARP ADJ: install adjacency for 10.29.83.254 on Vlan700 with conn id 0

Arp returns - 

Dec 22 19:29:26.302: IP ARP ADJ: queueing to install adjacency for 10.29.83.254 on Vlan700
Dec 22 19:29:26.302: ARP DYNAMIC: set up subblock for 10.29.83.254 in tableid 0
Dec 22 19:29:26.302: IPARP_TRACK: ARP tableid(0) 10.29.83.254 Vlan700 mode(1) modified.
Dec 22 19:29:26.302: IP ARP ADJ: process adjacency event INSTALL for 10.29.83.254 on Vlan700, with conn id 0
Dec 22 19:29:26.302: IP ARP ADJ: install adjacency for 10.29.83.254 on Vlan700 with conn id 0

Has anyone seen this or recommend any further debugs? 

My theory is proxy arp, bad subnet mask or a bad NIC on the NVR. I have rulled out duplicate IP because I have run continious pings from the switches and the firewall doesnt respond, when the arp changes to the NVR the pings still dont reply. 

The other piece of info is the NVR is on the switch management VLAN as well. 

Any info would be great

Thanks

Brad

[Richard Burts]

Brad

Proxy arp sends a response to an arp request but it does not update the arp table. So I do not believe that your issue involves proxy arp. If the arp table is being updated with a different mac address for that IP then I suggest running debug arp (or perhaps debug ip arp depending on the platform). When the problem happens again then look in the debug output and verify if some other device is responding to arp requests for that address.

Are these switches operating as layer 2 switches? Or do some of them have ip routing enabled? If they are operating as layer 2 switches then they do maintain an arp table but it is only used for management traffic. So I would not be surprised that the majority of the switches do not change their arp table entry.

HTH

Rick

[bradleyordner]

Hi,

Thanks, I will take another debug as I was only debugging the gateway address. Most of the switches are Layer 2 only, the ones that are updating the arp are layer 2 as well. When they do that, i lose management access to them from outside the subnet. 

[Richard Burts]

A layer 2 switch does maintain an arp table to be used for its management interface. If a switch management interface attempted to send something to an address outside of the management subnet then it would need to use its gateway. If the switch sent an arp request and received a response with a different mac address then its arp table would have the incorrect address and you would lose remote access to the management interface (until the arp entry times out which by default would be 4 hours). So some switches (but not all switches) being impacted by this is probably to be expected.

HTH

Rick