ARP table not updating to new MAC address

[ChuckHaynes]

Greetings,

We've been seeing a problem recently where a device's MAC address will get associcated with an IP address. After the device is removed, and another device given that IP address, nothing will be able to communicate with the new device. When you go into the router (router-on-a-stick), you will see that IP is still being associated with the MAC address from the old device. I have performed a clear arp and you can see the ARP time go to 0 (and sometimes see the entry totally disappear), however, it quickly returns. To my knowledge, I've cleared arp on all routers and layer 3 switches. I've also power cycled the new device several times and shut/no shut the switch port. This problem seems to have only started a couple months ago. At first, the fix was to wait (for days?) and then the ARP entry was finally released. The last time, I created a static ARP entry for the new MAC address to get it working. Does anyone have any idea why this could be happening and/or how to resolve it?

Thanks

[Dennis Mink]

Chuck,

is that router on a stick doing DHCP as well?

also, can you remember what changed a couple of months ago?

cheers

Please rate if useful

[ChuckHaynes]

The router is doing DHCP for the voice network, but the problem is occuring on the data network. A server at the corporate office (the center of our WAN) is where DHCP is being handled. I'm not aware of any changes to the network. We've had this issue once or twice at another location, but I'm not sure it that was just a fluke or if it's related. We've had it occur about five or six times at the location in question.

[Jigar Dave]

if static ip is defined on the device ( server), then that device continue talking with L3 device so L3 device end up keeping mac address on its arp table. so you need to check on that old device about static ip

[ChuckHaynes]

In these scenarios, the old device has been given a new (different) IP address, but it still has an ARP entry (confirmed by the MAC address) in the ARP table for its old IP address.

[lpassmore]

Nice problem Chuck.  I don't think I have ever seen this and a clear arp normally fixes the problem with the changing MAC address.

I guess you could do a debug arp and then clear the entry and see what you get.

Also maybe check how long the router has been on.  It may need a reboot if that is possible.

Are you running up against some bug?  Probably not since this is a very basic function but worth checking. 

How about reducing the ARP expiry time and see what happens.

If you have L3 switches that have interfaces with IP addresses in the same VLAN, are all the ARP entries the same? Maybe they are interfering with one another somehow?  Perhaps remove proxy arp especially on the switches.

What version router and switches?

[ChuckHaynes]

I will do a debug arp the next time I see the problem.

The router's uptime is currently 6 weeks. It has been rebooted since the problem first occured, but the issue remains.

What exactly are you suggestioning in reference to checking for a bug? Just looking at the release notes for the current loaded IOS?

I thought that I read where 4 hours was the default ARP timeout, but I'm thinking we have ours set at 5 hours for some reason.

This 'location' includes three seperate sites (each with its own router) that are connected via fiber and run Multigroup HSRP. Also, one of the sites has several (about 6-7) layer 3 switches that have interfaces with IP addresses, but the addresses are on a different VLAN. When I do a show arp, all the routers and switches show the same 'old' MAC address (even after a clear arp).

What exactly would removing proxy arp (if it's even enabled) do?

Routers are 2911's with 150-1.M2

Switches are (mostly) 3560's with 122-25.SEE2

[lpassmore]

Cisco has a bug search tool on their website you can use, but a quick look didn't find anything. I don't think I was really expecting to.

So trying to work out the issue..

I am a little unsure when you say 'but the addresses are on a different VLAN'.  I think we might need a diagram with IP addresses just to get a better picture. Do any of the switches have an IP address in the Data VLAN?  

You can see the ARP timeout by issuing a show interface command.

If they don't then they will not (or should not) have ARP entries for any address in the Data VLAN.  If they do then there is something seriously wrong.

So, assuming the switches do have an IP address in the Data VLAN, which device has the general 'default gateway' for the devices on the network?  This is probably the device that you need to be targeting, especially if it isn't the router.  Issuing a clear arp on that device should solve the problem. Doing it on any other device might just cause a gratuitous arp from the others which will update the tables with old information.

The proxy-arp feature is probably not the answer but it can be done on at the SVI level by the no ip proxy-arp command.  Be careful with this as having proxy-arp enabled can hide device mis-configurations in your network (no default-gateway, wrong subnet mask, etc).  If you disable proxy-arp, any such devices in the network might stop working.  However, it should really only be present on one device (normally the router) so maybe disable it on the switches if they have SVIs in the Data VLAN.

Still a good problem...

[ChuckHaynes]

Each of the three routers has about 8 different subinterfaces. One of the three sites has about 6-7 layer 3 switches. While those switches have interfaces with IP addresses in the 34 vlan, and SVI addresses in the 35 vlan, they are not in the same vlan (62) as the problem. All devices (with the exception of those layer 3 switches) use the router as their default gateway. The layer 3 switches are each their own small network and they use the next layer 3 switch upstream as their gateway.

[lpassmore]

OK. So if I am reading you correctly, VLAN 62 is the Data VLAN and none of the switches in the network have IP addresses (SVI) in this VLAN - only the routers do.
I am assuming you don't have static ARP entries for the devices that have problems.

If there are no other routers on the VLAN then let's wait until you can do a debug arp and see where the replies are coming from.

[ChuckHaynes]

Site 1 has VLAN 62 (data), VLAN 58 (monitoring), and some other VLANs (voice, wireless, etc.) Each of these VLANs contain switches with their SVIs in the 62 or 58 VLAN.

Site 2 has VLAN 53 (data), VLAN 58 (monitoring), and some other VLANs (voice, wireless, etc.) Each of these VLANs contain switches with their SVIs in the 53 or 58 VLAN.

Site 3 has VLAN 60 (data), VLAN 61 (monitoring), and some other VLANs (voice, wireless, etc.) Each of these VLANs contain switches with their SVIs in the 60 or 61 VLAN.- Plus this site also has about 6-7 layer 3 switches. The SVIs on these L3 switches are in the 34 and 35 VLANs, and the IP addresses on the interfaces of these L3 switches are also in the 35 VLANs. However, VLAN 35 isn't a normal (/24 for example) network. It's several small subnets.

[ChuckHaynes]

I think the fact that these three sites are connected via fiber from their main switches (the ones connected directly downstream from the router) complicates this scenario.

[lpassmore · Accepted solution]

Possibly not.  I think I can understand your network now.  A picture and configs would still be helpful; as would the following diagnostic information:

• a show interface from the data VLAN SVI;
• the results from your show arp;clear arp; show arp sequence;
• results from debug arp
• arp table from the new device

As long as each data VLAN has only one 'router' interface there should be no contamination from other devices sending ARP updates.  It is still very strange behaviour.  

[ChuckHaynes]

I was waiting until this issue happened again to provide the reults. It just started happening at another location (which doesn't have all the layer 3 switches and small subnets). I'm in meetings at the moment, but will provide additional information when I can.