2821 has to clear arp cache to regain connectivity to one or all devices on subnet

joshua.snider1 · ‎09-15-2015

Hi all,

We're having a problem with our Cisco 2821 router. As the title says, one of the local networks the router is connected to loses connection and we keep having to clear the arp cache to fix the problem.

More detail: so the router is connected to our main LAN (10.21.18.x), our management LAN (10.215.x.x), and a couple of other LANs (one is 10.21.19.x). Several (over 20) times a day, the router becomes unable to speak with a particular or all devices on the 10.215.x.x network. Other devices on the 10.215.x.x network can speak with eachother, however. Traffic between the 10.21.18.x and 10.21.19.x network are unaffected, it's just any traffic going to or originating from the 10.215.x.x network is unable to cross the router.

I eventually figured out that if I clear the arp cache on the 2821 (via command clear arp), the connection is restored.

The router still knows the mac address of a particular device on the 10.215.x.x network b/c if I run the command show arp | i [device IP], the destination mac address is the same before and after I run the clear arp command to restore connectivity.

Other steps I've taken:

Upgrading firmware from 12.4(15).T9 to 12.4(25g)

Replace HWIC card that 10.215.x.x & 10.21.19.x networks are on

Replace the whole 2821 router

Any ideas? And let me know if you need more info. Thanks!

Richard Bradfield · ‎09-15-2015

Hi Joshua,

the entry might have been in the ARP table but what was the age before you cleared it? I think devices will stay in the ARP table for a default of 4 hours before they are dropped. The problem could be the switch, I dont know what you network is like whether you have many switches. I have seen switches get stuck with saying a particular MAC is on one port (trunk to another switch) whereas it is really thru another port.

Anyway can you send me the relevent configuration, You have swapped a lot of things, but not changed the configuration? Also might be worth checking your switches.and worst case might be a device on the management Vlan causing the problem.

regards

Richard.

joshua.snider1 · ‎09-18-2015

Hey Richard,

Thanks for the reply! But I actually figured it out. I was doing a packet capture and happened to notice a highlighted field that told me that our there was a device with the same IP as the 2821. So that rogue device was responding to arp requests giving its mac address for that IP when it should have been the 2821.

We finally located the device, and will take it down soon. Weird thing is that this happened a few years back before I worked for the company-it's a linux video encoder-maybe some bug.

Anyway, thanks.

Josh