Re: 90% of hosts on lan all have arp timed out

sschulak · ‎11-03-2008

does any one know what can sporadically cause the router to clear the arp table for some host but not all. this is just one of many VLANS that do this. I have other that 90% of the arp entries timeout at the same time. I do not have arp timeout configured, and it happens all any time, 1 min all the way to 15 minutes, there seems to be no connection. Im going to run a sniffer but wanted to see if any one has seen this before. I have only seen this with Firewall and grat-arp, but this isnt that

its hard to see in the post but these are enties are aged at 9 minutes

ustem05sw01#sho ip arp vlan 129

Protocol Address Age (min) Hardware Addr Type Interface

Internet 172.16.129.60 141 0011.252b.da58 ARPA Vlan129

Internet 172.16.129.63 9 001e.37d0.8116 ARPA Vlan129

Internet 172.16.129.62 9 001a.6b66.d498 ARPA Vlan129

Internet 172.16.129.56 9 001e.378a.c65e ARPA Vlan129

Internet 172.16.129.58 15 0016.41e5.ccde ARPA Vlan129

Internet 172.16.129.53 9 000d.6079.100f ARPA Vlan129

Internet 172.16.129.52 9 001a.6b67.09fa ARPA Vlan129

Internet 172.16.129.55 9 0016.d4a2.1e31 ARPA Vlan129

Internet 172.16.129.49 9 0011.2586.3412 ARPA Vlan129

Internet 172.16.129.51 9 0016.41a8.0b19 ARPA Vlan129

Internet 172.16.129.45 9 001b.383c.cae3 ARPA Vlan129

Internet 172.16.129.47 9 000d.602d.c0a8 ARPA Vlan129

Internet 172.16.129.41 9 0016.41a7.f95b ARPA Vlan129

Internet 172.16.129.40 140 0016.41a8.bdaa ARPA Vlan129

Internet 172.16.129.42 8 0011.252b.d3c0 ARPA Vlan129

Internet 172.16.129.37 8 0016.4159.fa9e ARPA Vlan129

Internet 172.16.129.36 9 0015.58c9.43a0 ARPA Vlan129

Internet 172.16.129.38 45 0016.d4b7.232e ARPA Vlan129

Internet 172.16.129.32 9 0011.25a2.c0b0 ARPA Vlan129

Internet 172.16.129.29 9 001a.6b6c.4c41 ARPA Vlan129

Internet 172.16.129.28 9 001a.6b3c.e578 ARPA Vlan129

Internet 172.16.129.31 9 0016.d4a2.1e17 ARPA Vlan129

Internet 172.16.129.30 9 001a.6b66.f319 ARPA Vlan129

Internet 172.16.129.25 9 001a.6b6c.215c ARPA Vlan129

Internet 172.16.129.27 9 0016.41a8.b667 ARPA Vlan129

Internet 172.16.129.26 4 0015.582e.3e2b ARPA Vlan129

Internet 172.16.129.20 9 0016.4159.e423 ARPA Vlan129

Internet 172.16.129.23 9 001a.6b6c.3d89 ARPA Vlan129

Internet 172.16.129.22 9 001e.37cc.95b8 ARPA Vlan129

Internet 172.16.129.16 9 0001.6cea.2335 ARPA Vlan129

Internet 172.16.129.18 9 001f.e214.5693 ARPA Vlan129

Internet 172.16.129.13 9 000d.602a.f9ad ARPA Vlan129

Internet 172.16.129.15 9 001c.2570.772c ARPA Vlan129

Internet 172.16.129.11 9 001e.ec24.ea47 ARPA Vlan129

Internet 172.16.129.10 9 0016.41e3.7975 ARPA Vlan129

Internet 172.16.129.5 9 001e.37d4.159a ARPA Vlan129

Internet 172.16.129.4 9 001e.ec7b.71f0 ARPA Vlan129

Internet 172.16.129.7 151 000d.608a.f9cf ARPA Vlan129

aghaznavi · ‎11-07-2008

When you do a "clear arp" on a Cisco router, 2 things happen

a) It sends a gratuitous arp out

b) It re-arps for every entry in its arp table

This tells you that 1 of 2 things is happening

(a) somebody upstream or downstream from us is losing router's arp entry or replacing it with something else [most likely]

(b) router is corrupting the arp entry it has for somebody else or replaced it with something else [less likely]

That's why we need to double check the arp table for the devices before/after clear arp has been issued.

santosd72 · ‎11-10-2008

Have you resolved this issue? because i am having the same issue at this moment!! and i am wondering what is going on.

sschulak · ‎11-10-2008

do a show tech and look for the when you see this command for all the modules.

remote command switch show plat hard asicreg PINNACLE slot 1 port 1 error-counters print-non-zero.

It will show this command for each port if its a 6500 and I have errors on about 5-7 ports per modules

--- remote command switch show plat hard asicreg COIL slot 4 port 37 error-counters print-non-zero ---

COIL 4/37

00CC: CO_PTX_S_ASSERT_FC = FFFF

--- remote command switch show plat hard asicreg COIL slot 4 port 38 error-counters print-non-zero ---

--- remote command switch show plat hard asicreg COIL slot 4 port 39 error-counters print-non-zero ---

COIL 4/39

016F: CO_PTX_S_CBL_DROP = 0003

I actualy just got off the phone with TAC and they recomended resetting the module 1 at a time. The other weird variable, this only happens 7-5 which leads me to believe its a bad NIC on a computer that some user either shuts down at 5:00 literally.

Go and configure an access port to be a trunk and plug your sniffer in so you can see all the spanning-tree tcn's if you use wireshark apply this filter "stp.flags.tc == 1" and it will only show spantree BPDU with Topo chg bit set which indicates a possible problem. you would then have to go and trace out each vlan manually to see where and what the topology change is.

let me know what you find, i have to wait a few days before i can reset my modules,