Cisco 6509 - Funny ARP Table

[GR5516]

Hello all,

Having issues from users throughout the network in recent days, with apparent slowness.

Our main L3 core is a Cisco 6509 with Sup32, 12.2(18)SXF11, and processor usage is anything between 60-90%.

Processes showing high are ios-base and tcp.proc, and from ios-base, arp is at over 30%.

One of the things that is bothering be is that wne I do a sh ip arp summary I get

2540 addresses, 29 incomplete

and if i immediately do the command again, might get 2654 addresses, 124 incomplete, and this continues.

There is a huge amount of ARP on the network, so its possible there is a loop. I have done a scan of the largest VLAN, and while ARP is pretty high, I cannot see any evidence of an ARP attack, or indeed a scan.

Has anybody any ideas on where to look to address this, or seen it before.

Thanks very much in advance.

Regards

[balaji.bandi]

Can you post more information

show process cpu | ex 0.00%

show proces cpu history 

is IP CEF enabled ?

[GR5516]

Hi,

Additional information as requested:-

[Richard Burts]

Thanks for the additional information. I do not see anything in it that suggests that the slowness of the network is related to arp processing. And I do not see anything in the output that suggests problems with arp.

HTH

Rick

[GR5516]

Risk,

Many thanks for taking the time to reply.

Yes - the utilisation on that core switch has indeed gone down somewhat from what it initially was.

My main worry on it was the huge changes in the arp table, as per the graphic attached

I do not know if this is normal or not - and hence thought it might indicate an issue. In addition, a L2 6500 hanging off this core switch experiences high CPU and noticeable lags when telnetting to it for maintenance.

I initially thought the changeable ARP table might be due to an arp attack, or a network scan, but at this point I will possibly need to look elsewhere for the alleged slowness.

Thank you for your time.

[Richard Burts]

The original post asks about slowness in the network and then focuses on the arp table showing these results

2540 addresses, 29 incomplete

2654 addresses, 124 incomplete

I offer these comments which I hope may be helpful

- we do not know how the arp timers are set. This could have some impact on the volatility of the arp table.

- incomplete entries represent an address for which the switch sent an arp request and has not yet received a response. The entry remains in the table for a short interval and if no response is received the incomplete entry is removed. I would not expect incomplete entries to have a significant impact on the switch performance.

- especially if the arp timer is set to a lower value it is possible that a device was active and generated an entry in the arp table, was dormant for a while and the arp entry was removed, and became active again and created a new entry in the table. This could explain some volatility in the arp table. 

- another thing that can contribute to volatility in the arp table is instability in the network infrastructure. If an interface (physical or vlan) goes down the arp entries are removed and when the interface comes back up new arp entries are created. This could be associated with symptoms of slowness but the slowness is not really about the arp entries and is about the stability of the infrastructure.

- we do not know what percentage of the traffic is forwarded by logic at the line card and how much is punted to the cpu. This certainly could have an impact on performance of the switch and slowness in the network.

HTH

Rick

[GR5516]

thanks for your input