High CPU 3750 on ARP Input

[marcin.piesio]

Hi there,

SO i have 2 stacked 3750 with about 60% cpu due to ARP Input i have couple of 3560 which are serving the same network and not experiecing the same high cpu issue...

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

   9   187462121 262407725        714 57.66% 29.57% 27.51%   0 ARP Input

Below some details about  3750 conf:

no of served vlans ~ 250 with 2 devices per vlan

Total Mac Address Space Available: 5196

sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  172.21.119.253          -   0022.55e6.40c2  ARPA   Vlan19
Internet  172.21.119.250         20   001b.1711.3f15  ARPA   Vlan19
Internet  10.0.51.125            41   0008.0c00.6612  ARPA   Vlan16
Internet  10.0.0.78               -   0022.55e6.40c1  ARPA   Vlan16
Internet  10.0.8.102             14   0008.0c00.6080  ARPA   Vlan16
Internet  10.0.8.101              5   0008.0c00.5e79  ARPA   Vlan16
Internet  10.0.8.110             41   0008.0c00.64db  ARPA   Vlan16

i do not have any static routes just a default gateway pointing to switch managment vlan ip.

I have read some articles from google but cannot find anything useful

Any ideas ?

Thanks for your help!

Marcin

[Latchum Naidu]

Hi Marcin,

Make sure you dont have any default route pointing to interface instead IP.

Try to clear arp, cache, counters and nat table if you have.
Also try to reboot once in half peak hours if possible.

If even you dont have any default route pointing to interface instead its IP and still high cpu because of ARP inspection. Then an excessive amount of ARP requests can be caused by a malicious traffic stream which scans through locally attached subnets. An indication of such a stream is the presence of a very high number of incomplete ARP entries in the ARP table. Because incoming IP packets that trigger ARP requests have to be processed, troubleshooting this problem is essentially the same as troubleshooting high CPU utilization.

Follow the below link for troubleshooting steps....

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af3.shtml

Please rate if this helps you...

Regards,
Naidu.

[haneef001]

Thanks man.....This solved my issue. 

I gave ip route 0.0.0.0 0.0.0.0 gig 0/0, now i gave it on IP and CPU usage came down. 

How does it effect if i gave default route on interface rather than next hop address?

[Amit Singh]

Marcin,

Do you see anything in the logs of the switch? it could be accessive arp request which is eating up all the buffers.

Can you try setting up a SPAN session on the switch and use sniffer to find out who is causing unwanted ARP traffic over the network?

Cheers,

-amit singh

[marcin.piesio]

Hi Amit,

I never setup span port but reading this i think i will be able to do this

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/release/12.1_19_ea1/configuration/guide/swspan.html

I think i will set this per interface rather than vlan.

So what next ? do i connect laptop with Wireshark on it ?

Thanks!

Marcin

[Amit Singh]

Yes Marcin. Plese run wireshark on a laptop and that has to be connected to the destination port on the switch. You can either do a SPAN per port for per vlan basis. If all the ports are in a same vlan, use the vlan as source of the SPAN session else use per interface port spaning.

Cheers,

-amit singh