Re: ME 3400 Switch High CPU Utilization due to ARP input problem

rajkotomcr · ‎09-04-2012

We have Cisco ME3400 switch, with configuration described below.

GE0/1 port is uplink:

interface GigabitEthernet0/1

description ***MATED-IUB_C&U-Plan***

port-type nni

switchport trunk allowed vlan 331,450,4085

switchport mode trunk

GEo/2 is configured as below:

interface GigabitEthernet0/2

description ***From NIB***

port-type nni

switchport access vlan 4085

FA0/3 and FA0/15 are configured as below:

interface FastEthernet0/15 (and 3)

description ***NODE-B for rnc07***

port-type nni

switchport access vlan 4085

Now the problem is the some time the CPU Utilization is increased abruptly.

The spanning Tree configuration is biven below:

spanning-tree mode rapid-pvst

spanning-tree extend system-id

and IP Route is given below:

interface Vlan331

ip address 10.135.232.148 255.255.255.240

!

interface Vlan450

ip address 10.72.228.194 255.255.255.252

!

interface Vlan550

ip address 10.72.228.202 255.255.255.252

shutdown

!

interface Vlan4085

ip address 10.72.234.254 255.255.255.0

!

ip http server

no ip http secure-server

ip classless

ip route 0.0.0.0 0.0.0.0 10.135.232.158

ip route 10.72.224.96 255.255.255.240 10.72.228.193

ip route 10.72.228.64 255.255.255.240 10.72.228.193

!

At the time of High cpu utilization we get:

rjrncbb1# show processes cpu history sorted 1min

CPU utilization for five seconds: 98%/29%; one minute: 98%; five minutes: 98%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

7 90596994 55277982 1638 64.21% 64.67% 64.73% 0 ARP Input

195 4130 3888 1062 0.15% 0.25% 0.08% 1 Virtual Exec

and interface summary is :

rjrncbb1#show interfaces summary

*: interface is up

IHQ: pkts in input hold queue IQD: pkts dropped from input queue

OHQ: pkts in output hold queue OQD: pkts dropped from output queue

RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)

TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)

TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

------------------------------------------------------------------------

Vlan1 0 0 0 0 0 0 0 0 0

* Vlan331 0 0 0 0 0 1 0 0 0

Vlan337 0 0 0 0 0 0 0 0 0

* Vlan450 0 0 0 0 0 1 0 0 0

Vlan550 0 0 0 0 0 0 0 0 0

* Vlan4085 17 5533 0 0 1421000 2943 76000 151 0

FastEthernet0/1 0 0 0 0 0 0 0 0 0

FastEthernet0/2 0 0 0 0 0 0 0 0 0

* FastEthernet0/3 0 0 0 0 50000 50 2484000 4529 0

FastEthernet0/4 0 0 0 0 0 0 0 0 0

* FastEthernet0/5 0 0 0 0 4000 8 2463000 4498 0

FastEthernet0/6 0 0 0 0 0 0 0 0 0

FastEthernet0/7 0 0 0 0 0 0 0 0 0

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL

------------------------------------------------------------------------

FastEthernet0/8 0 0 0 0 0 0 0 0 0

FastEthernet0/9 0 0 0 0 0 0 0 0 0

FastEthernet0/10 0 0 0 0 0 0 0 0 0

FastEthernet0/11 0 0 0 0 0 0 0 0 0

FastEthernet0/12 0 0 0 0 0 0 0 0 0

FastEthernet0/13 0 0 0 0 0 0 0 0 0

FastEthernet0/14 0 0 0 0 0 0 0 0 0

* FastEthernet0/15 0 0 0 0 6414000 9113 15109000 13987 0

FastEthernet0/16 0 0 0 0 0 0 0 0 0

FastEthernet0/17 0 0 0 0 0 0 0 0 0

FastEthernet0/18 0 0 0 0 0 0 0 0 0

FastEthernet0/19 0 0 0 0 0 0 0 0 0

FastEthernet0/20 0 0 0 0 0 0 0 0 0

FastEthernet0/21 0 0 0 0 0 0 0 0 0

FastEthernet0/22 0 0 0 0 0 0 0 0 0

* FastEthernet0/23 0 0 0 0 3000 6 2316000 4493 0

FastEthernet0/24 0 0 0 0 0 0 0 0 0

* GigabitEthernet0/1 0 0 0 0 40281000 16323 23938000 36323 0

* GigabitEthernet0/2 0 0 0 0 16321000 27183 26887000 28656 0

NOTE:No separate counters are maintained for subinterfaces

Hence Details of subinterface are not shown

We have also tried Loopguard feature enabled on all ports:

but after some period same problem is repeated.

version information is below:

Cisco IOS Software, ME340x Software (ME340x-METROIPACCESSK9-M), Version 12.2(25)SEG3, RELEASE SOFTWARE (fc2)

Compiled Wed 25-Jul-07 22:56 by amvarma

Kindly help me on this issue at most urgent.

Thank you all.

Sandeep Choudhary · ‎09-04-2012

Hi Sandip,

High CPU utilization in the Address Resolution Protocol (ARP) Input process occurs if the router has to originate an excessive number of ARP requests. The router uses ARP for all hosts, not just those on the local subnet, and ARP requests are sent out as broadcasts, which causes more CPU utilization on every host in the network. ARP requests for the same IP address are rate-limited to one request every two seconds, so an excessive number of ARP requests would have to originate for different IP addresses. This can happen if an IP route has been configured pointing to a broadcast interface. A most obvious example is a default route such as:

ip route 0.0.0.0 0.0.0.0 Fastethernet0/0

In this case, the router generates an ARP request for each IP address that is not reachable through more specific routes, which practically means that the router generates an ARP request for almost every address on the Internet. For more information about configuring next hop address for static routing,

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800ef7b2.shtml

or

Alternatively, an excessive amount of ARP requests can be caused by a malicious traffic stream which scans through locally attached subnets. An indication of such a stream would be the presence of a very high number of incomplete ARP entries in the ARP table. Since incoming IP packets that would trigger ARP requests would have to be processed, troubleshooting this problem would essentially be the same as troubleshooting high CPU utilization in the

IP Input process

In addition to the other post would like to suggest to check out whether ip proxy-arp is enabled on any of the interfaces.

if yes try to disable the same on them and check out ...

it also proves to be a possible cause for the same..

http://www.cisco.com/warp/public/63/highcpu_processes.html#arp

Regards

Please rate if it helps.

rajkotomcr · ‎09-06-2012

I've changed 0.0.0.0 route to 10.0.0.0

ip route 10.0.0.0 255.0.0.0 10.135.232.158

ip route 10.72.224.96 255.255.255.240 10.72.228.193

ip route 10.72.228.64 255.255.255.240 10.72.228.193

But still the problem is the same..the cpu utilization increases abruptly due to ARP Input....below is the summary

rjrncbb1#show processes cpu sorted 1min

CPU utilization for five seconds: 72%/27%; one minute: 83%; five minutes: 43%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

7 127110 148856 853 39.93% 47.42% 24.51% 0 ARP Input

189 1183 419 2823 0.31% 0.34% 0.19% 1 Virtual Exec

200 1198 7 171142 0.00% 0.15% 0.17% 0 hulc running con

146 466 1273 366 0.00% 0.15% 0.10% 0 IP Input

4 295 31 9516 0.00% 0.04% 0.01% 0 Check heaps

108 193 137 1408 0.00% 0.03% 0.02% 0 HRPC qos request

99 66 5520 11 0.00% 0.01% 0.00% 0 Hulc LED Process

28 59 5 11800 0.00% 0.01% 0.00% 0 Per-minute Jobs

107 56 65 861 0.00% 0.01% 0.00% 0 HQM Stack Proces

please help me ...

I've also observed, a drop in Input que....at the time of high cpu utilization.

rsimoni · ‎09-07-2012

Hi Sandip,

how often do you see ARP input process going up? can you add 'show process cpu history'?

How many hosts are normally directly connected to this device?

can you also take 3 instances of 'show ip traffic' with 2-3 seconds interval between each of them and then take 3 more with same interval when you see high cpu.

take also 2 instances of 'sh interfa | i is up|drop|put rate|thro' with 1 second interval and again 2 more when CPU is high.

Riccardo

rajkotomcr · ‎09-14-2012

The hosts are connected to various locations, but connected to switch trhough single GE interface 0/2.

And some 4 to 5 hosts are connected to single FE port 0/15.

Well as per last two days observation, the ARP input process going up on aroung 15:00 in the day time.

But when I gave the following command,

ARP Timeout 30 (under Interface VLAN 4085).

And the show ip arp summary shows 96 ARP entries with 25 of them are incomplete status

Can you tell me whether this command worked for me or there might be some other reason of the ARP Processes suddenly gets down?

Sandeep Choudhary · ‎09-14-2012

HI Sandip,

please paste the output of these command:

sh ip arp summary

show interfaces accounting | i Ethe|ARP|Vlan

first we should know that, from where these traffic is generating?

Regards

rajkotomcr · ‎10-18-2012

Pls find the response to the above command below:

rjrncbb1#show ip arp summary

99 IP ARP entries, with 34 of them incomplete

rjrncbb1#show interfaces accounting | i Ethe|ARP|Vlan

Vlan1

Vlan331

ARP 4 240 46 2760

Interface Vlan337 is disabled

Vlan450

ARP 1 60 6 360

Interface Vlan550 is disabled

Vlan4085

ARP 17371462 1042318684 541831 32509860

Interface FastEthernet0/1 is disabled

FastEthernet0/2

FastEthernet0/3 ***NODE-B for rnc07***

FastEthernet0/4 ***NODE-B for rnc08***

FastEthernet0/5 ***connected_to_BSS-Switch***

FastEthernet0/6 ***connected_to_BSS-Switch***

FastEthernet0/15 ***NODE-B for rnc07***

GigabitEthernet0/1 ***MATED-IUB_C&U-Plan***

GigabitEthernet0/2 ***From NIB***

Pls guide me what to do next???

Sandeep Choudhary · ‎10-18-2012

HI Sandip,

I think the ARP input is coming from VLAN4085

paste the output of this command:show buffers input-interface vlan 4085

questions:

When this problem occurs(Is there any time slot , when it comes high??? like when u r doing backup or ??)

follow this document:

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af6.shtml

Regards

please rate if it helps.

rajkotomcr · ‎10-18-2012

rjrncbb1#show buffers input-interface vlan 4085

Header DataArea Pool Rcnt Size Link Enc Flags Input Output

249E780 741A574 RxQFB 1 60 1 1 210 Vl4085 None

275D400 74C35E4 RxQFB 1 60 1 1 210 Vl4085 None

27605B0 74CBA14 RxQFB 1 60 1 1 210 Vl4085 None

2760900 74CC2E4 RxQFB 1 60 1 1 210 Vl4085 None

2761CE0 74CF7C4 RxQFB 1 60 1 1 210 Vl4085 None

2763410 74D3574 RxQFB 1 60 1 1 210 Vl4085 None

2767300 74DDCE4 RxQFB 1 60 1 1 210 Vl4085 None

2767650 74DE5B4 RxQFB 1 60 1 1 210 Vl4085 None

2768A30 74E1A94 RxQFB 1 60 1 1 210 Vl4085 None

276A160 74E5844 RxQFB 1 60 1 1 210 Vl4085 None

276A4B0 74E6114 RxQFB 1 60 1 1 210 Vl4085 None

276C920 74EC204 RxQFB 1 60 1 1 210 Vl4085 None

276D9B0 74EEE14 RxQFB 1 60 1 1 210 Vl4085 None

276EA40 74F1A24 RxQFB 1 60 1 1 210 Vl4085 None

276ED90 74F22F4 RxQFB 1 60 1 1 210 Vl4085 None

2770B60 74F7244 RxQFB 1 60 1 1 210 Vl4085 None

2771BF0 74F9E54 RxQFB 1 60 1 1 210 Vl4085 None

2773670 74FE4D4 RxQFB 1 60 1 1 210 Vl4085 None

2797A20 755EB04 RxQFB 1 60 1 1 210 Vl4085 None

27A0890 75765F4 RxQFB 1 60 1 1 210 Vl4085 None

27A7290 7587FF4 RxQFB 1 60 1 1 210 Vl4085 None

27A9700 758E0E4 RxQFB 1 60 1 1 210 Vl4085 None

290D8A0 75D2E74 RxQFB 1 60 1 1 210 Vl4085 None

2911AE0 75DDEB4 RxQFB 1 60 1 1 210 Vl4085 None

291DE50 75FE6A4 RxQFB 1 60 1 1 210 Vl4085 None

291EB90 76009E4 RxQFB 1 60 1 1 210 Vl4085 None

2927A00 76184D4 RxQFB 1 60 1 1 210 Vl4085 None

292FE80 762E554 RxQFB 1 60 1 1 210 Vl4085 None

293B160 764C134 RxQFB 1 60 1 1 210 Vl4085 None

293DFC0 7653C94 RxQFB 1 60 1 1 210 Vl4085 None

2941B60 765DB34 RxQFB 1 60 1 1 210 Vl4085 None

2946790 766A5E4 RxQFB 1 60 1 1 210 Vl4085 None

294DB80 767DA54 RxQFB 1 60 1 1 210 Vl4085 None

294F2B0 7681804 RxQFB 1 60 1 1 210 Vl4085 None

2950690 7684CE4 RxQFB 1 60 1 1 210 Vl4085 None

295A8E0 769FCB4 RxQFB 1 60 1 1 210 Vl4085 None

30207C0 7B39EDC RxQ8 1 60 1 1 200 Vl4085 None

3022590 7B3EE2C RxQ8 1 60 1 1 200 Vl4085 None

30232D0 7B4116C RxQ8 1 60 1 1 200 Vl4085 None

3024D50 7B457EC RxQ8 1 60 1 1 200 Vl4085 None

3026130 7B48CCC RxQ8 1 60 1 1 200 Vl4085 None

3027510 7B4C1AC RxQ8 1 60 1 1 200 Vl4085 None

3028C40 7B4FF5C RxQ8 1 60 1 1 200 Vl4085 None

3029CD0 7B52B6C RxQ8 1 60 1 1 200 Vl4085 None

302BDF0 7B5838C RxQ8 1 60 1 1 200 Vl4085 None

302C7E0 7B59DFC RxQ8 1 60 1 1 200 Vl4085 None

30BC0D0 7BAB01C RxQ3 1 60 1 1 210 Vl4085 None

30BCE10 7BAD35C RxQ3 1 60 1 1 210 Vl4085 None

30BF280 7BB344C RxQ3 1 60 1 1 210 Vl4085 None

30BF5D0 7BB3D1C RxQ3 1 60 1 1 210 Vl4085 None

30C4550 7BC109C RxQ3 1 60 1 1 210 Vl4085 None

30C48A0 7BC196C RxQ3 1 60 1 1 210 Vl4085 None

30C5FD0 7BC571C RxQ3 1 60 1 1 210 Vl4085 None

30C6320 7BC5FEC RxQ3 1 60 1 1 210 Vl4085 None

30C6670 7BC68BC RxQ3 1 60 1 1 210 Vl4085 None

30C69C0 7BC718C RxQ3 1 60 1 1 210 Vl4085 None

30C6D10 7BC7A5C RxQ3 1 60 1 1 210 Vl4085 None

30C7060 7BC832C RxQ3 1 60 1 1 210 Vl4085 None

30C73B0 7BC8BFC RxQ3 1 60 1 1 210 Vl4085 None

30C7A50 7BC9D9C RxQ3 1 60 1 1 210 Vl4085 None

30C7DA0 7BCA66C RxQ3 1 60 1 1 210 Vl4085 None

30C80F0 7BCAF3C RxQ3 1 60 1 1 210 Vl4085 None

30C8440 7BCB80C RxQ3 1 60 1 1 210 Vl4085 None

30C8790 7BCC0DC RxQ3 1 60 1 1 210 Vl4085 None

30C8AE0 7BCC9AC RxQ3 1 60 1 1 210 Vl4085 None

30C9180 7BCDB4C RxQ3 1 60 1 1 210 Vl4085 None

30C9820 7BCECEC RxQ3 1 60 1 1 210 Vl4085 None

30CB5F0 7BD3C3C RxQ3 1 60 1 1 210 Vl4085 None

30CC680 7BD684C RxQ3 1 60 1 1 210 Vl4085 None

30D0C10 7BE215C RxQ3 1 60 1 1 210 Vl4085 None

30D3080 7BE824C RxQ3 1 60 1 1 210 Vl4085 None

Header DataArea Pool Rcnt Size Original Flags caller_pc

This problem might occur due to some wrong configuration on the other end of router, Gig 0/1, RNC end. and there is no any fix time slot..