09-04-2012 03:54 AM - edited 03-07-2019 08:40 AM
We have Cisco ME3400 switch, with configuration described below.
GE0/1 port is uplink:
interface GigabitEthernet0/1
description ***MATED-IUB_C&U-Plan***
port-type nni
switchport trunk allowed vlan 331,450,4085
switchport mode trunk
GEo/2 is configured as below:
interface GigabitEthernet0/2
description ***From NIB***
port-type nni
switchport access vlan 4085
FA0/3 and FA0/15 are configured as below:
interface FastEthernet0/15 (and 3)
description ***NODE-B for rnc07***
port-type nni
switchport access vlan 4085
Now the problem is the some time the CPU Utilization is increased abruptly.
The spanning Tree configuration is biven below:
spanning-tree mode rapid-pvst
spanning-tree extend system-id
and IP Route is given below:
interface Vlan331
ip address 10.135.232.148 255.255.255.240
!
interface Vlan450
ip address 10.72.228.194 255.255.255.252
!
interface Vlan550
ip address 10.72.228.202 255.255.255.252
shutdown
!
interface Vlan4085
ip address 10.72.234.254 255.255.255.0
!
ip http server
no ip http secure-server
ip classless
ip route 0.0.0.0 0.0.0.0 10.135.232.158
ip route 10.72.224.96 255.255.255.240 10.72.228.193
ip route 10.72.228.64 255.255.255.240 10.72.228.193
!
At the time of High cpu utilization we get:
rjrncbb1# show processes cpu history sorted 1min
CPU utilization for five seconds: 98%/29%; one minute: 98%; five minutes: 98%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
7 90596994 55277982 1638 64.21% 64.67% 64.73% 0 ARP Input
195 4130 3888 1062 0.15% 0.25% 0.08% 1 Virtual Exec
and interface summary is :
rjrncbb1#show interfaces summary
*: interface is up
IHQ: pkts in input hold queue IQD: pkts dropped from input queue
OHQ: pkts in output hold queue OQD: pkts dropped from output queue
RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)
TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)
TRTL: throttle count
Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
------------------------------------------------------------------------
Vlan1 0 0 0 0 0 0 0 0 0
* Vlan331 0 0 0 0 0 1 0 0 0
Vlan337 0 0 0 0 0 0 0 0 0
* Vlan450 0 0 0 0 0 1 0 0 0
Vlan550 0 0 0 0 0 0 0 0 0
* Vlan4085 17 5533 0 0 1421000 2943 76000 151 0
FastEthernet0/1 0 0 0 0 0 0 0 0 0
FastEthernet0/2 0 0 0 0 0 0 0 0 0
* FastEthernet0/3 0 0 0 0 50000 50 2484000 4529 0
FastEthernet0/4 0 0 0 0 0 0 0 0 0
* FastEthernet0/5 0 0 0 0 4000 8 2463000 4498 0
FastEthernet0/6 0 0 0 0 0 0 0 0 0
FastEthernet0/7 0 0 0 0 0 0 0 0 0
Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
------------------------------------------------------------------------
FastEthernet0/8 0 0 0 0 0 0 0 0 0
FastEthernet0/9 0 0 0 0 0 0 0 0 0
FastEthernet0/10 0 0 0 0 0 0 0 0 0
FastEthernet0/11 0 0 0 0 0 0 0 0 0
FastEthernet0/12 0 0 0 0 0 0 0 0 0
FastEthernet0/13 0 0 0 0 0 0 0 0 0
FastEthernet0/14 0 0 0 0 0 0 0 0 0
* FastEthernet0/15 0 0 0 0 6414000 9113 15109000 13987 0
FastEthernet0/16 0 0 0 0 0 0 0 0 0
FastEthernet0/17 0 0 0 0 0 0 0 0 0
FastEthernet0/18 0 0 0 0 0 0 0 0 0
FastEthernet0/19 0 0 0 0 0 0 0 0 0
FastEthernet0/20 0 0 0 0 0 0 0 0 0
FastEthernet0/21 0 0 0 0 0 0 0 0 0
FastEthernet0/22 0 0 0 0 0 0 0 0 0
* FastEthernet0/23 0 0 0 0 3000 6 2316000 4493 0
FastEthernet0/24 0 0 0 0 0 0 0 0 0
* GigabitEthernet0/1 0 0 0 0 40281000 16323 23938000 36323 0
* GigabitEthernet0/2 0 0 0 0 16321000 27183 26887000 28656 0
NOTE:No separate counters are maintained for subinterfaces
Hence Details of subinterface are not shown
We have also tried Loopguard feature enabled on all ports:
but after some period same problem is repeated.
version information is below:
Cisco IOS Software, ME340x Software (ME340x-METROIPACCESSK9-M), Version 12.2(25)SEG3, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Wed 25-Jul-07 22:56 by amvarma
Kindly help me on this issue at most urgent.
Thank you all.
09-04-2012 04:12 AM
Hi Sandip,
High CPU utilization in the Address Resolution Protocol (ARP) Input process occurs if the router has to originate an excessive number of ARP requests. The router uses ARP for all hosts, not just those on the local subnet, and ARP requests are sent out as broadcasts, which causes more CPU utilization on every host in the network. ARP requests for the same IP address are rate-limited to one request every two seconds, so an excessive number of ARP requests would have to originate for different IP addresses. This can happen if an IP route has been configured pointing to a broadcast interface. A most obvious example is a default route such as:
ip route 0.0.0.0 0.0.0.0 Fastethernet0/0
In this case, the router generates an ARP request for each IP address that is not reachable through more specific routes, which practically means that the router generates an ARP request for almost every address on the Internet. For more information about configuring next hop address for static routing,
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800ef7b2.shtml
or
Alternatively, an excessive amount of ARP requests can be caused by a malicious traffic stream which scans through locally attached subnets. An indication of such a stream would be the presence of a very high number of incomplete ARP entries in the ARP table. Since incoming IP packets that would trigger ARP requests would have to be processed, troubleshooting this problem would essentially be the same as troubleshooting high CPU utilization in the
IP Input process
In addition to the other post would like to suggest to check out whether ip proxy-arp is enabled on any of the interfaces.
if yes try to disable the same on them and check out ...
it also proves to be a possible cause for the same..
http://www.cisco.com/warp/public/63/highcpu_processes.html#arp
Regards
Please rate if it helps.
09-06-2012 11:14 PM
I've changed 0.0.0.0 route to 10.0.0.0
ip route 10.0.0.0 255.0.0.0 10.135.232.158
ip route 10.72.224.96 255.255.255.240 10.72.228.193
ip route 10.72.228.64 255.255.255.240 10.72.228.193
But still the problem is the same..the cpu utilization increases abruptly due to ARP Input....below is the summary
rjrncbb1#show processes cpu sorted 1min
CPU utilization for five seconds: 72%/27%; one minute: 83%; five minutes: 43%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
7 127110 148856 853 39.93% 47.42% 24.51% 0 ARP Input
189 1183 419 2823 0.31% 0.34% 0.19% 1 Virtual Exec
200 1198 7 171142 0.00% 0.15% 0.17% 0 hulc running con
146 466 1273 366 0.00% 0.15% 0.10% 0 IP Input
4 295 31 9516 0.00% 0.04% 0.01% 0 Check heaps
108 193 137 1408 0.00% 0.03% 0.02% 0 HRPC qos request
99 66 5520 11 0.00% 0.01% 0.00% 0 Hulc LED Process
28 59 5 11800 0.00% 0.01% 0.00% 0 Per-minute Jobs
107 56 65 861 0.00% 0.01% 0.00% 0 HQM Stack Proces
please help me ...
I've also observed, a drop in Input que....at the time of high cpu utilization.
09-07-2012 02:00 AM
Hi Sandip,
how often do you see ARP input process going up? can you add 'show process cpu history'?
How many hosts are normally directly connected to this device?
can you also take 3 instances of 'show ip traffic' with 2-3 seconds interval between each of them and then take 3 more with same interval when you see high cpu.
take also 2 instances of 'sh interfa | i is up|drop|put rate|thro' with 1 second interval and again 2 more when CPU is high.
Riccardo
09-14-2012 12:27 AM
The hosts are connected to various locations, but connected to switch trhough single GE interface 0/2.
And some 4 to 5 hosts are connected to single FE port 0/15.
Well as per last two days observation, the ARP input process going up on aroung 15:00 in the day time.
But when I gave the following command,
ARP Timeout 30 (under Interface VLAN 4085).
And the show ip arp summary shows 96 ARP entries with 25 of them are incomplete status
Can you tell me whether this command worked for me or there might be some other reason of the ARP Processes suddenly gets down?
09-14-2012 12:43 AM
HI Sandip,
please paste the output of these command:
sh ip arp summary
show interfaces accounting | i Ethe|ARP|Vlan
first we should know that, from where these traffic is generating?
Regards
10-18-2012 05:43 AM
Pls find the response to the above command below:
rjrncbb1#show ip arp summary
99 IP ARP entries, with 34 of them incomplete
rjrncbb1#show interfaces accounting | i Ethe|ARP|Vlan
Vlan1
Vlan331
ARP 4 240 46 2760
Interface Vlan337 is disabled
Vlan450
ARP 1 60 6 360
Interface Vlan550 is disabled
Vlan4085
ARP 17371462 1042318684 541831 32509860
Interface FastEthernet0/1 is disabled
FastEthernet0/2
FastEthernet0/3 ***NODE-B for rnc07***
FastEthernet0/4 ***NODE-B for rnc08***
FastEthernet0/5 ***connected_to_BSS-Switch***
FastEthernet0/6 ***connected_to_BSS-Switch***
FastEthernet0/15 ***NODE-B for rnc07***
GigabitEthernet0/1 ***MATED-IUB_C&U-Plan***
GigabitEthernet0/2 ***From NIB***
Pls guide me what to do next???
10-18-2012 06:20 AM
HI Sandip,
I think the ARP input is coming from VLAN4085
paste the output of this command:show buffers input-interface vlan 4085
questions:
When this problem occurs(Is there any time slot , when it comes high??? like when u r doing backup or ??)
follow this document:
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af6.shtml
Regards
please rate if it helps.
10-18-2012 06:38 AM
rjrncbb1#show buffers input-interface vlan 4085
Header DataArea Pool Rcnt Size Link Enc Flags Input Output
249E780 741A574 RxQFB 1 60 1 1 210 Vl4085 None
275D400 74C35E4 RxQFB 1 60 1 1 210 Vl4085 None
27605B0 74CBA14 RxQFB 1 60 1 1 210 Vl4085 None
2760900 74CC2E4 RxQFB 1 60 1 1 210 Vl4085 None
2761CE0 74CF7C4 RxQFB 1 60 1 1 210 Vl4085 None
2763410 74D3574 RxQFB 1 60 1 1 210 Vl4085 None
2767300 74DDCE4 RxQFB 1 60 1 1 210 Vl4085 None
2767650 74DE5B4 RxQFB 1 60 1 1 210 Vl4085 None
2768A30 74E1A94 RxQFB 1 60 1 1 210 Vl4085 None
276A160 74E5844 RxQFB 1 60 1 1 210 Vl4085 None
276A4B0 74E6114 RxQFB 1 60 1 1 210 Vl4085 None
276C920 74EC204 RxQFB 1 60 1 1 210 Vl4085 None
276D9B0 74EEE14 RxQFB 1 60 1 1 210 Vl4085 None
276EA40 74F1A24 RxQFB 1 60 1 1 210 Vl4085 None
276ED90 74F22F4 RxQFB 1 60 1 1 210 Vl4085 None
2770B60 74F7244 RxQFB 1 60 1 1 210 Vl4085 None
2771BF0 74F9E54 RxQFB 1 60 1 1 210 Vl4085 None
2773670 74FE4D4 RxQFB 1 60 1 1 210 Vl4085 None
2797A20 755EB04 RxQFB 1 60 1 1 210 Vl4085 None
27A0890 75765F4 RxQFB 1 60 1 1 210 Vl4085 None
27A7290 7587FF4 RxQFB 1 60 1 1 210 Vl4085 None
27A9700 758E0E4 RxQFB 1 60 1 1 210 Vl4085 None
290D8A0 75D2E74 RxQFB 1 60 1 1 210 Vl4085 None
2911AE0 75DDEB4 RxQFB 1 60 1 1 210 Vl4085 None
291DE50 75FE6A4 RxQFB 1 60 1 1 210 Vl4085 None
291EB90 76009E4 RxQFB 1 60 1 1 210 Vl4085 None
2927A00 76184D4 RxQFB 1 60 1 1 210 Vl4085 None
292FE80 762E554 RxQFB 1 60 1 1 210 Vl4085 None
293B160 764C134 RxQFB 1 60 1 1 210 Vl4085 None
293DFC0 7653C94 RxQFB 1 60 1 1 210 Vl4085 None
2941B60 765DB34 RxQFB 1 60 1 1 210 Vl4085 None
2946790 766A5E4 RxQFB 1 60 1 1 210 Vl4085 None
294DB80 767DA54 RxQFB 1 60 1 1 210 Vl4085 None
294F2B0 7681804 RxQFB 1 60 1 1 210 Vl4085 None
2950690 7684CE4 RxQFB 1 60 1 1 210 Vl4085 None
295A8E0 769FCB4 RxQFB 1 60 1 1 210 Vl4085 None
30207C0 7B39EDC RxQ8 1 60 1 1 200 Vl4085 None
3022590 7B3EE2C RxQ8 1 60 1 1 200 Vl4085 None
30232D0 7B4116C RxQ8 1 60 1 1 200 Vl4085 None
3024D50 7B457EC RxQ8 1 60 1 1 200 Vl4085 None
3026130 7B48CCC RxQ8 1 60 1 1 200 Vl4085 None
3027510 7B4C1AC RxQ8 1 60 1 1 200 Vl4085 None
3028C40 7B4FF5C RxQ8 1 60 1 1 200 Vl4085 None
3029CD0 7B52B6C RxQ8 1 60 1 1 200 Vl4085 None
302BDF0 7B5838C RxQ8 1 60 1 1 200 Vl4085 None
302C7E0 7B59DFC RxQ8 1 60 1 1 200 Vl4085 None
30BC0D0 7BAB01C RxQ3 1 60 1 1 210 Vl4085 None
30BCE10 7BAD35C RxQ3 1 60 1 1 210 Vl4085 None
30BF280 7BB344C RxQ3 1 60 1 1 210 Vl4085 None
30BF5D0 7BB3D1C RxQ3 1 60 1 1 210 Vl4085 None
30C4550 7BC109C RxQ3 1 60 1 1 210 Vl4085 None
30C48A0 7BC196C RxQ3 1 60 1 1 210 Vl4085 None
30C5FD0 7BC571C RxQ3 1 60 1 1 210 Vl4085 None
30C6320 7BC5FEC RxQ3 1 60 1 1 210 Vl4085 None
30C6670 7BC68BC RxQ3 1 60 1 1 210 Vl4085 None
30C69C0 7BC718C RxQ3 1 60 1 1 210 Vl4085 None
30C6D10 7BC7A5C RxQ3 1 60 1 1 210 Vl4085 None
30C7060 7BC832C RxQ3 1 60 1 1 210 Vl4085 None
30C73B0 7BC8BFC RxQ3 1 60 1 1 210 Vl4085 None
30C7A50 7BC9D9C RxQ3 1 60 1 1 210 Vl4085 None
30C7DA0 7BCA66C RxQ3 1 60 1 1 210 Vl4085 None
30C80F0 7BCAF3C RxQ3 1 60 1 1 210 Vl4085 None
30C8440 7BCB80C RxQ3 1 60 1 1 210 Vl4085 None
30C8790 7BCC0DC RxQ3 1 60 1 1 210 Vl4085 None
30C8AE0 7BCC9AC RxQ3 1 60 1 1 210 Vl4085 None
30C9180 7BCDB4C RxQ3 1 60 1 1 210 Vl4085 None
30C9820 7BCECEC RxQ3 1 60 1 1 210 Vl4085 None
30CB5F0 7BD3C3C RxQ3 1 60 1 1 210 Vl4085 None
30CC680 7BD684C RxQ3 1 60 1 1 210 Vl4085 None
30D0C10 7BE215C RxQ3 1 60 1 1 210 Vl4085 None
30D3080 7BE824C RxQ3 1 60 1 1 210 Vl4085 None
Header DataArea Pool Rcnt Size Original Flags caller_pc
This problem might occur due to some wrong configuration on the other end of router, Gig 0/1, RNC end. and there is no any fix time slot..
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide