high cpu with cisco 7600 and dont know why ???!!!

[Dr.X]

hi , i have cisco 7600 router ,

here is show run :

Gateway7600#sh run

Building configuration...

Current configuration : 4787 bytes

!

! Last configuration change at 09:08:04 UTC Sat Apr 20 2013 by xxxx

!

version 15.2

service timestamps debug datetime msec

service timestamps log datetime msec

service password-encryption

service counters max age 10

!

hostname Gateway7600

!

boot-start-marker

boot-end-marker

!

!

no logging buffered

enable secret 4 hELPh0MycCJUBD0yxSm5sU05Ew4KLvVXPCEVqVOWpb6

!

no aaa new-model

!

!

!

!

no ip domain lookup

!

!

!

ipv6 multicast rpf use-bgp

!

!

vtp mode transparent

mls flow ip interface-full

no mls flow ipv6

mls qos

mls cef error action reset

multilink bundle-name authenticated

!

!

!

!

spanning-tree mode pvst

spanning-tree extend system-id

system flowcontrol bus auto

diagnostic bootup level minimal

username dscfdscds privilege 0 password xscxzczxczxzxc18

!

redundancy

main-cpu

  auto-sync running-config

mode sso

!

!

!

!

vlan internal allocation policy ascending

vlan access-log ratelimit 2000

!

!

!

!

!

!

!

!

!

!

!

!

!

interface GigabitEthernet1/1

description bras1gateway

mac-address 503d.e5af.9c42

ip address 10.100.100.2 255.255.255.0

load-interval 30

!

interface GigabitEthernet1/2

description Gatewayvlans

mac-address 503d.e5af.9c41

no ip address

load-interval 30

!

interface GigabitEthernet1/2.2

description vlan2gateway

encapsulation dot1Q 2

ip address x.x.67.249 255.255.255.248

!

interface GigabitEthernet1/2.3

description vlan3gateway

encapsulation dot1Q 3

ip address 10.160.150.1 255.255.255.0

!

interface GigabitEthernet1/2.4

description vlan4gateway

encapsulation dot1Q 4

ip address x.x.64.1 255.255.255.224

!

interface GigabitEthernet1/2.5

description vlan5gateway

encapsulation dot1Q 5

ip address 1x.x.79.1 255.255.255.224

!

interface GigabitEthernet1/2.6

description vlan6gateway

encapsulation dot1Q 6

ip address 1x.x.66.245 255.255.255.252

!

interface GigabitEthernet1/2.7

description vlan7gateway

encapsulation dot1Q 7

ip address x.x.65.249 255.255.255.248

!

interface GigabitEthernet1/2.8

description vlan8gateway

encapsulation dot1Q 8

ip address 10.160.160.1 255.255.255.0

!

interface GigabitEthernet1/3

description bras2gateway

mac-address 503d.e5af.9c43

ip address 10.200.200.2 255.255.255.0

load-interval 30

!

interface GigabitEthernet1/4

no ip address

!

interface GigabitEthernet1/5

description 7600to7200

mac-address 503d.e5af.9c45

ip address 172.30.40.2 255.255.255.252

load-interval 30

!

interface GigabitEthernet1/6

mac-address 503d.e5af.9c46

no ip address

shutdown

!

interface GigabitEthernet1/7

mac-address 503d.e5af.9c47

no ip address

shutdown

!

interface GigabitEthernet1/8

mac-address 503d.e5af.9c48

no ip address

shutdown

!

interface GigabitEthernet1/9

mac-address 503d.e5af.9c50

no ip address

shutdown

!

interface Vlan1

no ip address

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

ip route 0.0.0.0 0.0.0.0 172.30.40.1

i

!

ip access-list extended filterips

deny   ip any host 213.8.5.42

deny   ip any host 184.170.252.36

permit ip any any

ip access-list extended rate-limit

permit ip x.x.x.x 0.0.0.31 any

!

!

snmp-server community err0r0 RO

!

!

control-plane

!

!

line con 0

password 7 11584B5643475D

logging synchronous

login

length 0

line vty 0 4

exec-timeout 357 0

password 7 07022E594207

logging synchronous

login local

transport input lat pad mop udptn telnet rlogin ssh nasi acercon

!

!

!

end

=====================

============

CPU utilization for five seconds: 89%/81%; one minute: 87%; five minutes: 76%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

170       80132      286020        280  6.47%  0.97%  0.47%   1 Virtual Exec    

237      672472     2132559        315  1.43%  1.68%  1.66%   0 IP Input        

   2       46200        5264       8776  0.07%  0.06%  0.05%   0 Load Meter      

  27        2604       25757        101  0.07%  0.00%  0.00%   0 IPC Periodic Tim

  58         520       26336         19  0.07%  0.00%  0.00%   0 Dynamic ARP Insp

174       26276       26384        995  0.07%  0.05%  0.04%   0 Per-Second Jobs 

236       14156      800538         17  0.07%  0.01%  0.00%   0 IP ARP Retry Age

   1          20          17       1176  0.00%  0.00%  0.00%   0 Chunk Manager   

   3           8           6       1333  0.00%  0.00%  0.00%   0 RSVP            

   4           0           1          0  0.00%  0.00%  0.00%   0 Inter Chassis Pr

   5           0          66          0  0.00%  0.00%  0.00%   0 Retransmission o

   6           0           2          0  0.00%  0.00%  0.00%   0 IPC ISSU Dispatc

   7           0           1          0  0.00%  0.00%  0.00%   0 PF Redun ICC Req

   8           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers

   9      372668       19347      19262  0.00%  0.39%  0.41%   0 Check heaps     

  10        1340         440       3045  0.00%  0.00%  0.00%   0 Pool Manager    

  11           0           1          0  0.00%  0.00%  0.00%   0 DiscardQ Backgro

  12           0           2          0  0.00%  0.00%  0.00%   0 Timers          

  13        1200       10674        112  0.00%  0.00%  0.00%   0 WATCH_AFS       

  14        5772       11292        511  0.00%  0.01%  0.00%   0 ARP Input      

  End = e   Freeze = f  

Enter Command:

=====================================

plz help how to determin the high cpu cause !!

i dont knwo why th cpu got shot high !!!!!

[Steve Fuller]

Hi,

I think this will be largely traffic related as there's no single entry in the process table that is using any amount of CPU. Take a look at the post understanding show process cpu sorted for an explanation of the two numbers seen in the CPU utilisation numbers.

Additionally you should read through the document Troubleshooting high CPU under interrupts on 7600 and 6500 boxes using "debug netdr" tool as this provides a way to determine what that traffic would be.

Regards

[Dr.X]

hi ,

could this be a bug in my platform ???

here is my ios version !!

ateway7600#sh version

Cisco IOS Software, c7600s3223_rp Software (c7600s3223_rp-ADVENTERPRISEK9-M), Version 15.2(4)S, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2012 by Cisco Systems, Inc.

Compiled Fri 20-Jul-12 18:18 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)SX3, RELEASE SOFTWARE (fc1)

BOOTLDR: Cisco IOS Software, c7600s3223_rp Software (c7600s3223_rp-ADVENTERPRISEK9-M), Version 15.2(4)S, RELEASE SOFTWARE (fc1)

Gateway7600 uptime is 8 hours, 46 minutes

Uptime for this control processor is 8 hours, 46 minutes

System returned to ROM by power-on (SP by power-on)

System image file is "sup-bootdisk:c7600s3223-adventerprisek9-mz.152-4.S.bin"

Last reload type: Normal Reload

Last reload reason: Reload Command

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco CISCO7604 (R7000) processor (revision 2.0) with 458752K/65536K bytes of memory.

Processor board ID FOX1509G2DK

R7000 CPU at 300MHz, Implementation 39, Rev 3.3, 256KB L2, 1024KB L3 Cache

Last reset from power-on

1 Virtual Ethernet interface

9 Gigabit Ethernet interfaces

1915K bytes of non-volatile configuration memory.

65536K bytes of Flash internal SIMM (Sector size 512K).

Configuration register is 0x2102

[Steve Fuller]

I think it's too early to say whether this is a bug until you've looked a little further into what could be causing the problem.

As I mentioned above, the utilisation would appear to be traffic related as the majority of the processor utilisation is Interrupt based. So some questions:

- Has the CPU suddenly increased and if so when?

- Were there any changes at or around the time of the increase?

- Is the traffic volume through the router as you would expect or has that increased?

If you run the command show int | in ^[A-Z]|put rate is there any interface that is carrying a significant volume of traffic?

If you're running MLS and NetFlow then you can also run the command show mls netflow ip nowrap and check if there is one IP address that accounts for a large volume of the traffic.

Regards

[Dr.X]

hi ,

its unstable cpu behaviuor ,

i mean that sometimes cpu gets high ,

and seems get higher quickly

if i remove the snmp it gets lower !!!

i also removed logging host x.x.x.x

also the cpu gots  lower ,

but after that , the cpu gets higher again ??!!!!!

dont know why !!!

i want to tell you something i tested

the 7600 router is connected with two identical  routers .

i mean that the 7600 is the gateway of the two other routers

the other two routers neraly carry the same number of users

about 1500 user per router and about 250 M .

the strange issue is , if i discconect the 7600 from router 1 and router 2 is connected

the cpu become 5 %

if i disconnected  router 2 and still connceting router  1  ,  the cpu still high .50-80 %

i  mean that althoug both routers are neraly identiacal , i  found that the problem is from one of the two routers.

i read  a alillte bit , i found i  need to enable fast switching !!

not sure if i must enable it on the other two routers !!!!

im really shocked !!!

[Dr.X]

output of R1 interfacve that connected with 7600

Bras1#sh interfaces gigabitEthernet 0/1

GigabitEthernet0/1 is up, line protocol is up

  Hardware is MV64460 Internal MAC, address is 0026.caf7.c01b (bia 0026.caf7.c01b)

  Description: Internet From Gateway

  Internet address is 10.100.100.100/24

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 12/255, rxload 72/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is RJ45

  output flow-control is XON, input flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 18/75/242/10452 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  30 second input rate 284177000 bits/sec, 48479 packets/sec

  30 second output rate 49358000 bits/sec, 35228 packets/sec

     209485959 packets input, 1108925719 bytes, 83 no buffer

     Received 73 broadcasts, 0 runts, 0 giants, 0 throttles

     41 input errors, 0 CRC, 0 frame, 0 overrun, 41 ignored

     0 watchdog, 1944 multicast, 0 pause input

     0 input packets with dribble condition detected

     169982882 packets output, 264503997 bytes, 0 underruns

     2 output errors, 0 collisions, 1 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     2 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

[Steve Fuller]

If you run the command show proc cpu sorted | ex 0.0 a number of times, do you see the CPU utilisation going up and down, or staying up?

Also are the two numbers for the 5-second utilisation always close as they were in your first post e.g., 89%/81% or do you sometimes see the first number much higher than the second e.g., 89%/10%. In the case where the two numbers are not close what is the entry in the process table that's using the CPU?

Regards

[Dr.X]

hi , not always the numbers are close .!! i mean sometimes ths two numbers are close and sometimes no !

i mean , now , at this moment its stable ,

=========================================================

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 25%/23%; one minute: 28%; five minutes: 29%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

237     1008688     3053126        330  1.03%  1.54%  1.46%   0 IP Input        

434      172868       17712       9759  0.31%  0.16%  0.16%   0 HIDDEN VLAN Proc

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 28%/26%; one minute: 28%; five minutes: 29%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

237     1008800     3053480        330  0.87%  1.49%  1.45%   0 IP Input        

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 28%/26%; one minute: 28%; five minutes: 29%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

237     1008832     3053621        330  0.87%  1.49%  1.45%   0 IP Input        

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 32%/28%; one minute: 28%; five minutes: 29%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

237     1008900     3053998        330  1.19%  1.47%  1.44%   0 IP Input        

434      172888       17717       9758  0.31%  0.16%  0.16%   0 HIDDEN VLAN Proc

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 32%/28%; one minute: 28%; five minutes: 29%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

237     1008916     3054230        330  1.19%  1.47%  1.44%   0 IP Input        

434      172888       17718       9757  0.31%  0.16%  0.16%   0 HIDDEN VLAN Proc

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 36%/32%; one minute: 29%; five minutes: 30%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   9      479456       24985      19189  1.59%  0.49%  0.42%   0 Check heaps     

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 36%/32%; one minute: 29%; five minutes: 30%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   9      479456       24985      19189  1.59%  0.49%  0.42%   0 Check heaps     

237     1009140     3054796        330  0.47%  1.39%  1.43%   0 IP Input        

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 34%/29%; one minute: 29%; five minutes: 30%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

170       16568        5289       3132  1.83%  0.45%  0.10%   1 Virtual Exec    

237     1009200     3055028        330  1.83%  1.42%  1.43%   0 IP Input        

   9      479456       24985      19189  0.63%  0.50%  0.42%   0 Check heaps     

434      172908       17722       9756  0.39%  0.16%  0.16%   0 HIDDEN VLAN Proc

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 34%/29%; one minute: 29%; five minutes: 30%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

170       16636        5294       3142  1.83%  0.45%  0.10%   1 Virtual Exec    

237     1009268     3055309        330  1.83%  1.42%  1.43%   0 IP Input        

   9      479456       24985      19189  0.63%  0.50%  0.42%   0 Check heaps     

434      172908       17723       9756  0.39%  0.16%  0.16%   0 HIDDEN VLAN Proc

Gateway7600#show proc cpu sorted | ex 0.0

CPU utilization for five seconds: 33%/30%; one minute: 29%; five minutes: 30%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

170       16692        5299       3150  1.67%  0.55%  0.13%   1 Virtual Exec    

237     1009296     3055406        330  0.71%  1.37%  1.42%   0 IP Input        

Gateway7600#

================================

=====================================

i monitor the cpu process live , by the command

show process cpu monitor

==============

dont  knwo why suddenly it gets high !!!

also , im really surprised .

why snmp cause high cpu ??!!!!!

[Steve Fuller · Accepted solution]

I ran the following snmpwalk against a Cisco 7604 in my test lab...

[sfuller@rhel8 ~]$ snmpwalk ocs7604-1 1.3.6.1

SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS Software, c7600s72033_rp Software (c7600s72033_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRB7, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2009 by Cisco Systems, Inc.

Compiled Fri 04-Sep-09 20:06

SNMPv2-MIB::sysObjectID.0 = OID: CISCO-SMI::ciscoProducts.658

[..]

And managed to push the CPU utilisation to 50% when it was doing little else i.e., switching traffic.

ocs7604-1#sh proc cp sort | ex 0.0


CPU utilization for five seconds: 50%/23%; one minute: 27%; five minutes: 8%


PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process


449       10312     72670        141 11.75%  6.85%  1.97%   0 SNMP ENGINE


447        8148    105812         77  5.27%  4.12%  1.28%   0 IP SNMP


191      198916   1908768        104  3.59%  2.84%  0.88%   0 IP Input


  7     1641396    120679      13601  2.31%  0.45%  0.32%   0 Check heaps


314         728       224       3250  1.51%  0.77%  0.21%   1 Virtual Exec


448        1188     52559         22  0.87%  0.66%  0.21%   0 PDU DISPATCHER


296      885668    276572       3202  0.31%  0.16%  0.15%   0 HIDDEN VLAN Proc


Based on this it's quite possible to push the CPU to very high levels if you're also routing a lot of traffic through the device.

I guess the question is who's polling and why so much?

If you run the debug netdr capture (as detailed in the link in my first response) while it's running high you should be able to see who is the source of the SNMP traffic. The following is an example while the above snmpwalk was taking place:

------- dump of incoming inband packet -------

interface NULL, routine mistral_process_rx_packet_inlin

dbus info: src_vlan 0x3F5(1013), src_indx 0xC0(192), len 0x71(113)

  bpdu 0, index_dir 0, flood 0, dont_lrn 0, dest_indx 0x380(896)

  38020000 03F50000 00C00000 71000000 00110520 0E000040 00000000 03800000

mistral hdr: req_token 0x0(0), src_index 0xC0(192), rx_offset 0x76(118)

  requeue 0, obl_pkt 0, vlan 0x3F5(1013)

destmac 00.27.0D.46.E4.40, srcmac 64.87.88.5D.5A.72, protocol 0800

protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 95, identifier 0

  df 1, mf 0, fo 0, ttl 62, src 192.168.11.115, dst 192.168.2.132

    udp src 48998, dst 161 len 75 checksum 0x662A

Here we can see the packet is SNMP (destined UDP port 161) and from this we can see the source is 192.168.11.115.

Regards

[Dr.X]

hi steve ,  the cisco 7600 now is stable .

i dont know why ??!!!

im also afraid to enable snmp agian ,  but i will start hitting the books about cpu usgae in7600 and read more

thanks alot for interest  and replies

with my best regards