Hi,Could you plesae try the

Russell Gibbons · ‎05-29-2015

It appears that we are hitting 100% CPU ~every 30 minutes. When it hits, the main difference in CPU is from a process called 'iprouting.iosproc', which I haven't been able to find any good info on. As far as I can tell, it is not causing any dropped packets, but it is being pointed at by the business for a somewhat-failing SaaS solution. Any ideas on how I can mitigate this? I am showing a 'normal' and 'during' output below. I also was able to get a detailed PID.

Thanks,

Russell

HMMBCore#sh proc cpu detailed 16430
CPU utilization for five seconds: 100%; one minute: 86%; five minutes: 65%
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16430 70.5% 26.6% 8.8% iprouting.iosproc 52d02h
2 2.3% 2.6% 1.4% 10 Receive 6d06h
3 0.0% 0.0% 0.0% 10 Receive 1d13h
4 0.0% 0.0% 0.0% 11 Nanosleep 1h19m
5 39.2% 13.8% 4.3% 10 Receive 2d16h
6 0.0% 0.0% 0.0% 10 Receive 0.000
7 0.0% 0.0% 0.0% 10 Receive 2d12h
8 4.7% 1.0% 0.2% 10 Receive 1d22h
9 24.3% 9.2% 2.9% 10 Receive 5d18h
Process sbin/iprouting.iosproc, type IOS, PID = 16430
CPU utilization for five seconds: 64%/1%; one minute: 19%; five minutes: 4%
Task Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Prio Task Name
1 8369852 28282896 295 0.00% 0.00% 0.00% 0 P Hot Service Task
2 8409462 28325590 296 0.00% 0.00% 0.00% 0 P Hot Service Task
3 8421799 28324781 297 0.00% 0.00% 0.00% 0 P Hot Service Task
4 19421122094171544336 465 63.78% 19.23% 4.66% 1 M Service Task
5 10800559 262662394 41 0.03% 0.01% 0.00% 0 M Service Task
6 149675 59816674 2 0.00% 0.00% 0.00% 0 M Service Task
7 0 3 0 0.00% 0.00% 0.00% 0 C Chunk Manager
8 204536 24001511 8 0.00% 0.00% 0.00% 0 C Load Meter
9 0 1 0 0.00% 0.00% 0.00% 0 H Connection Mgr
10 22006 6287890 3 0.00% 0.00% 0.00% 0 M Service Task
11 43260347 13386614 3231 0.00% 0.01% 0.00% 0 L Check heaps
12 0 1 0 0.00% 0.00% 0.00% 0 M Transport Port A
13 0 1 0 0.00% 0.00% 0.00% 0 H Socket Timers
14 26095087 3260875 8002 0.43% 0.06% 0.00% 0 M IP Background
15 37972240 33849616 1121 0.01% 0.00% 0.00% 0 M IP RIB Update
16 0 1 0 0.00% 0.00% 0.00% 0 M clns_write_ Svc
17 33357 3016947 11 0.00% 0.00% 0.00% 0 M IP-EIGRP Router
18 9682 2003260 4 0.00% 0.00% 0.00% 0 M Per-minute Jobs
19 1851158381097334293 168 0.00% 0.01% 0.00% 0 M IP-EIGRP(0): PDM
20 3811 678281 5 0.00% 0.00% 0.00% 0 M Service Task
21 311579401913983598 16 0.00% 0.00% 0.00% 0 M IP-EIGRP(0): HEL
22 7690059 9982154 770 0.00% 0.00% 0.00% 0 P Hot Service Task

When pegged:
HMMBCore#sh proc cpu | ex 0.0% 0.0% 0.0%
CPU utilization for five seconds: 100%; one minute: 97%; five minutes: 77%
PID 5Sec 1Min 5Min Process
1 0.1% 0.4% 0.4% kernel
12304 0.0% 0.1% 0.0% wdsysmon.proc
16407 36.1% 35.3% 45.1% ios-base
16426 0.5% 0.6% 0.8% raw_ip.proc
16428 0.8% 1.0% 3.4% tcp.proc
16429 3.3% 3.0% 3.6% udp.proc
16430 49.8% 55.0% 22.8% iprouting.iosproc
16431 0.7% 0.6% 0.5% cdp2.iosproc
HMMBCore#sh proc cpu his
11111111111111111111111111111111111111111111
7788789900000000000000000000000000000000000000000000999999
7921920900000000000000000000000000000000000000000000999999
100 ***************************************************
90 ****************************************************
80 **********************************************************
70 **********************************************************
60 **********************************************************
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
* = maximum CPU% # = average CPU%
11111 1
9996666668669867666776665656000005657677766676667768787680
9728182054653413025253582851000004565595097948224325648320
100 #* ***#* #
90 #** * ####* * #
80 ##* * ** * ##### * ** **** *#
70 ###* * ****** * *** ** * ##### * ********** ** **** ##
60 ###**#***##*##*#****#*** ***##### **######***#**#**####*##
50 ########################*#*###############################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
1111111111111111111111111111111111111111111111111111111111111111111111
0000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000
100 **********************************************************************
90 **********************************************************************
80 **********************************************************************
70 **********************************************************************
60 #**********************#*****************#************************##*#
50 ##***********#***#########**********#***#########**************#######
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

HMMBCore#

Normal:
HMMBCore#sh proc cpu | ex 0.0% 0.0% 0.0%
CPU utilization for five seconds: 53%; one minute: 56%; five minutes: 55%
PID 5Sec 1Min 5Min Process
1 0.2% 0.3% 0.2% kernel
12304 0.1% 0.0% 0.0% wdsysmon.proc
16407 40.5% 43.1% 41.9% ios-base
16426 0.7% 0.9% 0.9% raw_ip.proc
16428 5.0% 5.1% 5.1% tcp.proc
16429 3.7% 3.5% 3.7% udp.proc
16430 2.0% 2.1% 2.2% iprouting.iosproc
16431 0.6% 0.5% 0.5% cdp2.iosproc
HMMBCore#sh proc cpu his

6666665555554444555555555555555555555556555555555554445555
1104204454539999124664345674642011333680525213434229890345
100
90
80
70
60 ****** * * ** *** * **** * *
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
* = maximum CPU% # = average CPU%
1 11111
7666666666856667677099966666686698676667766656560000056576
8042593082183127305097281820546534130252535828510000045655
100 *#* ***#*
90 ##** * ####*
80 * * * *###* * ** * ##### *
70 * ** * * * **####* * ****** * *** ** * ##### * **
60 #*******#*#*******#####**#***##*##*#****#*** ***##### **##
50 ############################################*#*###########
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
1111111111111111111111111111111111111111111111111111111111111111111111
0000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000
100 **********************************************************************
90 **********************************************************************
80 **********************************************************************
70 **********************************************************************
60 #**********************#*****************#************************##*#
50 ##***********#***#########**********#***#########**************#######
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

Charles Hill · ‎05-29-2015

Take a look at the document below.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/63992-6k-high-cpu.html

what is the iOS?

hope this helps,

please rate helpful posts

thanks

Russell Gibbons · ‎05-29-2015

Currently reading through that doc. IOS is s72033-ipservicesk9_wan-vz.122-33.SXH8.bin

bryantvalencia · ‎06-03-2015

Are you using a NMS? If so, which one?

InayathUlla Sharieff · ‎06-03-2015

Hi,

Could you plesae try the following:-

snmp-server view cutdown iso included
snmp-server view cutdown 1.3.6.1.2.1.4.21 excluded
snmp-server view cutdown 1.3.6.1.2.1.4.22 excluded
snmp-server view cutdown at excluded
snmp-server view cutdown 1.3.6.1.6.3.15 excluded
snmp-server view cutdown 1.3.6.1.6.3.16 excluded
snmp-server view cutdown 1.3.6.1.6.3.18 excluded

snmp-server community <community name> view cutdown RO 8
snmp-server community <community name> view cutdown RO 8

+ You can move to IOS image from ION image - as problem is most probably related just to ION image
+ Disabling SNMP polling of the routing process
+ Restart the process by means of process restart iprouting.iosproc command in EXEC mode
+ Configuration for rate-limiters to protect the CPU

mls rate-limit unicast ip icmp redirect 0
mls rate-limit all ttl-failure 100 10
mls rate-limit unicast ip options 500 50

The ultimate fix would be to migrate a non-Modular version.

HTH

regards

Inayath

High CPU Utilization on 6509vss: iprouting.iosproc