cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1426
Views
5
Helpful
34
Replies

7609-S high CPU utilization due to IP Input

oscar.garciaj8
Beginner
Beginner

Hi!

I have a Cisco 7609-S and recently experimented high CPU utilization. I attach an important information:

 

Cisco IOS Software, c7600s72033_rp Software (c7600s72033_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRE6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Thu 15-Mar-12 08:07 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)
BOOTLDR: Cisco IOS Software, c7600s72033_rp Software (c7600s72033_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRE6, RELEASE SOFTWARE (fc1)

RM-MTY-01 uptime is 3 years, 29 weeks, 3 days, 7 hours, 46 minutes
Uptime for this control processor is 3 years, 29 weeks, 3 days, 4 hours, 27 minutes
System returned to ROM by power cycle (SP by power on)
System restarted at 08:12:35 CST Wed Jan 30 2019
System image file is "disk0:c7600s72033-advipservicesk9-mz.122-33.SRE6.bin"
Last reload type: Normal Reload


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco CISCO7609-S (R7000) processor (revision 1.0) with 983008K/65536K bytes of memory.
Processor board ID FOX11110L6L
SR71000 CPU at 600MHz, Implementation 1284, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
22 Virtual Ethernet interfaces
148 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

------------------ show process cpu history ------------------




999999999999999999999999999999999999999999999999999999999999
866666888888888889999999998888999998888888888888886666668888
100 **********************************************************
90 **********************************************************
80 **********************************************************
70 **********************************************************
60 **********************************************************
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)




999999999999999999999999999999999999999999999999999999999999
999999999999999987999999999999889998999999999899899999999999
100 *#*##**#***************#********##******##****#***#*******
90 ##########################################################
80 ##########################################################
70 ##########################################################
60 ##########################################################
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%




999999999999999999999999999999999999999999999999999999999999999999999999
999999999999999999999999999999999999999999999999999999999999999999999999
100 **********************************************************************
90 #######***********##########******************************************
80 ###############################******************#********************
70 ################################****************####*#*#*#************
60 ######################################################################
50 ######################################################################
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

RM-MTY-01#show processes cpu sorted
CPU utilization for five seconds: 62%/47%; one minute: 78%; five minutes: 80%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
220 1973557112 3640113288 0 5.27% 5.40% 5.42% 0 IP Input
267 95668 126861 754 0.71% 2.63% 2.81% 1 Virtual Exec
12 644210716 793653526 811 0.71% 0.68% 0.65% 0 ARP Input
268 185841784 536975016 346 0.55% 0.51% 0.50% 0 Ethernet OAM Pro
213 923390972 995653054 927 0.55% 0.69% 0.67% 0 ADJ resolve proc
564 570645604 1705060643 334 0.47% 0.46% 0.41% 0 PDU DISPATCHER
548 205415980 969531015 211 0.47% 0.46% 0.47% 0 NHRP
557 26138296 2191722850 0 0.31% 0.30% 0.31% 0 HSRP Common

 

Please help me to resolve this issue.

 

Thanks in advance.

34 Replies 34

Georg Pauwen
VIP Master VIP Master
VIP Master

Hello,

--> RM-MTY-01 uptime is 3 years, 29 weeks, 3 days, 7 hours, 46 minutes

A 'simple' reboot (preferably after hours) might resolve this. An uptime of >3 years can lead to all sorts of issues...

Thanks for your comments. What do you mean "preferably after hours"?

So, you think a reboot is enough to face this problem? Don't you think we are facing a bug?

Hello,

with 'after hours' I mean, after hopefully nobody needs connectivity through this router...possibly at night.

A very long uptime can lead to memory exhaustion a.o. I would schedule a reboot and check if the problem persists.

Thanks for your comments,

Also I see these HSRP logs on the device:

Aug 23 13:02:01.574: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:02:01.630: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:02:03.202: %LINK-3-UPDOWN: Interface GigabitEthernet2/36, changed state to down
Aug 23 13:02:04.182: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/36, changed state to down
Aug 23 13:02:07.642: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/36, changed state to up
Aug 23 13:02:07.658: %LINK-3-UPDOWN: Interface GigabitEthernet2/36, changed state to up
Aug 23 13:02:13.299: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:03:09.903: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:03:09.967: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:03:21.383: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:04:33.444: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:04:33.500: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:04:46.040: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:04:47.152: %OSPF-5-ADJCHG: Process 1000, Nbr 201.151.25.14 on Vlan100 from FULL to DOWN, Neighbor Down: Dead timer expired
Aug 23 13:04:59.392: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (4) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:05:02.396: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:05:03.784: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:05:15.384: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:05:18.336: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:05:30.716: %OSPF-5-ADJCHG: Process 1000, Nbr 201.151.25.14 on Vlan100 from LOADING to FULL, Loading Done
Aug 23 13:09:17.802: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:09:31.831: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:10:34.547: %LINK-3-UPDOWN: Interface GigabitEthernet2/36, changed state to down
Aug 23 13:10:35.531: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/36, changed state to down
Aug 23 13:10:36.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/36, changed state to up
Aug 23 13:10:36.539: %LINK-3-UPDOWN: Interface GigabitEthernet2/36, changed state to up
Aug 23 13:15:04.686: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:15:11.154: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (4) is UP
Aug 23 13:16:35.107: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (4) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:16:40.199: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:17:55.632: %CLEAR-5-COUNTERS: Clear counter on all interfaces by rtijerina on vty1 (100.111.64.140)
Aug 23 13:22:15.148: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:22:18.308: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (4) is UP
Aug 23 13:22:49.889: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:22:50.037: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:23:00.069: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (4) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:23:01.585: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:23:04.793: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:23:20.321: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:23:24.537: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:28:07.549: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:28:07.969: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:28:18.653: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:28:30.765: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:28:34.905: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:29:41.098: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:29:44.386: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:31:40.643: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:32:08.459: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:32:08.507: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:32:10.811: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:32:21.236: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:33:30.936: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:33:31.004: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:33:42.168: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:34:43.861: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:34:43.913: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:34:45.265: %SYS-6-EXIT_CONFIG: User rtijerina has exited tty session 2(100.111.64.140)
Aug 23 13:34:53.829: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:38:08.267: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:38:08.379: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:38:19.963: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:44:56.795: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:44:56.859: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:45:08.043: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:48:07.189: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:48:07.357: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:48:18.293: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:48:33.161: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:48:33.661: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:48:35.557: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:48:45.605: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:48:50.333: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:51:15.875: %SYS-6-EXIT_CONFIG: User fsilva has exited tty session 1(100.111.64.140)
Aug 23 13:53:10.580: %LINK-3-UPDOWN: Interface GigabitEthernet2/36, changed state to down
Aug 23 13:53:11.712: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/36, changed state to down
Aug 23 13:53:13.856: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:53:15.380: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/36, changed state to up
Aug 23 13:53:15.392: %LINK-3-UPDOWN: Interface GigabitEthernet2/36, changed state to up
Aug 23 13:53:16.612: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:53:16.764: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:53:21.164: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (4) is UP
Aug 23 13:53:28.748: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:53:34.624: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:53:35.632: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:53:46.188: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:55:57.929: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (4) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:56:06.173: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP

 

Do you think it is caused by the high CPU utilization or HSRP logs are causing the high CPU utilization?

 

Thanks in advance.

Hello,

the lines below (obviously) indicate that there are problems with the HSRP group, physical interface, and OSPF adjacency. It is hard to tell if these problems are caused by the high CPU, but that is very well possible. 

Aug 23 13:04:59.392: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (4) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:05:02.396: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Standby -> Active
Aug 23 13:05:03.784: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Active -> Speak
Aug 23 13:05:15.384: %HSRP-5-STATECHANGE: Vlan10 Grp 253 state Speak -> Standby
Aug 23 13:05:18.336: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:05:30.716: %OSPF-5-ADJCHG: Process 1000, Nbr 201.151.25.14 on Vlan100 from LOADING to FULL, Loading Done
Aug 23 13:09:17.802: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is DOWN (Discovery Hello Hold Timer expired)
Aug 23 13:09:31.831: %LDP-5-NBRCHG: LDP Neighbor 207.248.231.243:0 (3) is UP
Aug 23 13:10:34.547: %LINK-3-UPDOWN: Interface GigabitEthernet2/36, changed state to down
Aug 23 13:10:35.531: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/36, changed state to down
Aug 23 13:10:36.523: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/36, changed state to up

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Recognize Your Peers