04-04-2011 10:33 AM - edited 03-06-2019 04:26 PM
I recently had a 2621 router that was hanging up at 100% cpu utilization when large data files were transfered across the network. I replaced the router with a spare 2621 and the problem went away. I decided to play around with the problem router to see what happened. I connected a PC to each ethernet port on the router and attempted to transfer a 300mb file between them while consoled into the router. The file transfer went so slow that after 20min it was only 10 percent completed, but the cpu never went over 8% during that time. I upgraded the IOS with the same results. Since the original config had ip routes and an access-list, I next wiped the configuration only adding back an ip address on each ethernet interface...nice and clean. This time the file transferred normally as it should, but the console session locked up until the transfer completed. The cpu reached 100% utilization transferring that single file. I tried another spare 2621 router with a clean config and it also locked up with 100% cpu.
Am I missing something? I am currently working towards my CCNA so I still have alot more to learn when it comes to networking. I am hoping someone can shed some light.
04-06-2011 02:58 AM
It might be a bug. Please post your #sh version here.
04-06-2011 03:18 PM
I tested this scenario on two different 2621 routers with the same results. Below are both sh ver from each router:
Router#sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-D-M), Version 12.1(5)T, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Sat 11-Nov-00 12:18 by ccai
Image text-base: 0x80008088, data-base: 0x80A88C1C
ROM: System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
Router uptime is 3 minutes
System returned to ROM by power-on
System image file is "flash:c2600-d-mz.121-5.T"
cisco 2621 (MPC860) processor (revision 0x102) with 27648K/5120K bytes of memory.
Processor board ID JAD045257KI (2663601130)
M860 processor: part number 0, mask 49
Bridging software.
X.25 software, Version 3.0.0.
2 FastEthernet/IEEE 802.3 interface(s)
1 Serial network interface(s)
32K bytes of non-volatile configuration memory.
8192K bytes of processor board System flash (Read/Write)
Configuration register is 0x2102
Router#sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-I-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Mon 17-Mar-08 15:23 by dchih
ROM: System Bootstrap, Version 12.2(6r), RELEASE SOFTWARE (fc1)
Router uptime is 0 minutes
System returned to ROM by power-on
System image file is "flash:c2600-i-mz.123-26.bin"
cisco 2621 (MPC860) processor (revision 0x00) with 28672K/4096K bytes of memory.
Processor board ID JAD062206DQ (2653152028)
M860 processor: part number 0, mask 49
Bridging software.
X.25 software, Version 3.0.0.
2 FastEthernet/IEEE 802.3 interface(s)
32K bytes of non-volatile configuration memory.
8192K bytes of processor board System flash (Read/Write)
Configuration register is 0x2102
Since it is happening on two routers, it makes me think it is me and not the hardware/IOS. Any help would be greatly appreciated.
04-07-2011 12:38 AM
I know you say the console session locks when you try the transfer but can you try and so #sh proc cpu before it locks up? I didn't a specific bug related to this in either the 12.1 or 12.3 version.
Also, do you see anything in the log files?
04-07-2011 12:34 PM
Okay, so here is what I did:
(Keep in mind I tried everything on both 2621 routers and got the same results)
Since I knew the console session would hang up during the file transfer, I pre-typed sh proc cpu so I could just tap the enter key and hope it goes through. I started the file transfer, the console hanged, I hit the enter key, and when the file transfer completed...this is what I got:
Router#sh proc cpu
CPU utilization for five seconds: 100%/99%; one minute: 15%; five minutes: 3%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 4 51 78 0.04% 0.00% 0.00% 0 Load Meter
2 1344 193 6963 0.00% 0.17% 0.25% 0 Exec
3 100 27 3703 0.00% 0.02% 0.00% 0 Check heaps
4 4 1 4000 0.00% 0.00% 0.00% 0 Chunk Manager
5 4 2 2000 0.00% 0.00% 0.00% 0 Pool Manager
6 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
8 0 36 0 0.00% 0.00% 0.00% 0 ALARM_TRIGGER_SC
9 4 10 400 0.00% 0.00% 0.00% 0 Environmental mo
10 28 43 651 0.00% 0.00% 0.00% 0 ARP Input
11 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers
12 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
13 4 3 1333 0.00% 0.00% 0.00% 0 Entity MIB API
14 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
15 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
16 16 133 120 0.00% 0.00% 0.00% 0 Net Background
17 4 14 285 0.00% 0.00% 0.00% 0 Logger
18 9556 239 39983 0.00% 0.00% 0.00% 0 TTY Background
19 0 285 0 0.00% 0.00% 0.00% 0 Per-Second Jobs
20 0 2 0 0.00% 0.00% 0.00% 0 Hawkeye Backgrou
22 0 1 0 0.00% 0.00% 0.00% 0 Net Input
23 0 52 0 0.00% 0.00% 0.00% 0 Compute load avg
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
24 84 6 14000 0.00% 0.01% 0.00% 0 Per-minute Jobs
25 4 2 2000 0.00% 0.00% 0.00% 0 Service-module a
26 160 347 461 0.00% 0.03% 0.01% 0 IP Input
27 8 29 275 0.00% 0.00% 0.00% 0 CDP Protocol
28 0 3 0 0.00% 0.00% 0.00% 0 MOP Protocols
29 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
30 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper
31 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add Route
32 0 38 0 0.00% 0.00% 0.00% 0 IP Background
33 0 6 0 0.00% 0.00% 0.00% 0 Adj Manager
34 0 1 0 0.00% 0.00% 0.00% 0 TCP Timer
35 0 1 0 0.00% 0.00% 0.00% 0 TCP Protocols
36 4 1 4000 0.00% 0.00% 0.00% 0 Probe Input
37 4 1 4000 0.00% 0.00% 0.00% 0 RARP Input
38 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer
39 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
40 4 2 2000 0.00% 0.00% 0.00% 0 DHCPD Receive
41 0 6 0 0.00% 0.00% 0.00% 0 IP Cache Ager
42 0 1 0 0.00% 0.00% 0.00% 0 COPS
43 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
44 0 3 0 0.00% 0.00% 0.00% 0 X.25 Background
45 0 1 0 0.00% 0.00% 0.00% 0 ISDN Timer
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
47 4 310 12 0.00% 0.00% 0.00% 0 Router Autoconf
48 0 1 0 0.00% 0.00% 0.00% 0 ISDNMIB Backgrou
49 0 1 0 0.00% 0.00% 0.00% 0 CallMIB Backgrou
50 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
51 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
52 0 3 0 0.00% 0.00% 0.00% 0 DHCPD Timer
54 0 29 0 0.00% 0.00% 0.00% 0 DHCP Client
55 0 70 0 0.00% 0.00% 0.00% 0 DHCPD Database
57 4 2 2000 0.00% 0.00% 0.00% 0 CCP manager
58 0 2 0 0.00% 0.00% 0.00% 0 PPP manager
59 4 262 15 0.00% 0.00% 0.00% 0 Multilink PPP
60 0 2 0 0.00% 0.00% 0.00% 0 Multilink PPP ou
61 0 2 0 0.00% 0.00% 0.00% 0 Multilink event
62 0 1 0 0.00% 0.00% 0.00% 0 FR LMI
63 4 2849 1 0.00% 0.00% 0.00% 0 FR Broadcast Out
64 0 1 0 0.00% 0.00% 0.00% 0 FR ARP
65 0 17 0 0.00% 0.00% 0.00% 0 FR TUNNEL
66 0 1 0 0.00% 0.00% 0.00% 0 FRF9 manager
67 0 1 0 0.00% 0.00% 0.00% 0 FRF9 timed event
You can see cpu at 100%, but I don't see any specific process using it.
I discovered something else...under my normal testing conditions where I wiped the config, reloaded the switch, added ip addresses to the ethernet interfaces only...as can be seen below:
Current configuration : 601 bytes
!
version 12.1
no service single-slot-reload-enable
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname Router
!
logging rate-limit console 10 except errors
!
ip subnet-zero
!
!
no ip finger
!
!
!
!
interface FastEthernet0/0
ip address 192.168.1.1 255.255.255.0
duplex auto
speed auto
!
interface Serial0/0
no ip address
shutdown
!
interface FastEthernet0/1
ip address 192.168.2.1 255.255.255.0
duplex auto
speed auto
!
ip classless
no ip http server
!
!
line con 0
transport input none
line aux 0
line vty 0 4
!
no scheduler allocate
end
I get the 100% cpu and console session hanging up issue, but the 300mb file will transfer in 40sec. If I next save the config and reload the switch, then try transfering the same 300mb file results in the cpu util is low, the console session does not freeze, but the file takes 35min to transfer. Traffic through the router is incredibly slow in that situation.
Any ideas?
04-08-2011 12:21 AM
This is from the Cisco output interpreter:
SHOW PROCESS CPU NOTIFICATIONS (if any)
INFO: Total CPU Utilization is comprised of process and interrupt percentages.
Total CPU Utilization: 100%
Process Utilization: 1%
Interrupt Utilization: 99%
These values are found on the first line of the output:
CPU utilization for five seconds: x%/y%; one minute: a%; five minutes: b%
Total CPU Utilization: x%
Process Utilization: (x - y)%
Interrupt Utilization: y%
Process Utilization is the difference between the Total and Interrupt (x and y).
The one and five minute utilizations are exponentially decayed averages (rather
than an arithmetic average), therefore recent values have more influence on the
calculated average.
ERROR: Total CPU Utilization is at 100% for the past 5 seconds, which is very
high (>90%).
This can cause the following symptoms:
- Input queue drops
- Slow performance
- Slow response in Telnet or unable to Telnet to the router
- Slow response on the console
- Slow or no response to ping
- Router doesn't send routing updates
The following processes are causing excessive CPU usage:
PID CPU Time Process
ERROR: No Processes were found having a CPU utilization greater than 10%
NOTE: If there are processes with a utilization greater than 10%, ensure that
there is no line wrapping as this will interfere with the analysis.
REFERENCE: For troubleshooting information, please visit Troubleshooting High
CPU Utilization on Cisco Routers
WARNING: Interrupt CPU Utilization is 99%, which is very high (>60%)
CPU interrupts are primarily caused by fast switching of traffic. Interrupts are
also generated any time a character is output from the console or auxiliary ports
of a router.
TRY THIS: One of the following may be causing this to happen:
- Configured voice ports: Even if there is no traffic, software continues to
monitor channel associated signaling (CAS).
- Active ATM interfaces: Even with no traffic, ATM interfaces continue to send
null cells (per ATM standard).
- An inappropriate switching path is configured on the router. If you have a
Cisco 7000 or Cisco 7500 series router, try improving its performance by
using the 'ip route-cache {path}' command (path can be cef, distributed, or
cbus, depending on the platform). If there are access lists linked to
interfaces or if ip accounting is turned on, configure NetFlow switching
using the 'ip route-cache flow' command.
- The CPU is performing memory alignment corrections. If there are
%ALIGN-3-CORRECT messages logged, then the high CPU utilization is caused by
memory alignment corrections. Capture the output of the 'show align' command,
decode the tracebacks and search for a bug in your version of IOS.
- The router is overloaded with traffic. Use the 'show interfaces' command and
paste into Output Interpreter to determine which interface is overloaded
- There maybe a bug in the version of IOS running on the router. Check the
Bug Navigator for a bug that reports similar symptoms in a similar environment.
REFERENCE: For additional troubleshooting information, please visit Troubleshooting
High CPU Utilization on Cisco Routers
i.e. as you said, there doesn't appear to be anything wrong. Check out this doc. but I don't know to be honest.
http://www.cisco.com/en/US/customer/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml#ts_strat
05-04-2011 01:40 PM
Interrupt load is predominantly related to traffic handling. Though that is not always the answer.
Look at the 'show stacks'. The bottom part of that are the interrupt stacks. When there is an interrupt a stack is pushed to save the current status, the interrupt is handled, and the stack is popped to proceed where the processor was interrupted.
The show stacks can give you a rough idea of what was causing interrupts. Typically the interfaces generate the bulk of CPU impacting interrupts.
Look at 'show int switching'. In this old code this was a hidden command, so type out the word 'switching'. Per interface this command shows you how the system is switching traffic.
On this platform the switching path of fast or CEF switching is about 17 times faster and more capable than process switching.
Also in this older code you have to configure CEF switching to enable it. The global config t command to do this is 'ip cef'. Try adding this to your configuration and see if that improves the CPU load. Note in this old code there may be CEF related bugs, I do not know.
If the traffic is process switching then use 'show ip traffic' to get an idea of the reason. If you put that comand here I can explain some of what it tells you.
I really would not run the 12.1T images. They are far beyond obsolete and a cursory search against that image family and the keywords 'high interrupt cpu' finds a number of bugs and prior cases.
........Mike
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide