cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1606
Views
0
Helpful
6
Replies

2621 Router

onetwothreesix
Level 1
Level 1

I recently had a 2621 router that was hanging up at 100% cpu utilization when large data files were transfered across the network.  I replaced the router with a spare 2621 and the problem went away.  I decided to play around with the problem router to see what happened.  I connected a PC to each ethernet port on the router and attempted to transfer a 300mb file between them while consoled into the router.  The file transfer went so slow that after 20min it was only 10 percent completed, but the cpu never went over 8% during that time.  I upgraded the IOS with the same results.  Since the original config had ip routes and an access-list, I next wiped the configuration only adding back an ip address on each ethernet interface...nice and clean.  This time the file transferred normally as it should, but the console session locked up until the transfer completed.  The cpu reached 100% utilization transferring that single file.  I tried another spare 2621 router with a clean config and it also locked up with 100% cpu.

Am I missing something?  I am currently working towards my CCNA so I still have alot more to learn when it comes to networking.  I am hoping someone can shed some light.

6 Replies 6

IAN WHITMORE
Level 4
Level 4

It might be a bug. Please post your #sh version here.

I tested this scenario on two different 2621 routers with the same results.  Below are both sh ver from each router:

Router#sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-D-M), Version 12.1(5)T,  RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Sat 11-Nov-00 12:18 by ccai
Image text-base: 0x80008088, data-base: 0x80A88C1C

ROM: System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)

Router uptime is 3 minutes
System returned to ROM by power-on
System image file is "flash:c2600-d-mz.121-5.T"

cisco 2621 (MPC860) processor (revision 0x102) with 27648K/5120K bytes of memory.
Processor board ID JAD045257KI (2663601130)
M860 processor: part number 0, mask 49
Bridging software.
X.25 software, Version 3.0.0.
2 FastEthernet/IEEE 802.3 interface(s)
1 Serial network interface(s)
32K bytes of non-volatile configuration memory.
8192K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102


Router#sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-I-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Mon 17-Mar-08 15:23 by dchih

ROM: System Bootstrap, Version 12.2(6r),  RELEASE SOFTWARE (fc1)

Router uptime is 0 minutes
System returned to ROM by power-on
System image file is "flash:c2600-i-mz.123-26.bin"

cisco 2621 (MPC860) processor (revision 0x00) with 28672K/4096K bytes of memory.
Processor board ID JAD062206DQ (2653152028)
M860 processor: part number 0, mask 49
Bridging software.
X.25 software, Version 3.0.0.
2 FastEthernet/IEEE 802.3 interface(s)
32K bytes of non-volatile configuration memory.
8192K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

Since it is happening on two routers, it makes me think it is me and not the hardware/IOS.  Any help would be greatly appreciated.

I know you say the console session locks when you try the transfer but can you try and so #sh proc cpu before it locks up? I didn't a specific bug related to this in either the 12.1 or 12.3 version.

Also, do you see anything in the log files?

Okay, so here is what I did:

(Keep in mind I tried everything on both 2621 routers and got the same results)

Since I knew the console session would hang up during the file transfer, I pre-typed sh proc cpu so I could just tap the enter key and hope it goes through.  I started the file transfer, the console hanged, I hit the enter key, and when the file transfer completed...this is what I got:

Router#sh proc cpu
CPU utilization for five seconds: 100%/99%; one minute: 15%; five minutes: 3%
PID  Runtime(ms)  Invoked  uSecs    5Sec   1Min   5Min TTY Process
   1           4        51     78   0.04%  0.00%  0.00%   0 Load Meter
   2        1344       193   6963   0.00%  0.17%  0.25%   0 Exec
   3         100        27   3703   0.00%  0.02%  0.00%   0 Check heaps
   4           4         1   4000   0.00%  0.00%  0.00%   0 Chunk Manager
   5           4         2   2000   0.00%  0.00%  0.00%   0 Pool Manager
   6           0         2      0   0.00%  0.00%  0.00%   0 Timers
   7           0         2      0   0.00%  0.00%  0.00%   0 Serial Backgroun
   8           0        36      0   0.00%  0.00%  0.00%   0 ALARM_TRIGGER_SC
   9           4        10    400   0.00%  0.00%  0.00%   0 Environmental mo
  10          28        43    651   0.00%  0.00%  0.00%   0 ARP Input
  11           0         2      0   0.00%  0.00%  0.00%   0 DDR Timers
  12           0         2      0   0.00%  0.00%  0.00%   0 Dialer event
  13           4         3   1333   0.00%  0.00%  0.00%   0 Entity MIB API
  14           0         1      0   0.00%  0.00%  0.00%   0 SERIAL A'detect
  15           0         1      0   0.00%  0.00%  0.00%   0 Critical Bkgnd
  16          16       133    120   0.00%  0.00%  0.00%   0 Net Background
  17           4        14    285   0.00%  0.00%  0.00%   0 Logger
  18        9556       239  39983   0.00%  0.00%  0.00%   0 TTY Background
  19           0       285      0   0.00%  0.00%  0.00%   0 Per-Second Jobs
  20           0         2      0   0.00%  0.00%  0.00%   0 Hawkeye Backgrou
  22           0         1      0   0.00%  0.00%  0.00%   0 Net Input
  23           0        52      0   0.00%  0.00%  0.00%   0 Compute load avg
PID  Runtime(ms)  Invoked  uSecs    5Sec   1Min   5Min TTY Process
  24          84         6  14000   0.00%  0.01%  0.00%   0 Per-minute Jobs
  25           4         2   2000   0.00%  0.00%  0.00%   0 Service-module a
  26         160       347    461   0.00%  0.03%  0.01%   0 IP Input
  27           8        29    275   0.00%  0.00%  0.00%   0 CDP Protocol
  28           0         3      0   0.00%  0.00%  0.00%   0 MOP Protocols
  29           0         1      0   0.00%  0.00%  0.00%   0 X.25 Encaps Mana
  30           0         1      0   0.00%  0.00%  0.00%   0 Asy FS Helper
  31           0         1      0   0.00%  0.00%  0.00%   0 PPP IP Add Route
  32           0        38      0   0.00%  0.00%  0.00%   0 IP Background
  33           0         6      0   0.00%  0.00%  0.00%   0 Adj Manager
  34           0         1      0   0.00%  0.00%  0.00%   0 TCP Timer
  35           0         1      0   0.00%  0.00%  0.00%   0 TCP Protocols
  36           4         1   4000   0.00%  0.00%  0.00%   0 Probe Input
  37           4         1   4000   0.00%  0.00%  0.00%   0 RARP Input
  38           0         1      0   0.00%  0.00%  0.00%   0 HTTP Timer
  39           0         1      0   0.00%  0.00%  0.00%   0 Socket Timers
  40           4         2   2000   0.00%  0.00%  0.00%   0 DHCPD Receive
  41           0         6      0   0.00%  0.00%  0.00%   0 IP Cache Ager
  42           0         1      0   0.00%  0.00%  0.00%   0 COPS
  43           0         1      0   0.00%  0.00%  0.00%   0 PAD InCall
  44           0         3      0   0.00%  0.00%  0.00%   0 X.25 Background
  45           0         1      0   0.00%  0.00%  0.00%   0 ISDN Timer
PID  Runtime(ms)  Invoked  uSecs    5Sec   1Min   5Min TTY Process
  47           4       310     12   0.00%  0.00%  0.00%   0 Router Autoconf
  48           0         1      0   0.00%  0.00%  0.00%   0 ISDNMIB Backgrou
  49           0         1      0   0.00%  0.00%  0.00%   0 CallMIB Backgrou
  50           0         1      0   0.00%  0.00%  0.00%   0 SNMP ConfCopyPro
  51           0         1      0   0.00%  0.00%  0.00%   0 Syslog Traps
  52           0         3      0   0.00%  0.00%  0.00%   0 DHCPD Timer
  54           0        29      0   0.00%  0.00%  0.00%   0 DHCP Client
  55           0        70      0   0.00%  0.00%  0.00%   0 DHCPD Database
  57           4         2   2000   0.00%  0.00%  0.00%   0 CCP manager
  58           0         2      0   0.00%  0.00%  0.00%   0 PPP manager
  59           4       262     15   0.00%  0.00%  0.00%   0 Multilink PPP
  60           0         2      0   0.00%  0.00%  0.00%   0 Multilink PPP ou
  61           0         2      0   0.00%  0.00%  0.00%   0 Multilink event
  62           0         1      0   0.00%  0.00%  0.00%   0 FR LMI
  63           4      2849      1   0.00%  0.00%  0.00%   0 FR Broadcast Out
  64           0         1      0   0.00%  0.00%  0.00%   0 FR ARP
  65           0        17      0   0.00%  0.00%  0.00%   0 FR TUNNEL
  66           0         1      0   0.00%  0.00%  0.00%   0 FRF9 manager
  67           0         1      0   0.00%  0.00%  0.00%   0 FRF9 timed event

You can see cpu at 100%, but I don't see any specific process using it.

I discovered something else...under my normal testing conditions where I wiped the config, reloaded the switch, added ip addresses to the ethernet interfaces only...as can be seen below:

Current configuration : 601 bytes
!
version 12.1
no service single-slot-reload-enable
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname Router
!
logging rate-limit console 10 except errors
!
ip subnet-zero
!
!
no ip finger
!
!
!
!
interface FastEthernet0/0
ip address 192.168.1.1 255.255.255.0
duplex auto
speed auto
!
interface Serial0/0
no ip address
shutdown
!
interface FastEthernet0/1
ip address 192.168.2.1 255.255.255.0
duplex auto
speed auto
!
ip classless
no ip http server
!
!
line con 0
transport input none
line aux 0
line vty 0 4
!
no scheduler allocate
end

I get the 100% cpu and console session hanging up issue, but the 300mb file will transfer in 40sec.  If I next save the config and reload the switch, then try transfering the same 300mb file results in the cpu util is low, the console session does not freeze, but the file takes 35min to transfer.  Traffic through the router is incredibly slow in that situation.

Any ideas?

This is from the Cisco output interpreter:

SHOW PROCESS CPU NOTIFICATIONS (if any)

INFO: Total CPU Utilization is comprised of process and interrupt percentages.
  Total CPU Utilization: 100%
  Process Utilization: 1%
  Interrupt Utilization: 99%
These values are found on the first line of the output:
 CPU utilization for five seconds: x%/y%; one minute: a%; five minutes: b%
  Total CPU Utilization: x%
  Process Utilization: (x - y)%
  Interrupt Utilization: y%
Process Utilization is the difference between the Total and Interrupt (x and y).
The one and five minute utilizations are exponentially decayed averages (rather
than an arithmetic average), therefore recent values have more influence on the
calculated average.

ERROR: Total CPU Utilization is at 100% for the past 5 seconds, which is very
high (>90%).
This can cause the following symptoms:
 - Input queue drops
 - Slow performance
 - Slow response in Telnet or unable to Telnet to the router
 - Slow response on the console
 - Slow or no response to ping
 - Router doesn't send routing updates
The following processes are causing excessive CPU usage:
 PID     CPU Time     Process
ERROR: No Processes were found having a CPU utilization greater than 10%
NOTE: If there are processes with a utilization greater than 10%, ensure that
there is no line wrapping as this will interfere with the analysis.
REFERENCE: For troubleshooting information, please visit Troubleshooting High
CPU Utilization on Cisco Routers

WARNING: Interrupt CPU Utilization is 99%, which is very high (>60%)
CPU interrupts are primarily caused by fast switching of traffic. Interrupts are
also generated any time a character is output from the console or auxiliary ports
of a router.
TRY THIS: One of the following may be causing this to happen:
 - Configured voice ports: Even if there is no traffic, software continues to
   monitor channel associated signaling (CAS).
 - Active ATM interfaces: Even with no traffic, ATM interfaces continue to send
   null cells (per ATM standard).
 - An inappropriate switching path is configured on the router. If you have a
   Cisco 7000 or Cisco 7500 series router, try improving its performance by
   using the 'ip route-cache {path}' command (path can be cef, distributed, or
   cbus, depending on the platform). If there are access lists linked to
   interfaces or if ip accounting is turned on, configure NetFlow switching
   using the 'ip route-cache flow' command.
 - The CPU is performing memory alignment corrections. If there are
   %ALIGN-3-CORRECT messages logged, then the high CPU utilization is caused by
   memory alignment corrections. Capture the output of the 'show align' command,
   decode the tracebacks and search for a bug in your version of IOS.
 - The router is overloaded with traffic. Use the 'show interfaces' command and
   paste into Output Interpreter to determine which interface is overloaded
 - There maybe a bug in the version of IOS running on the router. Check the 
Bug Navigator   for a bug that reports similar symptoms in a similar environment.
REFERENCE: For additional troubleshooting information, please visit Troubleshooting
High CPU Utilization on Cisco Routers

i.e. as you said, there doesn't appear to be anything wrong. Check out this doc. but I don't know to be honest.

http://www.cisco.com/en/US/customer/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml#ts_strat

Interrupt load is predominantly related to traffic handling.  Though that is not always the answer.

Look at the 'show stacks'.  The bottom part of that are the interrupt stacks.  When there is an interrupt a stack is pushed to save the current status, the interrupt is handled, and the stack is popped to proceed where the processor was interrupted.

The show stacks can give you a rough idea of what was causing interrupts.  Typically the interfaces generate the bulk of CPU impacting interrupts.

Look at 'show int switching'.  In this old code this was a hidden command, so type out the word 'switching'. Per interface this command shows you how the system is switching traffic.

On this platform the switching path of fast or CEF switching is about 17 times faster and more capable than process switching.

Also in this older code you have to configure CEF switching to enable it.  The global config t command to do this is 'ip cef'.  Try adding this to your configuration and see if that improves the CPU load.  Note in this old code there may be CEF related bugs, I do not know. 

If the traffic is process switching then use 'show ip traffic' to get an idea of the reason.  If you put that comand here I can explain some of what it tells you.

I really would not run the 12.1T images.  They are far beyond obsolete and a cursory search against that image family and the keywords 'high interrupt cpu' finds a number of bugs and prior cases.

........Mike

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: