11-22-2003 09:26 AM - edited 03-02-2019 11:55 AM
We have a high cpu utilization problem on our Internet 7513 router.
ios : rsp-jsv-mz.121-20.bin
interface use distributed CEF
Help me. please... T.T
------------------ show process cpu ------------------
CPU utilization for five seconds: 99%/97%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 14340 1802506 7 0.00% 0.00% 0.00% 0 Load Meter
2 169512 35626500 4 0.00% 0.02% 0.00% 0 CEF IPC Backgrou
3 16545528 1109470 14913 0.00% 0.22% 0.15% 0 Check heaps
4 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager
5 6592 5030 1310 0.00% 0.00% 0.00% 0 Pool Manager
6 0 2 0 0.00% 0.00% 0.00% 0 Timers
7 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
8 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler
9 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager
10 274696 8977305 30 0.00% 0.00% 0.00% 0 IPC Periodic Tim
11 182736 8977305 20 0.00% 0.00% 0.00% 0 IPC Deferred Por
12 12751492 127646600 99 0.08% 0.09% 0.09% 0 IPC Seat Manager
13 3090548 4663415 662 0.00% 0.01% 0.00% 0 ARP Input
14 43772 2245769 19 0.00% 0.00% 0.00% 0 HC Counter Timer
15 36 334 107 0.00% 0.00% 0.00% 0 DDR Timers
16 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
17 0 1 0 0.00% 0.00% 0.00% 0 Entity MIB API
18 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
19 0 1 0 0.00% 0.00% 0.00% 0 Microcode Loader
20 0 2 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
21 291912 2108017 138 0.00% 0.00% 0.00% 0 Net Background
22 40 1289 31 0.00% 0.00% 0.00% 0 Logger
23 277868 8977299 30 0.00% 0.00% 0.00% 0 TTY Background
24 10311952 8977305 1148 0.00% 0.00% 0.00% 0 Per-Second Jobs
25 72 580 124 0.00% 0.00% 0.00% 0 Inode Table Dest
26 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Refr
27 0 1 0 0.00% 0.00% 0.00% 0 stuckinfo_proces
28 0 1 0 0.00% 0.00% 0.00% 0 IP Crashinfo Inp
29 131556 9014473 14 0.00% 0.00% 0.00% 0 DSX3MIB ll handl
30 0 2 0 0.00% 0.00% 0.00% 0 VSI Master
31 1700560 8977306 189 0.08% 0.02% 0.00% 0 RSP Background
32 0 1 0 0.00% 0.00% 0.00% 0 Memory Scanner
33 7012 150252 46 0.00% 0.00% 0.00% 0 Slave Time
34 0 1 0 0.00% 0.00% 0.00% 0 Slave IPC OIR
35 358272 396 904727 0.00% 0.00% 0.05% 2 Virtual Exec
36 60512 2549980 23 0.00% 0.00% 0.00% 0 Chassis Daemon
37 8512 150260 56 0.00% 0.00% 0.00% 0 RSP Chassis Back
38 0 1 0 0.00% 0.00% 0.00% 0 MIP Mailbox
39 0 1 0 0.00% 0.00% 0.00% 0 vcq_proc
40 0 1 0 0.00% 0.00% 0.00% 0 CT3 Mailbox
41 0 1 0 0.00% 0.00% 0.00% 0 CE3 Mailbox
42 6486376 129502902 50 0.08% 0.06% 0.06% 0 IPC CBus process
43 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM Input
44 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER
45 27566712 37112358 742 0.08% 0.10% 0.12% 0 IP Input
46 0 1 0 0.00% 0.00% 0.00% 0 DSS Process
47 424 15007 28 0.00% 0.00% 0.00% 0 MOP Protocols
48 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
49 576 150314 3 0.00% 0.00% 0.00% 0 LDP Background
50 0 1 0 0.00% 0.00% 0.00% 0 frr_tunnel
51 2826764 160401 17623 0.00% 0.02% 0.00% 0 IP Background
52 420040 305166 1376 0.00% 0.00% 0.00% 0 IP RIB Update
53 0 1 0 0.00% 0.00% 0.00% 0 SNMP Timers
54 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add Route
55 24156 1013504 23 0.00% 0.00% 0.00% 0 TCP Timer
56 1192 899 1325 0.00% 0.00% 0.00% 0 TCP Protocols
57 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
58 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
59 0 1 0 0.00% 0.00% 0.00% 0 HTTP Timer
60 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
61 16 13 1230 0.00% 0.00% 0.00% 0 DHCPD Receive
62 959060 151078 6348 0.00% 0.00% 0.00% 0 IP Cache Ager
63 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
64 4 2 2000 0.00% 0.00% 0.00% 0 X.25 Background
65 11700 150260 77 0.00% 0.00% 0.00% 0 TCP Intercept Ti
66 0 2 0 0.00% 0.00% 0.00% 0 SPX Input
67 84928 150312 565 0.00% 0.00% 0.00% 0 Adj Manager
68 1046232 13183072 79 0.00% 0.01% 0.00% 0 CEF process
69 286732 44670692 6 0.00% 0.01% 0.00% 0 MDFS RP process
70 4 2 2000 0.00% 0.00% 0.00% 0 TC-ATM Proc
71 0 2 0 0.00% 0.00% 0.00% 0 Tag Input
72 50772 506 100339 0.47% 0.81% 0.39% 0 Exec
73 4 1 4000 0.00% 0.00% 0.00% 0 SYSMGT Events
74 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
75 28516 17939497 1 0.00% 0.00% 0.00% 0 cbus utilization
76 585928 4034288 145 0.00% 0.00% 0.00% 0 Net Input
77 27340 1802507 15 0.00% 0.00% 0.00% 0 Compute load avg
78 2366268 150259 15747 0.00% 0.01% 0.00% 0 Per-minute Jobs
79 21834992 7233025 3018 0.00% 0.07% 0.20% 0 IP SNMP
80 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
81 4 3 1333 0.00% 0.00% 0.00% 0 SNMP Traps
82 0 2 0 0.00% 0.00% 0.00% 0 IP Flow Backgrou
83 2381796 19318529 123 0.00% 0.07% 0.06% 0 IP-EIGRP Hello
85 0 9 0 0.00% 0.00% 0.00% 0 TCP Listener
86 47412 9017589 5 0.00% 0.00% 0.00% 0 NTP
87 20304 475045 42 0.00% 0.00% 0.00% 0 CEF Scanner
89 344 75153 4 0.00% 0.00% 0.00% 0 DHCPD Timer
90 38716 2554184 15 0.00% 0.00% 0.00% 0 DHCPD Database
91 2557308 9695069 263 0.00% 0.00% 0.00% 0 IP-EIGRP Router
92 4984536 23393200 213 0.00% 0.05% 0.07% 0 BGP Router
93 874020 1616454 540 0.00% 0.00% 0.00% 0 BGP I/O
94 192078900 1222840 157083 0.00% 0.66% 0.84% 0 BGP Scanner
thanks for reading..
11-22-2003 01:44 PM
Hello,
do you have a CCO logon ? I checked the bug toolkit and there are a couple of bugs in 12.1-20 which might apply to you, but since I haven´t seen your config I cannot say for sure. Check out this link for the bugs related to high CPU utilization in version 12.1-20:
You might also have hit the Blaster or Nachi worm, which causes high CPU utilization as well. Check out these links:
For the Nachi worm:
http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_tech_note09186a00801b143a.shtml
For the Blaster worm:
http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_tech_note09186a00801aedd6.shtml
And last but not least, since there doesn´t seem to be a specific process responsible for the CPU utilization, it might be due to high interrupts. Check out this document to troubleshoot high CPU utilization on 7500 routers due to interrupts:
Lots of documents...
Regards,
Georg
11-22-2003 06:19 PM
It is indeed due to interrupts, which generally means inappropriate switching path (i.e., not CEF) and/or too many packets. If CEF is enabled and being used for most packets (look at 'show interface switching' or 'show cef not-cef-switched' to confirm) then there may be worms of some sort on your network causing a lot of excess traffic.
11-23-2003 02:50 PM
I would chek the output of show alignment, and if the numbers there are not high and increasing, then it's definitely traffic. I can't say this is high, since I don't know the "normal" traffic load on the router, but if it is high, then I'd agree that it's probably some sort of worm on your network/etc.
:-)
Russ.W
11-24-2003 05:49 AM
I can't check the output of show alignment..T.T
7513router#sh alignment
No alignment data has been recorded.
No spurious memory references have been recorded.
7513router#
11-24-2003 08:52 AM
Hi,
Could you check if you have CEF enabled on the router. CEF is less CPU-intensive than fast/process switching.
If it is not enabled the output would be as follows:
Router#sh ip cef
%CEF not running
CEF can be enabled using the global config mode:
Router(config)#ip cef
CEF Overview:
Configuring CEF:
Hope this helps,
Rajesh
11-25-2003 03:34 AM
Actually, that just means there are no alignment errors or spurious accesses, which means it's pure switching traffic taking up the processor time. It's just a straight traffic issue, which implies either a worm of some type, or perhaps a lot of stuff being done in the CEF path, or perhaps just a lot of traffic.
What sort of features do you have configured on the interfaces on this router? How much traffic load is it handling?
Russ.W
11-27-2003 03:50 AM
7513router all cef switching..use..
sh conf
...
boot system disk0:rsp-isv-mz.121-20.bin
boot system disk0:rsp-jsv-mz.122-15.T5.bin
enable secret xxxx
...
ip cef distributed
....
7513router#sh ip int
GigabitEthernet3/0/0 is up, line protocol is up
Directed broadcast forwarding is disabled
Multicast reserved groups joined: 224.0.0.10
Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are never sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is enabled
IP CEF switching is enabled
IP Distributed switching is enabled
IP Flow switching turbo vector
IP Flow CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, Flow, Distributed, CEF
...
Serial8/1/0 is up, line protocol is up
Directed broadcast forwarding is disabled
Multicast reserved groups joined: 224.0.0.10
Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are never sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is enabled
IP Flow switching is enabled
IP CEF switching is enabled
IP Distributed switching is enabled
IP Flow switching turbo vector
IP Flow CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, Flow, Distributed, CEF
...
Serial8/1/1 is administratively down, line protocol is down
Internet protocol processing disabled
...
FastEthernet10/0/0 is up, line protocol is up
Directed broadcast forwarding is disabled
Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are never sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is enabled
IP CEF switching is enabled
IP Distributed switching is enabled
IP Flow switching turbo vector
IP Flow CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, Flow, Distributed, CEF
...
FastEthernet10/1/0 is up, line protocol is up
Directed broadcast forwarding is disabled
Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are never sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is disabled
IP Flow switching is enabled
IP CEF switching is enabled
IP Distributed switching is enabled
IP Flow switching turbo vector
IP Flow CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, Flow, Distributed, CEF
...
Serial12/0 is up, line protocol is up
Directed broadcast forwarding is disabled
Multicast reserved groups joined: 224.0.0.10
Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are never sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is enabled
IP Flow switching is enabled
IP CEF switching is enabled
IP Flow switching turbo vector
IP Flow CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, Flow, CEF, No Distributed
...
Serial12/1 is up, line protocol is up
Directed broadcast forwarding is disabled
Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are never sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is enabled
IP Flow switching is disabled
IP CEF switching is enabled
IP Feature Fast switching turbo vector
IP Feature CEF switching turbo vector
IP multicast fast switching is disabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, CEF, No Distributed
...
Serial12/2 is up, line protocol is up
Directed broadcast forwarding is disabled
Multicast reserved groups joined: 224.0.0.10
Outgoing access list is no_icmp
Inbound access list is not set
Proxy ARP is disabled
Security level is default
Split horizon is enabled
ICMP redirects are never sent
ICMP unreachables are never sent
ICMP mask replies are never sent
IP fast switching is enabled
IP fast switching on the same interface is enabled
IP Flow switching is enabled
IP CEF switching is enabled
IP Flow switching turbo vector
IP Flow CEF switching turbo vector
IP multicast fast switching is enabled
IP multicast distributed fast switching is disabled
IP route-cache flags are Fast, Flow, CEF, No Distributed
...
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide