- Cisco Community
- Technology and Support
- Networking
- Routing
- High CPU Utlisation on 7206VXR due to traffic?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
High CPU Utlisation on 7206VXR due to traffic?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-20-2011 10:06 AM - edited 03-04-2019 12:08 PM
I have a 7206VXR router doing policy routing with CEF enabled on all interfaces, no Cache Misses, Align Errors or failed buffers. CPU peaks up to 80% every two or three minutes. I suspect it is traffic related as I have a standby box and when it becomes the active on in the HSRP pair to high CPU load and LAN interface input errors follows it. see various show command output below. The router has a NPE-G1 and I'm using a gigabit port off of this on the LAn side and a E3 Frame Relay interface on the WAN side.
UK-DHC-Policy1#show buffers
Buffer elements:
1118 in free list (1000 max allowed)
3539658545 hits, 0 misses, 1119 created
Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50, peak 229 @ 7w0d):
48 in free list (20 min, 150 max allowed)
1720684438 hits, 837 misses, 1533 trims, 1533 created
22 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 55 @ 7w0d):
23 in free list (10 min, 150 max allowed)
98891775 hits, 406 misses, 1204 trims, 1204 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50):
50 in free list (5 min, 150 max allowed)
95778567 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
10 in free list (0 min, 100 max allowed)
18523488 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Interface buffer pools:
IPC buffers, 4096 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
Header pools:
Header buffers, 0 bytes (total 511, permanent 256, peak 511 @ 7w0d):
255 in free list (256 min, 1024 max allowed)
171 hits, 85 misses, 0 trims, 255 created
0 failures (0 no memory)
256 max cache size, 256 in cache
2108702127 hits in cache, 0 misses in cache
Particle Clones:
1024 clones, 42083382 hits, 0 misses
Public particle pools:
F/S buffers, 128 bytes (total 512, permanent 512):
0 in free list (0 min, 512 max allowed)
512 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
512 max cache size, 512 in cache
42083382 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 2048, permanent 2048):
2048 in free list (1024 min, 4096 max allowed)
5086501 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Private particle pools:
GigabitEthernet0/1 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
128 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
uksmh-policy2#sh proc cpu sorted
CPU utilization for five seconds: 73%/72%; one minute: 36%; five minutes: 22%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
163 2929932 643557 4552 1.27% 0.10% 0.05% 0 SNMP ENGINE
30 82984 1717990 48 0.15% 0.04% 0.02% 0 Net Background
158 9048 14775 612 0.15% 0.16% 0.04% 2 Virtual Exec
161 53220 1257209 42 0.07% 0.00% 0.00% 0 IP SNMP
5 459820 324954 1415 0.00% 0.01% 0.00% 0 Check heaps
6 12 741 16 0.00% 0.00% 0.00% 0 Pool Manager
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
9 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
10 0 2 0 0.00% 0.00% 0.00% 0 ATM AutoVC Perio
#sh int switching
Interface FastEthernet0/0 is disabled
Interface FastEthernet0/1 is disabled
GigabitEthernet0/1
Throttle count 0
Drops RP 0 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 223415 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 238985 59471001 111572 44606700
Cache misses 7320946 - - -
Fast 844148655 2030865100 738366 88944496
Auton/SSE 0 0 0 0
Protocol CLNS
Switching path Pkts In Chars In Pkts Out Chars Out
Process 39954 57441512 153335 180687731
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 285 21945
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 101 6060 113 6780
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 2867 1287283 2866 1130214
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 17184 1031040
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
Interface GigabitEthernet0/2 is disabled
Interface GigabitEthernet0/3 is disabled
Serial2/0
Throttle count 0
Drops RP 0 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 844922 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 773193 124383404 549859 300238429
Cache misses 2290 - - -
Fast 32075869 2653400782 900062537 3933863886
Auton/SSE 0 0 0 0
Protocol CLNS
Switching path Pkts In Chars In Pkts Out Chars Out
Process 525033 1650667569 1049032 3287269184
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 53263 16511530 106590 39118530
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 53264 15606352 319763 4156920
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
#sh int g0/1
GigabitEthernet0/1 is up, line protocol is up
Hardware is BCM1250 Internal MAC, address is 000d.ed02.291b (bia 000d.ed02.291b)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is RJ45
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 03:49:16
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 6477000 bits/sec, 4354 packets/sec
5 minute output rate 2000 bits/sec, 4 packets/sec
60218511 packets input, 3023533341 bytes, 0 no buffer
Received 6937 broadcasts, 0 runts, 0 giants, 0 throttles
65 input errors, 0 CRC, 0 frame, 65 overrun, 0 ignored
0 watchdog, 34454 multicast, 0 pause input
0 input packets with dribble condition detected
70352 packets output, 13435108 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
#sh int s2/0
Serial2/0 is up, line protocol is up
Hardware is M2T-E3 pa
MTU 4470 bytes, BW 34010 Kbit, DLY 200 usec,
reliability 255/255, txload 45/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive set (10 sec)
Restart-Delay is 0 secs
LMI enq sent 10268, LMI stat recvd 10268, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 1023 LMI type is CISCO frame relay DTE
FR SVC disabled, LAPF state down
Broadcast queue 0/256, broadcasts sent/dropped 18808/0, interface broadcasts 17096
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1d04h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1894900
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 9000 bits/sec, 8 packets/sec
5 minute output rate 6132000 bits/sec, 4442 packets/sec
17942673 packets input, 1153473923 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
512329612 packets output, 1864863937 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
- Labels:
-
Other Routing
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-20-2011 10:09 AM
Correct show buffers output for review.
#sh buffers
Buffer elements:
1118 in free list (1000 max allowed)
29642789 hits, 0 misses, 1119 created
Public buffer pools:
Small buffers, 104 bytes (total 59, permanent 50, peak 183 @ 1d04h):
55 in free list (20 min, 150 max allowed)
4258846 hits, 126 misses, 299 trims, 308 created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 34, permanent 25, peak 40 @ 6d02h):
33 in free list (10 min, 150 max allowed)
1535040 hits, 40 misses, 92 trims, 101 created
1 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50):
50 in free list (5 min, 150 max allowed)
5974124 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
10 in free list (0 min, 100 max allowed)
784125 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Interface buffer pools:
IPC buffers, 4096 bytes (total 2, permanent 2):
2 in free list (1 min, 8 max allowed)
0 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
Header pools:
Header buffers, 0 bytes (total 511, permanent 256, peak 511 @ 5w2d):
255 in free list (256 min, 1024 max allowed)
171 hits, 85 misses, 0 trims, 255 created
0 failures (0 no memory)
256 max cache size, 256 in cache
11711654 hits in cache, 0 misses in cache
Particle Clones:
1024 clones, 0 hits, 0 misses
Public particle pools:
F/S buffers, 128 bytes (total 512, permanent 512):
0 in free list (0 min, 512 max allowed)
512 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
512 max cache size, 512 in cache
0 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 2048, permanent 2048):
2048 in free list (1024 min, 4096 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Private particle pools:
GigabitEthernet0/1 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
878084773 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/2 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
128 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/3 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
128 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
Serial2/0 buffers, 512 bytes (total 256, permanent 256):
0 in free list (0 min, 256 max allowed)
256 hits, 0 fallbacks
256 max cache size, 132 in cache
37815607 hits in cache, 0 misses in cache
10 buffer threshold, 0 threshold transitions
Serial2/1 buffers, 512 bytes (total 256, permanent 256):
0 in free list (0 min, 256 max allowed)
256 hits, 0 fallbacks
256 max cache size, 132 in cache
124 hits in cache, 0 misses in cache
10 buffer threshold, 0 threshold transitions
FastEthernet0/0 buffers, 768 bytes (total 559, permanent 400):
129 in free list (128 min, 1200 max allowed)
387 hits, 64 misses, 657 trims, 816 created
0 failures (0 no memory)
400 max cache size, 302 in cache
44387326 hits in cache, 30 misses in cache
14 buffer threshold, 11011 threshold transitions
FastEthernet0/1 buffers, 768 bytes (total 529, permanent 400):
129 in free list (128 min, 1200 max allowed)
357 hits, 43 misses, 657 trims, 786 created
0 failures (0 no memory)
400 max cache size, 272 in cache
128 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-21-2011 04:00 PM
uksmh-policy2#sh proc cpu sorted
CPU utilization for five seconds: 73%/72%; one minute: 36%; five minutes: 22%
This shows that the high CPU is happening at interrupt level instead of process level, so it is likely traffic related or a feature that doesn't require punts to the processor. To truly know what is using the CPU at interrupt level you'd need to do CPU profiling which only TAC/Cisco Internal could decode.
In this situation a case with TAC would be required.
If that isn't possible we may be able to see what's causing the spikes in other ways.
One that comes to mind is an EEM script that triggers when there is high CPU and shows what the interfaces look like when the spikes occur.
Before you configure the EEM script please change the load-interval on your interfaces to 30 seconds instead of 5 minutes. If you have a burst of traffic come into router we might have a problem seeing it in a 30s window (the smallest we can configure) but it's very hard in a 5min window.
The command is, config mode in an interface: load-interval 30
You may need to change the location that the results of this script are stored at in the router. You can change the save location by editing the below script's "flash:" to "disk0:" or whatever location you'd like to target. This script will automatically try to remove itself from the router after it has run, so if you want it to run more than once you would need to reapply the script. Running the script again would append to the high_cpu.txt file, not overwrite it.
config t
event manager applet high_cpu
event snmp oid 1.3.6.1.4.1.9.2.1.56.0 get-type exact entry-op ge entry-val 70 exit-time 300 poll-interval 10
action 1.0 syslog msg "*** CPU Utilization is high, running EEM high_cpu"
action 2.0 cli command "en"
action 2.1 cli command "show clock | append flash:high_cpu.txt"
action 3.0 syslog msg "*** high_cpu *** --> CPU statistics"
action 3.1 cli command "show process cpu history | append flash:high_cpu.txt"
action 3.2 cli command "show process cpu sorted | append flash:high_cpu.txt"
action 4.0 syslog msg "*** high_cpu *** capturing --> Interface Statistics"
action 4.1 cli command "show interface | append flash:high_cpu.txt"
action 5.0 syslog msg "*** Removing EEM script high_cpu.txt from device."
action 5.1 cli command "config t"
action 5.2 cli command "no event manager applet high_cpu.txt"
action 6.0 syslog msg "*** EEM Script Finished. Exiting."
end
If you still don't see some kind of burst of traffic on an interface then I'd suggest looking at your running config for features like long access lists, IP Inspect, NAT, VPNs, etc. All of these features are hardware/CEF switched on contemporary IOSes and so will drive up the CPU at interrupt level as well.
Hope that helps some,
Brian
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:
