cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
Popup Hotspot Using ISR 1000 with WiFi/LTE for Teleworkers and Micro Branchesr
600
Views
0
Helpful
2
Replies
Highlighted
Beginner

High CPU Utlisation on 7206VXR due to traffic?

I have a 7206VXR router doing policy routing with CEF enabled on all interfaces, no Cache Misses, Align Errors or failed buffers. CPU peaks up to 80% every two or three minutes. I suspect it is traffic related as I have a standby box and when it becomes the active on in the HSRP pair to high CPU load and LAN interface input errors follows it. see various show command output below. The router has a NPE-G1 and I'm using a gigabit port off of this on the LAn side and a E3 Frame Relay interface on the WAN side.

UK-DHC-Policy1#show buffers
Buffer elements:
     1118 in free list (1000 max allowed)
     3539658545 hits, 0 misses, 1119 created

Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50, peak 229 @ 7w0d):
     48 in free list (20 min, 150 max allowed)
     1720684438 hits, 837 misses, 1533 trims, 1533 created
     22 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 55 @ 7w0d):
     23 in free list (10 min, 150 max allowed)
     98891775 hits, 406 misses, 1204 trims, 1204 created
     0 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50):
     50 in free list (5 min, 150 max allowed)
     95778567 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
     10 in free list (0 min, 100 max allowed)
     18523488 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
     0 in free list (0 min, 10 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
     0 in free list (0 min, 4 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Interface buffer pools:
IPC buffers, 4096 bytes (total 2, permanent 2):
     2 in free list (1 min, 8 max allowed)
     0 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 511, permanent 256, peak 511 @ 7w0d):
     255 in free list (256 min, 1024 max allowed)
     171 hits, 85 misses, 0 trims, 255 created
     0 failures (0 no memory)
     256 max cache size, 256 in cache
     2108702127 hits in cache, 0 misses in cache

Particle Clones:
     1024 clones, 42083382 hits, 0 misses

Public particle pools:
F/S buffers, 128 bytes (total 512, permanent 512):
     0 in free list (0 min, 512 max allowed)
     512 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     42083382 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 2048, permanent 2048):
     2048 in free list (1024 min, 4096 max allowed)
     5086501 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Private particle pools:
GigabitEthernet0/1 buffers, 512 bytes (total 1000, permanent 1000):
     0 in free list (0 min, 1000 max allowed)
     1000 hits, 0 fallbacks
     1000 max cache size, 872 in cache
     128 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions

uksmh-policy2#sh proc cpu sorted
CPU utilization for five seconds: 73%/72%; one minute: 36%; five minutes: 22%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
163     2929932    643557       4552  1.27%  0.10%  0.05%   0 SNMP ENGINE
  30       82984   1717990         48  0.15%  0.04%  0.02%   0 Net Background
158        9048     14775        612  0.15%  0.16%  0.04%   2 Virtual Exec
161       53220   1257209         42  0.07%  0.00%  0.00%   0 IP SNMP
   5      459820    324954       1415  0.00%  0.01%  0.00%   0 Check heaps
   6          12       741         16  0.00%  0.00%  0.00%   0 Pool Manager
   7           0         2          0  0.00%  0.00%  0.00%   0 Timers
   8           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
   9           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer
  10           0         2          0  0.00%  0.00%  0.00%   0 ATM AutoVC Perio

#sh int switching

Interface FastEthernet0/0 is disabled


Interface FastEthernet0/1 is disabled

GigabitEthernet0/1
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs     223415      Drops          0

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     238985   59471001     111572   44606700
            Cache misses    7320946          -          -          -
                    Fast  844148655 2030865100     738366   88944496
               Auton/SSE          0          0          0          0

    Protocol  CLNS
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      39954   57441512     153335  180687731
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  DEC MOP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0        285      21945
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process        101       6060        113       6780
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  CDP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process       2867    1287283       2866    1130214
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      17184    1031040
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.

Interface GigabitEthernet0/2 is disabled


Interface GigabitEthernet0/3 is disabled

Serial2/0
          Throttle count          0
                   Drops         RP          0         SP          0
             SPD Flushes       Fast          0        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs     844922      Drops          0

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     773193  124383404     549859  300238429
            Cache misses       2290          -          -          -
                    Fast   32075869 2653400782  900062537 3933863886
               Auton/SSE          0          0          0          0

    Protocol  CLNS
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     525033 1650667569    1049032 3287269184
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  CDP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      53263   16511530     106590   39118530
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process      53264   15606352     319763    4156920
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.

#sh int g0/1
GigabitEthernet0/1 is up, line protocol is up
  Hardware is BCM1250 Internal MAC, address is 000d.ed02.291b (bia 000d.ed02.291b)
    MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 03:49:16
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 6477000 bits/sec, 4354 packets/sec
  5 minute output rate 2000 bits/sec, 4 packets/sec
     60218511 packets input, 3023533341 bytes, 0 no buffer
     Received 6937 broadcasts, 0 runts, 0 giants, 0 throttles
     65 input errors, 0 CRC, 0 frame, 65 overrun, 0 ignored
     0 watchdog, 34454 multicast, 0 pause input
     0 input packets with dribble condition detected
     70352 packets output, 13435108 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

#sh int s2/0
Serial2/0 is up, line protocol is up
  Hardware is M2T-E3 pa
  MTU 4470 bytes, BW 34010 Kbit, DLY 200 usec,
     reliability 255/255, txload 45/255, rxload 1/255
  Encapsulation FRAME-RELAY, crc 16, loopback not set
  Keepalive set (10 sec)
  Restart-Delay is 0 secs
  LMI enq sent  10268, LMI stat recvd 10268, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 1023  LMI type is CISCO  frame relay DTE
  FR SVC disabled, LAPF state down
  Broadcast queue 0/256, broadcasts sent/dropped 18808/0, interface broadcasts 17096
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d04h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1894900
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 9000 bits/sec, 8 packets/sec
  5 minute output rate 6132000 bits/sec, 4442 packets/sec
     17942673 packets input, 1153473923 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
              0 parity
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     512329612 packets output, 1864863937 bytes, 0 underruns
     0 output errors, 0 applique, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
   rxLOS inactive, rxLOF inactive, rxAIS inactive
   txAIS inactive, rxRAI inactive, txRAI inactive

2 REPLIES 2
Highlighted
Beginner

Re: High CPU Utlisation on 7206VXR due to traffic?

Correct show buffers output for review.

#sh buffers
Buffer elements:
     1118 in free list (1000 max allowed)
     29642789 hits, 0 misses, 1119 created

Public buffer pools:
Small buffers, 104 bytes (total 59, permanent 50, peak 183 @ 1d04h):
     55 in free list (20 min, 150 max allowed)
     4258846 hits, 126 misses, 299 trims, 308 created
     0 failures (0 no memory)
Middle buffers, 600 bytes (total 34, permanent 25, peak 40 @ 6d02h):
     33 in free list (10 min, 150 max allowed)
     1535040 hits, 40 misses, 92 trims, 101 created
     1 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50):
     50 in free list (5 min, 150 max allowed)
     5974124 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
     10 in free list (0 min, 100 max allowed)
     784125 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
     0 in free list (0 min, 10 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
     0 in free list (0 min, 4 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Interface buffer pools:
IPC buffers, 4096 bytes (total 2, permanent 2):
     2 in free list (1 min, 8 max allowed)
     0 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 511, permanent 256, peak 511 @ 5w2d):
     255 in free list (256 min, 1024 max allowed)
     171 hits, 85 misses, 0 trims, 255 created
     0 failures (0 no memory)
     256 max cache size, 256 in cache
     11711654 hits in cache, 0 misses in cache

Particle Clones:
     1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 128 bytes (total 512, permanent 512):
     0 in free list (0 min, 512 max allowed)
     512 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     0 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 2048, permanent 2048):
     2048 in free list (1024 min, 4096 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Private particle pools:
GigabitEthernet0/1 buffers, 512 bytes (total 1000, permanent 1000):
     0 in free list (0 min, 1000 max allowed)
     1000 hits, 0 fallbacks
     1000 max cache size, 872 in cache
     878084773 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions
GigabitEthernet0/2 buffers, 512 bytes (total 1000, permanent 1000):
     0 in free list (0 min, 1000 max allowed)
     1000 hits, 0 fallbacks
     1000 max cache size, 872 in cache
     128 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions
GigabitEthernet0/3 buffers, 512 bytes (total 1000, permanent 1000):
     0 in free list (0 min, 1000 max allowed)
     1000 hits, 0 fallbacks
     1000 max cache size, 872 in cache
     128 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions
Serial2/0 buffers, 512 bytes (total 256, permanent 256):
     0 in free list (0 min, 256 max allowed)
     256 hits, 0 fallbacks
     256 max cache size, 132 in cache
     37815607 hits in cache, 0 misses in cache
     10 buffer threshold, 0 threshold transitions
Serial2/1 buffers, 512 bytes (total 256, permanent 256):
     0 in free list (0 min, 256 max allowed)
     256 hits, 0 fallbacks
     256 max cache size, 132 in cache
     124 hits in cache, 0 misses in cache
     10 buffer threshold, 0 threshold transitions
FastEthernet0/0 buffers, 768 bytes (total 559, permanent 400):
     129 in free list (128 min, 1200 max allowed)
     387 hits, 64 misses, 657 trims, 816 created
     0 failures (0 no memory)
     400 max cache size, 302 in cache
     44387326 hits in cache, 30 misses in cache
     14 buffer threshold, 11011 threshold transitions
FastEthernet0/1 buffers, 768 bytes (total 529, permanent 400):
     129 in free list (128 min, 1200 max allowed)
     357 hits, 43 misses, 657 trims, 786 created
     0 failures (0 no memory)
     400 max cache size, 272 in cache
     128 hits in cache, 0 misses in cache
     14 buffer threshold, 0 threshold transitions

Highlighted
Beginner

Re: High CPU Utlisation on 7206VXR due to traffic?

uksmh-policy2#sh proc cpu sorted
CPU utilization for five seconds: 73%/72%; one minute: 36%; five minutes: 22%

This shows that the high CPU is happening at interrupt level instead of process level, so it is likely traffic related or a feature that doesn't require punts to the processor.  To truly know what is using the CPU at interrupt level you'd need to do CPU profiling which only TAC/Cisco Internal could decode.

In this situation a case with TAC would be required.

If that isn't possible we may be able to see what's causing the spikes in other ways.

One that comes to mind is an EEM script that triggers when there is high CPU and shows what the interfaces look like when the spikes occur.

Before you configure the EEM script please change the load-interval on your interfaces to 30 seconds instead of 5 minutes. If you have a burst of traffic come into router we might have a problem seeing it in a 30s window (the smallest we can configure) but it's very hard in a 5min window.

The command is, config mode in an interface:  load-interval 30

You may need to change the location that the results of this script are stored at in the router. You can change the save location by editing the below script's "flash:" to "disk0:" or whatever location you'd like to target.  This script will automatically try to remove itself from the router after it has run, so if you want it to run more than once you would need to reapply the script.  Running the script again would append to the high_cpu.txt file, not overwrite it.

config t

event manager applet high_cpu

event snmp oid 1.3.6.1.4.1.9.2.1.56.0 get-type exact entry-op ge entry-val 70 exit-time 300 poll-interval 10

action 1.0 syslog msg "*** CPU Utilization is high, running EEM high_cpu"

action 2.0 cli command "en"

action 2.1 cli command "show clock | append flash:high_cpu.txt"

action 3.0 syslog msg "*** high_cpu *** --> CPU statistics"

action 3.1 cli command "show process cpu history | append flash:high_cpu.txt"

action 3.2 cli command "show process cpu sorted | append flash:high_cpu.txt"

action 4.0 syslog msg "*** high_cpu *** capturing --> Interface Statistics"

action 4.1 cli command "show interface | append flash:high_cpu.txt"

action 5.0 syslog msg "*** Removing EEM script high_cpu.txt from device."

action 5.1 cli command "config t"

action 5.2 cli command "no event manager applet high_cpu.txt"

action 6.0 syslog msg "*** EEM Script Finished.  Exiting."

end

If you still don't see some kind of burst of traffic on an interface then I'd suggest looking at your running config for features like long access lists, IP Inspect, NAT, VPNs, etc.  All of these features are hardware/CEF switched on contemporary IOSes and so will drive up the CPU at interrupt level as well.

Hope that helps some,

Brian