cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
796
Views
5
Helpful
8
Replies

High CPU utilization on 7206VXR NPE-G2 due to interrupts

Dear Experts, I've a Cisco 7206VXR with NPE-G2 controller with a very high CPU due to interrupts.


The router handles a full BGP table and has about 240 Mbit/s of aggregated VoIP traffic.

The bandwidth is low so I suspect that the issue is due to the nature of VoIP packet (very small packet).

I know that the NPE-G2 is able to support up to 2 million of packet per second.


The current scenario is the following:
- about 160'000 incoming packet per second on a g0/3 interface
- about 160'000 outgoing packet pet second on a g0/3 interface
- about 160'000 incoming packet per second on a g0/1 + g0/2 interface
- about 160'000 outgoing packet pet second on a g0/1 + g0/2 interface

So the total amount of traffic is 640'000 packet per second that is distant from the upper limit of 2 million.


I've already try different IOS:

c7200p-adventerprisek9_sna-mz.124-24.T1.bin
c7200p-adventerprisek9-mz.151-4.M.bin
c7200p-adventerprisek9-mz.152-4.M7.bin

but the behaviour is practically the same.

There is no drop at the moment:

show interface switching:

GigabitEthernet0/1 - 2 - 3
Throttle count 0
Drops RP 0 SP 0
SPD Flushes Fast 0 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 14919 Drops 0

and IP CEF is working properly.

Do you have suggestion for me?

Thanks in advance for your great support!

8 Replies 8

Philip D'Ath
VIP Alumni
VIP Alumni

You are not going to like this - but my guy feeling is you have probably hit its performance limit - because of the small packet size distribution.

You have no NAT, no policy routing, nothing that needs CPU intervention.

Time to get the cash flow card out.  Because you need a full BGP table you are probably going to have to look at an ASR1k.

http://www.cisco.com/c/en/us/products/routers/asr-1000-series-aggregation-services-routers/models-comparison.html

You could also consider a 6880, but based on your existing config, I think the ASR is a better fit so far.

http://www.cisco.com/c/en/us/products/switches/catalyst-6880-x-switch/index.html

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I think if you search these support forums, you'll likely find others complain that the NPE-G2 doesn't always seem to achieve it's 2 Mpps rating.

BTW, for PPS purposes, you only need to count ingress traffic, not both ingress and egress.

Like Philip, not much jumps out from your config that accounts for such high CPU usage at such low PPS rates.  I don't see "normal" ACLs, but you might still want to enable the Turbo ACL feature.

I do see voice service voip, of which I'm unfamiliar.

Your graph appears to show your router appears to be at its typical recommended working max, so unless you're experiencing some performance issues or you need to handle even more traffic, you're probably okay (for now).

If you do need to upgrade, as recommended by Phillip, the ASR1K would be the next logical step, but perhaps better and less expensive performance might also be found in the 4451.

Hi again.
I tried the IOS c7200p-adventerprisek9-mz.122-33.SRE2 also but the cpu load is practically the same.

These are some show output:

router#sh int
GigabitEthernet0/1 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is 0024.50b0.101b (bia 0024.50b0.101b)
Description:
Internet address is
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 3/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, media type is RJ45
output flow-control is XON, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:02, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 12744000 bits/sec, 14941 packets/sec
5 minute output rate 8111000 bits/sec, 8967 packets/sec
242053564 packets input, 2655783780 bytes, 0 no buffer
Received 2 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
175521826 packets output, 4128504300 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
10 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

GigabitEthernet0/2 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is 0024.50b0.101a (bia 0024.50b0.101a)
Description:
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 12/255, rxload 12/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, media type is RJ45
output flow-control is XON, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 48133000 bits/sec, 61574 packets/sec
5 minute output rate 50541000 bits/sec, 63535 packets/sec
702192632 packets input, 3198303575 bytes, 0 no buffer
Received 752 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 6548 multicast, 0 pause input
718325237 packets output, 1083873475 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
2 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/2.1 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is 0024.50b0.101a (bia 0024.50b0.101a)
Description:
Internet address is
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 12/255, rxload 12/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1804.
ARP type: ARPA, ARP Timeout 04:00:00
Keepalive set (10 sec)
Last clearing of "show interface" counters never
GigabitEthernet0/3 is up, line protocol is up
Hardware is MV64460 Internal MAC, address is 0024.50b0.1019 (bia 0024.50b0.1019)
Description:
Internet address is
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 15/255, rxload 14/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1000Mbps, media type is RJ45
output flow-control is XON, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 56696000 bits/sec, 72509 packets/sec
5 minute output rate 58836000 bits/sec, 76527 packets/sec
893835947 packets input, 2454296917 bytes, 0 no buffer
Received 29 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 565 multicast, 0 pause input
944217480 packets output, 2888394305 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
2 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

router#sh ip cef switching statistics

Path      Reason                         Drop     Punt      Punt2Host
RP LES Packet destined for us 0           24022   0
RP LES Total                             0           24022   0

RP PAS No route                       7173     0           0
RP PAS Packet destined for us 0           24022   0
RP PAS No adjacency               1086     0           0
RP PAS TTL expired                  0           0          10685
RP PAS Neighbor resolution req 12523  1          0
RP PAS Total                               20782   24023 10685

All Total                                        20782   48045 10685

Could be a problem of not CEF routed packet?

I see some punt packets.

Do you have suggestons for me?

Thanks.

I noticed also a strange 4ms Timer Interrupt in show stacks.

router#sh stacks

Interrupt level stacks:
Level Called Unused/Size Name
1 1297587812 6596/9000 Network interfaces
2 0 9000/9000 DMA level Interrupt
3 8 8772/9000 PA Management Int Handler
4 3309 8892/9000 Console Uart
5 0 9000/9000 OIR/Error Interrupt
7 31070716 9000/9000 4ms Timer Interrupt

Can you explain me what is this process?

Regards.

I don't know what this process does.

I'll guess some hardware timer fires for synchronising something, and this process is handling it.

A profiling CPU output:

router#show profile terse
PROF A3D4 3043DFF 4
PROFSKIP 2A9E54
0 0 0 0 0 0 1 0
PROFSKIP 2AD774
1 0 0 0 0 0 0 0
PROFSKIP 2AF674
0 1 0 0 0 0 0 0
PROFSKIP 2AFB14
0 0 0 0 0 0 0 1
PROFSKIP 2AFB54
0 1 0 0 0 0 0 0
PROFSKIP 2B0F34
0 0 1 0 0 0 0 0
PROFSKIP 2B0F74
0 0 0 1 0 0 0 0
PROFSKIP 2B1174
0 0 0 0 0 1 0 0
PROFSKIP 2B2554
0 0 0 1 0 0 0 0
PROFSKIP 2B3E34
0 0 0 0 0 0 1 0
PROFSKIP 2B44F4
0 0 0 0 0 1 0 0
PROFSKIP 2B5FF4
2 0 8 0 1 3 2 0
0 0 4 1 2 0 2 3
0 0 0 4 1 0 2 0
0 0 4 0 0 0 1 0
0 0 0 0 7 0 2 0
0 0 2 0 2 0 0 0
0 0 5 0 0 0 0 4
0 1 0 0 0 0 0 5
1 0 0 0 1 0 2 0
0 0 0 0 2 0 2 0
0 1 2 0 5 0 0 0
0 0 2 0 0 0 0 0
0 0 0 0 1 0 3 0
5 0 0 0 0 0 0 0
1 3 1 0 0 0 0 39
0 3 0 0 3 1 0 2
0 0 1 0 0 0 0 0
PROFSKIP 2B6374
0 0 0 1 87 5 0 1
1 0 0 0 0 0 0 0
PROFSKIP 2B6434
0 0 326 143 0 18 4 0
PROFSKIP 2B6714
4 1 1 0 9 32 0 0
PROFSKIP 2B6934
0 0 0 0 3 0 1 0
6 0 0 2 1 0 0 0
PROFSKIP 2B69D4
0 0 0 5 4 0 0 1
PROFSKIP 2B6AB4
0 0 5 1 2 0 0 0
PROFSKIP 2B6B34
0 0 0 0 2 0 0 0
PROFSKIP 2B6C74
0 0 0 0 1 1 2 0
0 0 0 2 0 1 0 0
1 0 0 1 0 0 0 0
PROFSKIP 2B6CF4
0 0 0 0 0 0 3 5
PROFSKIP 2B6E14
1 40 0 3 0 1 0 4
7 0 0 0 0 0 0 0
PROFSKIP 2B6F54
0 0 0 0 2 0 0 0
PROFSKIP 2B70F4
0 0 0 0 0 0 0 1
0 0 0 0 0 0 2 4
0 0 2 0 0 0 0 0
0 0 1 0 2 2 1 0
PROFSKIP 2B9514
0 0 0 0 15 0 0 0
0 0 0 0 1 0 0 0
PROFSKIP 2B9774
0 0 0 0 1 0 0 0
0 0 0 0 0 0 1 0
PROFSKIP 2B9814
0 0 0 0 0 1 0 0
PROFSKIP 2E5B74
0 0 0 1 0 0 0 0
PROFSKIP 2F6C14
0 0 6 0 0 0 0 0
0 8 0 0 0 0 0 0
0 0 0 0 0 3 1 4
2 0 0 0 0 0 0 0
PROFSKIP 2F8654
0 0 0 0 1 0 0 0
PROFSKIP 302134
0 0 0 0 3 1 3 0
1 0 0 0 2 10 5 0
3 3 0 4 0 0 0 0
PROFSKIP 3021B4
0 0 0 1 0 1 1 3
1 0 5 0 0 0 0 0
PROFSKIP 302754
0 0 0 0 0 57 5 0
0 0 0 0 0 1 0 1
PROFSKIP 31B554
0 1 0 0 0 0 0 0
PROFSKIP 3341D4
0 0 0 0 0 1 3 0
PROFSKIP 377EB4
0 0 1 0 0 0 0 0
PROFSKIP 379134
0 1 0 0 0 0 0 0
PROFSKIP 38E614
0 1 0 0 0 0 0 0
PROFSKIP 38E794
2 1 2 0 0 0 0 0
PROFSKIP 4A6774
0 0 0 0 0 1 0 0
PROFSKIP 4A7C74
0 1 0 0 0 0 0 0
PROFSKIP 4B7474
0 0 0 0 0 0 10 0
49 48 0 0 0 0 0 0
PROFSKIP 4CE2B4
1 0 0 0 0 0 0 0
PROFSKIP 4D1094
0 1 0 0 0 0 0 0
PROFSKIP 4D3FF4
0 0 0 0 1 0 0 0
PROFSKIP 4E1BB4
0 0 0 0 0 0 0 1
PROFSKIP 4E7FF4
0 0 0 0 0 0 1 0
PROFSKIP 52C714
0 24 0 48 19 118 0 0
24 0 146 0 0 6 0 262
PROFSKIP 585B94
0 0 0 0 0 1 0 0
PROFSKIP 59C514
0 1 0 0 0 0 0 0
PROFSKIP 5D3534
0 0 0 0 0 24 108 36
113 0 30 1 30 0 21 0
31 0 25 26 0 58 32 25
19 21 19 48 27 20 0 1
110 1 0 0 0 0 0 0
0 0 27 0 25 68 0 18
0 25 0 0 13 0 16 3
22 23 0 20 44 0 41 46
33 52 56 52 21 15 0 0
26 0 21 31 61 0 19 0
24 25 0 25 17 0 22 0
17 19 23 7 0 124 3 0
0 0 0 0 4 0 0 2
PROFSKIP 5D3734
0 18 0 0 16 0 0 22
24 27 34 5 85 3 61 5
64 2 70 3 70 3 3 0
PROFSKIP 5D37B4
0 0 0 102 2 0 3 33
61 0 0 0 0 0 0 0
0 16 0 0 25 0 0 28
21 20 23 15 26 54 76 0
20 0 0 0 0 0 0 0
PROFSKIP 5D38D4
0 0 0 0 0 0 0 23
15 25 8 81 0 4 15 1
0 25 12 32 27 35 26 26
20 0 0 8 74 0 0 0
PROFSKIP 7F3D74
0 0 0 0 0 0 1 0
1 0 0 0 0 5 0 0
0 0 0 0 1 0 0 0
0 1 0 0 1 1 0 0
0 1 0 1 0 0 0 0
0 0 0 1 1 0 3 0
0 0 2 0 0 0 0 0
PROFSKIP 7F3ED4
0 4 1 1 1 0 1 0
0 0 0 1 0 1 0 2
0 2 0 6 0 0 0 0
PROFSKIP 7F3FD4
4 0 0 0 1 0 0 0
PROFSKIP 7F4034
1 0 0 0 0 0 3 6
0 0 0 3 0 0 0 0
0 0 0 0 3 0 0 0
PROFSKIP 833694
0 0 0 0 0 4 0 0
PROFSKIP 9E7194
0 0 1 0 0 0 0 0
PROFSKIP 9F5B54
0 1 0 0 0 0 0 0
PROFSKIP A24254
30 269 10 0 0 0 0 0
PROFSKIP D83094
0 1 0 0 0 0 0 0
PROFSKIP DD2FD4
0 0 0 0 0 0 1 0
PROFSKIP 1151D74
0 0 0 0 1 0 0 0
PROFSKIP 1155894
0 0 0 0 59 207 50 0
59 5 0 52 0 3 4 0
10 68 0 53 74 53 0 42
3 45 4 0 59 0 28 47
21 0 53 7 12 0 0 0
PROFSKIP 1155CB4
0 0 0 0 0 0 0 58
241 0 56 0 59 0 141 0
0 66 134 9 0 58 50 154
52 6 4 41 51 5 0 0
PROFSKIP 1158694
0 0 0 0 0 0 90 379
108 0 90 0 293 0 0 77
55 61 0 120 8 0 0 0
PROFSKIP 11591B4
0 0 0 0 0 129 413 130
0 104 0 113 0 303 0 0
31 108 0 0 0 0 0 0
0 0 0 6 19 100 94 294
PROFSKIP 11592F4
0 90 390 108 0 113 0 123
0 95 0 110 99 188 108 0
257 0 0 18 92 98 9 99
PROFSKIP 11593D4
0 0 0 0 0 0 15 6
84 81 293 18 2 97 11 0
0 0 0 0 9 92 0 13
191 7 0 0 0 0 0 0
PROFSKIP 1159494
0 0 0 0 0 298 15 0
99 110 0 1 100 7 4 104
295 0 0 0 0 0 0 0
PROFSKIP 1159754
0 64 200 55 0 44 0 160
0 0 47 11 0 0 0 0
55 26 6 44 151 8 3 37
56 7 0 0 0 0 0 0
PROFSKIP 115C394
0 0 0 0 0 0 0 106
386 121 0 98 0 100 0 104
0 85 101 11 5 0 0 0
PROFSKIP 115C414
0 0 0 547 664 140 201 311
124 111 0 0 109 379 113 0
91 0 107 0 98 0 115 0
277 0 0 0 0 0 0 0
0 0 0 0 0 0 20 0
93 5 0 104 106 0 0 0
PROFSKIP 115C514
0 0 0 96 135 99 298 96
311 21 0 0 107 0 0 19
0 94 124 0 115 98 0 0
0 0 14 0 132 0 110 13
PROFSKIP 115C5B4
0 0 3 86 0 0 90 204
PROFSKIP 115C874
66 533 2185 0 0 843 0 71
338 386 75 366 0 0 0 0
0 0 0 0 0 0 0 536
0 0 1102 75 248 0 302 212
0 82 35 0 0 0 0 0
PROFSKIP 115CE54
0 0 0 0 0 0 0 114
391 124 0 105 107 0 305 0
0 14 17 79 36 98 101 279
0 85 275 20 0 100 6 0
84 43 0 91 12 0 78 116
0 18 75 13 0 89 62 0
PROFSKIP 1164BD4
0 0 47 264 77 42 0 11
54 0 0 52 14 0 0 0
0 0 0 0 47 22 9 34
57 3 0 0 0 0 0 0
PROFSKIP 117A3D4
0 0 0 1 0 0 0 0
PROFSKIP 1181254
0 1 0 0 0 0 0 0
PROFSKIP 1183C54
0 0 0 0 0 0 0 1
PROFSKIP 1183D14
0 0 0 0 0 1 0 0
PROFSKIP 17A5A74
0 1 0 0 0 0 0 0
PROFSKIP 17A6114
0 0 0 0 0 1 0 0
PROFSKIP 1A1C174
0 0 1 0 0 0 0 0
PROFSKIP 1A4C534
0 0 0 0 0 0 0 1
PROFSKIP 1C30B94
0 0 0 0 0 1 0 0
PROFSKIP 1C5E1D4
0 0 100 0 89 294 199 0
110 0 13 97 181 117 98 216
115 197 219 0 0 0 0 120
132 344 208 188 121 314 0 108
0 0 0 9 2 0 0 0
PROFSKIP 1C5EE54
95 26 21 92 209 70 58 196
66 76 184 155 0 0 0 0
PROFSKIP 1C5EF74
0 0 1 0 0 0 0 0
0 0 0 0 0 1 1 4
0 0 0 0 0 0 0 1
1 0 0 0 0 2 0 2
1 1 3 6 0 0 0 0
3 2 3 3 3 0 2 0
0 0 1 0 0 0 0 1
PROFSKIP 1C619D4
1 0 0 0 0 0 0 0
PROFSKIP 1C61F54
0 0 0 0 1 0 0 0
PROFSKIP 1C62534
0 0 0 0 0 0 131 381
0 110 14 12 1 0 0 0
263 0 686 40 0 0 0 0
PROFSKIP 1C67634
0 1 0 0 1 0 0 1
PROFSKIP 1C67FF4
0 23 0 11 0 4 6 0
2 12 4 0 0 0 0 0
PROFSKIP 1C68054
0 0 0 0 21 0 0 0
PROFSKIP 1C68094
0 0 0 0 6 0 0 0
PROFSKIP 1C693D4
0 0 0 0 0 2 9 2
0 1 0 0 0 2 6 0
0 1 0 1 0 0 1 0
PROFSKIP 1C69474
0 0 0 0 0 4 0 0
PROFSKIP 1C69534
0 0 0 3 0 0 0 0
PROFSKIP 1C7D334
2 0 0 0 0 0 0 0
0 0 4 0 1 0 0 0
0 0 18 0 0 38 0 452
PROFSKIP 1C7D3B4
0 0 0 0 0 0 0 23
106 0 0 0 0 0 0 0
PROFSKIP 1C7D414
20 109 0 0 0 0 0 0
PROFSKIP 1C7D454
5 2 0 0 0 0 2 0
PROFSKIP 1C7D4F4
4 0 0 1 0 0 0 1
0 0 0 29 0 0 27 0
3 0 0 535 0 0 0 0
0 0 0 0 0 0 0 27
9 0 1 4 0 0 0 0
PROFSKIP 1C7D5F4
0 0 3 0 0 9 0 0
PROFSKIP 1C7D634
0 0 0 0 0 8 0 0
PROFSKIP 1C7D6B4
0 0 0 0 0 0 0 2
0 0 0 0 2 0 0 0
PROFSKIP 1C7D894
0 0 0 0 0 0 0 1
PROFSKIP 1C816B4
0 0 0 0 0 0 4 1
PROFSKIP 1C816F4
0 0 0 0 1 0 0 0
PROFSKIP 1C817B4
0 0 0 1 0 0 0 0
PROFSKIP 1C81854
0 0 0 0 0 0 1 0
PROFSKIP 1C82834
0 0 0 0 0 1 0 0
PROFSKIP 1C828B4
1 0 0 0 0 0 0 0
PROFSKIP 1C82934
0 0 0 0 1 0 0 0
PROFSKIP 1C82A34
0 0 0 1 0 0 0 0
0 0 1 0 0 0 0 0
PROFSKIP 1C82D34
0 0 1 1 0 0 0 0
PROFSKIP 1C83C54
1 0 0 0 0 0 0 0
PROFSKIP 1CB6E14
0 0 0 0 1 0 1 0
PROFSKIP 1CB6E74
0 0 0 0 0 0 0 2
11 16 0 3 3 0 1 1
0 0 0 0 0 0 0 7
4 0 0 10 0 0 2 0
0 0 0 0 0 0 0 2
PROFSKIP 1CB6F54
0 0 0 1 0 0 0 0
0 1 0 0 0 0 0 0
PROFSKIP 1CB9E34
0 0 0 0 0 1 0 0
PROFSKIP 1CBCF74
0 0 0 1 0 0 0 0
PROFSKIP 1CBDE74
0 0 0 0 1 0 0 0
PROFSKIP 1CBEAB4
0 0 0 0 1 3 11 0
0 0 2 0 2 0 0 0
0 0 82 4 0 0 0 0
PROFSKIP 1CBEB34
0 0 0 0 0 5 51 0
PROFSKIP 1CBEB74
3 5 0 2 0 0 4 0
PROFSKIP 1CBEBB4
0 0 0 0 0 0 0 2
2 2 3 3 9 0 0 0
PROFSKIP 1CBEC14
0 3 0 3 0 5 0 0
3 0 0 4 0 88 2 0
8 0 0 0 0 0 0 0
1 16 3 0 3 0 0 0
PROFSKIP 1CBECB4
0 0 0 0 3 4 1 6
0 2 2 0 3 0 1 3
2 7 0 1 2 0 2 0
0 0 11 2 0 0 0 0
0 0 0 0 0 0 1 5
0 2 0 0 2 2 0 0
PROFSKIP 1CBEDD4
0 0 0 2 6 0 1 0
0 5 0 0 0 0 0 0
PROFSKIP 1CBEF14
0 0 0 0 1 0 0 0
PROFSKIP 1CBF0B4
0 0 0 0 0 0 1 0
PROFSKIP 1CBF254
0 0 0 0 1 2 0 0
0 5 2 3 0 0 2 0
PROFSKIP 1CC96F4
0 0 0 0 1 0 0 0
2 0 0 0 0 1 0 0
PROFSKIP 1CC9754
0 0 0 2 0 0 0 0
PROFSKIP 1CC97F4
0 0 0 0 0 2 0 0
0 24 3 0 2 0 0 0
PROFSKIP 1CC99D4
3 0 0 2 0 2 1 0
1 0 0 0 0 0 2 0
0 0 1 0 0 0 1 0
PROFSKIP 1CC9AD4
0 0 0 0 0 1 0 0
PROFSKIP 1CC9B14
0 0 0 0 0 0 3 0
0 6 0 0 0 0 2 0
0 2 0 0 2 0 0 0
0 0 4 1 0 0 0 1
0 1 0 0 0 0 0 4
PROFSKIP 1CC9C34
0 0 3 0 0 11 0 2
0 0 1 0 0 1 0 0
1 0 0 2 0 0 9 0
1 0 0 0 0 0 0 0
PROFSKIP 1CC9CD4
0 0 0 1 0 0 0 0
PROFSKIP 1CC9D94
0 0 0 0 2 0 0 0
PROFSKIP 1CCA0D4
0 0 0 0 2 0 0 0
PROFSKIP 1CD13B4
0 0 0 0 0 0 105 102
286 382 0 117 0 0 112 106
112 92 0 104 0 0 118 11
PROFSKIP 1CD1514
0 0 91 0 95 0 153 409
117 285 0 111 117 0 0 0
PROFSKIP 1CD5714
0 0 0 0 0 1 0 0
PROFSKIP 1CD5D94
0 0 0 0 0 1 0 0
PROFSKIP 1CD9EB4
0 0 0 0 0 0 1 0
PROFSKIP 1CDC914
0 166 397 4 0 0 0 0
PROFSKIP 1CFCF34
0 0 0 1 0 0 0 0
PROFSKIP 1D041D4
0 0 0 0 0 0 7 32
0 11 0 8 0 8 0 8
0 7 1 0 14 0 0 5
0 0 5 4 0 4 16 0
5 0 5 22 0 7 15 4
6 0 0 0 1 0 2 10
13 0 11 10 1 4 0 0
10 0 0 7 15 8 0 4
5 6 14 0 0 0 0 0
PROFSKIP 1D0F4D4
55 1 0 11 0 3 27 0
4 0 4 0 8 0 2 0
6 4 4 2 37 22 0 0
45 5 0 0 0 0 11 8
0 0 0 0 7 0 0 38
0 0 3 2 0 0 4 21
26 0 2 0 0 0 1 15
1 10 24 1 0 21 3 21
14 0 0 9 0 0 14 4
11 4 7 0 0 0 0 0
PROFSKIP 1D0F754
0 0 0 0 0 0 42 0
5 0 0 7 4 0 1 0
36 0 20 0 2 11 0 0
PROFSKIP 1D0F854
0 0 0 0 0 7 0 22
0 0 5 5 3 8 0 1
2 6 0 12 0 0 5 0
7 0 0 0 0 0 0 0
PROFSKIP 1D0F914
0 10 15 9 0 26 7 0
2 11 0 0 0 0 0 0
0 0 0 0 35 0 4 8
8 3 11 0 0 0 0 0
PROFSKIP 217F414
66 0 149 150 0 0 0 0
PROFSKIP 220A6F4
1 159 0 4 0 0 0 0
0 0 0 0 0 46 0 0
44 0 0 0 0 0 0 0
PROFSKIP 2216B14
0 0 0 0 0 19 221 14
PROFSKIP 224C414
0 0 0 0 0 6 27 0
0 5 0 0 9 0 0 0
0 0 2 0 0 5 0 0
PROFSKIP 224C494
0 0 0 0 5 6 0 0
14 0 6 2 35 0 0 5
0 0 0 8 0 0 0 1
0 0 6 3 0 0 0 0
PROFSKIP 224C534
2 2 0 6 25 0 0 0
PROFSKIP 224E8F4
0 1 25 15 0 13 0 0
PROFSKIP 224E934
2 23 0 22 0 0 0 0
PROFSKIP 2264EF4
0 0 24 168 13 16 0 13
0 0 17 24 0 18 4 0
24 0 0 64 0 0 20 0
PROFSKIP 2264FF4
0 21 0 63 43 24 42 2
PROFSKIP 226BF94
0 0 0 0 1 0 0 0
PROFSKIP 25CCB74
0 0 0 13 98 0 0 0
PROFSKIP 25CDC14
0 0 1 10 3 2 1 0
PROFSKIP 25CDCB4
0 0 0 0 0 0 0 275
0 22 0 5 3 0 0 0
PROFSKIP 25CDD34
0 0 0 0 0 0 0 3
0 0 0 0 0 0 1 0
0 0 0 26 0 0 0 0
PROFSKIP 25CE3F4
0 0 255 0 0 0 0 0
0 0 0 224 0 0 0 0
PROFSKIP 25CE454
2 6 107 0 29 4 22 160
1 74 41 0 97 26 161 0
31 122 0 0 0 192 29 115
8 0 24 44 0 35 0 0
21 78 35 0 0 25 0 0
2 137 8 0 74 4 0 29
19 41 0 8 0 58 55 20
19 29 72 2 2 24 0 4
116 23 0 0 11 6 0 5
PROFSKIP 25E9914
72 73 20 2 32 0 31 25
49 0 57 0 30 47 0 34
0 20 37 95 0 0 0 0
PROFSKIP 25EDB34
2 109 0 45 0 3 0 52
0 6 0 39 36 71 7 69
5 41 0 0 121 0 0 0
0 0 0 0 10 0 0 40
1554 0 3 0 0 9 0 0
25 23 0 11 146 0 9 5
0 0 0 0 8 0 156 26
8 27 0 16 83 23 0 0
0 0 10 1 0 40 0 0
15 13 0 12 17 20 74 12
0 18 0 34 22 0 4 0
29 21 16 163 0 0 0 0
PROFSKIP 25EE214
0 0 0 0 2 2 0 0
PROFSKIP 25EE3D4
0 0 0 0 0 0 1 0
0 3 0 0 8 0 10 10
0 0 6 0 0 0 5 0
0 5 0 2 22 0 0 0
PROFSKIP 25EE6D4
0 0 0 0 0 17 28 26
37 27 0 0 152 17 79 0
0 21 1 30 16 0 0 0
PROFSKIP 25EF354
0 0 0 0 0 0 12 0
0 29 51 80 0 0 11 0
25 0 58 142 0 36 7 30
0 14 0 0 24 0 0 17
28 16 22 11 34 18 23 46
11 28 21 29 0 22 0 40
103 0 5 22 0 0 23 20
24 15 27 21 26 31 35 40
24 40 21 36 42 0 0 0
PROFSKIP 25EF854
0 0 0 0 0 5 7 0
1 0 0 0 0 0 0 0
PROFSKIP 25EFA54
0 0 0 0 0 23 21 0
0 0 29 0 15 0 0 78
56 0 0 0 0 0 0 0
PROFSKIP 25F0094
7500 0 0 0 0 0 0 0
0 17 3 92 10 0 4 0
15 0 0 0 0 0 2 0
1 95 0 0 0 14 5 0
13 0 0 0 0 1 96 0
0 13 18 0 0 93 2 1
10 4825 0 0 0 0 0 0
0 0 146 0 1 12 15 12
0 140 37 0 0 0 5 1048
0 12 0 5 36 0 29 0
4 0 0 0 12 0 119 383
0 10 9 0 23 0 118 0
PROFSKIP 25F02D4
0 0 0 0 0 0 0 14
0 9 2 0 371 38 0 0
0 0 0 0 0 3 0 10
22 0 0 0 22 31 18 9
0 0 8 0 0 0 0 0
PROFSKIP 25F0474
0 0 0 0 56 66 0 0
41 70 0 3 53 11 34 19
5 0 0 14 15 34 0 0
8 0 11 11 13 9 9 9
13 13 11 13 10 0 0 0
PROFSKIP 25F0574
0 0 0 0 0 0 0 10
0 10 20 10 0 12 105 17
14 26 22 0 0 0 0 0
PROFSKIP 25F0634
0 0 0 0 0 137 0 14
0 0 113 106 40 0 0 160
10 22 0 111 17 24 10 0
16 0 5 0 0 19 576 0
131 0 45 0 109 18 0 0
PROFSKIP 25F0714
16 0 21 0 0 1 3 46
1 2 10 14 0 0 0 69
1 4 90 26 1 6 0 0
PROFSKIP 25F0894
0 0 0 0 0 0 84 0
1 0 1 0 4 31 27 101
0 19 0 0 30 0 0 20
0 19 17 20 19 26 25 48
32 19 20 16 26 0 44 0
18 31 0 1 37 0 22 29
20 26 20 23 34 16 21 25
30 12 0 48 27 0 0 21
19 17 28 0 0 0 0 0
PROFSKIP 25F0AB4
0 0 0 0 1 0 0 0
PROFSKIP 25F0B14
0 0 0 0 0 2 0 0
PROFSKIP 25F0D14
0 0 0 0 0 0 0 39
69 79 206 0 0 0 0 0
PROFSKIP 25F0E94
0 0 0 13 91 18 0 18
0 21 17 24 18 43 25 20
0 0 0 106 23 31 93 0
21 0 0 1 9 0 0 0
PROFSKIP 25F0F94
68 0 0 0 0 0 0 0
PROFSKIP 25F1074
0 0 0 40 39 77 27 23
72 0 15 40 20 34 0 0
PROFSKIP 25F10D4
0 14 0 0 0 0 0 0
PROFSKIP 25F1734
0 0 0 0 0 0 0 41
75 0 16 65 0 37 24 11
30 28 4 31 16 21 21 25
10 58 15 13 15 16 22 145
0 0 14 0 0 0 18 0
0 0 17 30 15 0 106 0
12 0 0 0 0 0 0 29
8 0 26 20 0 34 0 67
58 13 0 0 0 0 0 0
PROFSKIP 25F1894
0 31 24 21 20 57 0 0
17 0 0 22 22 13 18 20
11 24 15 11 12 11 20 0
PROFSKIP 25F21B4
0 24 50 1 0 59 0 35
21 9 31 39 11 5 3 48
32 78 0 0 107 1 0 16
0 0 9 30 1062 0 0 0
0 52 0 0 0 0 0 0
0 0 0 0 0 0 0 33
0 4 0 0 4 0 4 0
119 5 25 10 0 4 0 0
1 0 2 0 2 0 0 71
8 8 0 8 20 0 24 11
3 0 1 0 2 55 21 5
9 2 4 28 0 0 2 2
78 0 0 4 11 0 0 0
PROFSKIP 25F2394
0 1 15 39 0 0 15 45
21 0 16 0 3 20 59 0
0 11 1 28 10 9 18 15
25 0 0 0 0 0 0 0
PROFSKIP 25F2474
0 0 0 17 3 0 0 0
0 26 2 0 0 0 57 100
932 2 0 0 42 1 8 47
24 6 0 0 2 0 4 0
0 63 0 0 0 0 0 0
0 0 0 0 0 0 0 3
5 0 10 0 22 6 6 0
1 13 20 3 1 2 0 10
2 0 1 3 0 14 0 5
0 8 0 62 3 7 37 4
5 3 5 0 0 1 9 58
0 0 9 13 0 0 0 0
PROFSKIP 25F2A94
0 20 113 28 64 0 17 0
30 29 21 22 27 30 23 30
20 22 16 51 30 26 20 25
15 108 0 19 0 3 0 89
9 0 0 0 0 0 0 13
16 44 26 54 0 0 18 0
0 26 16 13 20 16 24 16
23 18 24 28 19 0 0 1
28 48 1 0 26 19 0 10
56 0 24 1 0 0 0 0
PROFSKIP 25F2C34
0 0 0 0 0 0 20 20
0 27 8 13 0 0 0 0
PROFSKIP 25F2DB4
0 0 0 0 0 0 0 20
33 0 0 0 0 0 0 0
PROFSKIP 25F2E54
18 22 0 22 0 0 0 0
PROFSKIP 25F2E94
0 0 0 0 22 94 26 0
0 0 0 0 20 0 0 0
41 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0
0 0 0 0 0 0 17 0
56 45 26 1 20 15 0 0
27 19 0 0 0 0 0 9
0 19 32 22 62 0 54 66
0 26 0 0 22 17 0 45
0 24 23 0 52 0 0 40
PROFSKIP 25F3014
0 0 0 22 0 73 75 0
PROFSKIP 25F3094
0 0 0 0 0 0 2 0
0 0 1 0 0 0 0 0
PROFSKIP 25F3174
0 0 0 0 0 0 2 0
PROFSKIP 25F3234
23 20 0 19 2 54 29 0
0 12 10 0 17 16 0 27
17 22 37 59 42 112 0 44
PROFSKIP 25F3354
0 0 0 0 0 14 71 45
21 0 0 0 0 0 0 0
PROFSKIP 25F3634
0 0 0 0 23 24 0 27
0 15 0 0 0 0 0 0
PROFSKIP 25F3914
19 28 91 1 36 0 26 47
63 0 8 27 63 8635 0 0
0 0 0 0 0 0 0 31
0 6 0 18 0 163 1 0
12 0 0 0 0 0 0 0
PROFSKIP 25F3A94
0 29 62 21 25 81 0 0
0 0 0 3495 0 0 0 0
0 0 0 0 0 0 93 13
7 0 0 0 0 0 0 0
PROFSKIP 2642414
0 0 0 0 0 0 1 0
PROFSKIP 26431F4
0 0 0 0 0 0 1 0
3 0 3 1 3 0 4 2
PROFSKIP 2643334
0 0 0 2 1 5 0 1
0 0 0 0 1 1 0 0
0 0 0 0 0 1 1 0
1 4 0 0 3 0 8 3
1 0 0 1 0 0 14 2
0 0 2 0 0 1 0 2
1 1 2 2 0 4 0 0
PROFSKIP 2643CB4
0 0 0 104 387 103 284 0
118 0 0 130 107 103 115 121
230 101 101 98 120 19 8 0
PROFSKIP 2643E14
0 0 599 210 542 116 293 0
110 123 218 279 187 216 115 112
PROFSKIP 26440B4
0 0 0 0 1 0 0 0
PROFSKIP 2644A34
0 0 1 0 0 0 0 0
PROFSKIP 2647194
0 1 0 0 0 3 0 0
1 0 1 0 0 0 0 3
0 2 3 0 0 0 0 0
PROFSKIP 2647214
0 1 0 0 0 0 0 0
PROFSKIP 26495B4
0 0 0 0 0 0 0 2
2 0 0 0 0 0 0 0
PROFSKIP 26496F4
0 0 0 44 179 61 0 54
6 41 0 0 0 0 0 0
PROFSKIP 26497B4
0 0 36 225 0 103 7 0
PROFSKIP 264AEB4
0 0 0 162 432 117 0 110
0 0 121 0 108 0 108 0
106 184 0 10 0 0 0 0
PROFSKIP 264B074
0 0 0 0 0 289 0 170
120 107 105 0 81 0 324 321
121 111 312 13 8 104 0 0
PROFSKIP 264B9B4
0 0 0 0 0 0 0 53
0 52 171 0 0 0 0 0
PROFSKIP 264BAD4
54 0 9 51 0 0 0 0
PROFSKIP 264BCF4
0 0 0 0 0 3 0 51
3 2 0 0 0 0 0 0
PROFSKIP 264C174
0 0 0 0 0 0 55 6
0 56 0 56 0 48 0 59
94 53 0 0 6 45 0 2
113 0 0 63 1 77 0 0
PROFSKIP 264C214
0 0 77 0 64 0 68 167
0 59 123 0 135 47 0 64
0 0 0 0 0 0 338 63
133 0 66 7 0 10 0 0
0 0 0 0 0 43 2 56
50 0 0 0 0 0 0 0
PROFSKIP 264C3B4
0 0 0 0 0 0 52 4
PROFSKIP 264C5D4
0 0 0 0 0 0 42 0
6 0 54 0 51 78 163 0
57 0 8 54 4 4 43 8
PROFSKIP 264C654
0 0 0 0 0 43 73 0
0 0 0 46 0 0 0 0
70 0 123 0 199 66 64 143
100 0 0 14 0 3 2 3
PROFSKIP 264CA94
0 0 0 0 0 0 4 1
0 7 0 0 0 0 0 0
PROFSKIP 264CF34
0 0 0 0 0 0 45 5
38 8 0 0 52 0 16 50
160 48 54 158 69 0 0 3
0 0 60 0 147 14 51 57
PROFSKIP 26562F4
0 0 0 0 0 0 0 1
PROFSKIP 267AE14
23 37 64 27 86 0 0 28
0 7 0 18 0 7 0 15
33 50 40 21 27 14 24 0
76 25 154 0 0 0 68 0
0 0 0 1 23 0 29 23
17 0 0 17 23 0 0 0
PROFSKIP 267AEF4
24 0 0 7 0 116 72 21
0 17 39 0 20 66 0 0
PROFSKIP 267AF74
0 0 0 0 0 0 22 0
44 0 0 0 24 0 0 0
PROFSKIP 267AFD4
0 0 0 0 0 27 13 65
0 0 20 0 1 28 20 33
21 32 0 0 0 0 0 0
PROFSKIP 267B094
26 19 0 0 29 0 15 10
35 18 20 30 0 0 0 0
PROFSKIP 267B0F4
0 0 0 23 18 66 0 0
35 65 0 0 29 20 0 0
PROFSKIP 267B154
0 0 12 0 0 0 0 0
PROFSKIP 267B234
0 0 0 0 0 25 26 18
0 16 0 0 22 23 0 14
PROFSKIP 267B2B4
0 28 0 0 14 0 27 19
0 0 124 0 28 0 71 111
104 0 0 0 80 72 13 0
21 0 16 0 2 41 0 18
0 24 0 87 0 26 20 0
0 25 0 21 0 0 0 0
PROFSKIP 267B414
0 0 34 0 22 24 71 12
72 113 0 0 0 0 0 0
PROFSKIP 274B274
0 0 5 1 102 10 11 0
6 0 5 3 8 0 2 5
3 0 6 13 0 6 17 0
PROFSKIP 280B5D4
0 0 0 0 0 0 9 0
PROFSKIP 280B9D4
0 0 23242 231 0 15 0 0
PROFSKIP 29B9A14
0 0 0 0 0 0 0 1
PROFSKIP 2DE4334
0 0 0 0 0 0 1 0
PROFSKIP 2EA8754
0 0 0 0 0 50 202 61
0 49 5 3 0 23 34 43
10 0 0 0 0 0 0 0
0 0 0 0 0 0 42 131
0 58 3 0 0 0 0 0
PROFSKIP 2EA8814
0 0 0 0 0 2 20 0
PROFSKIP 2EA8854
0 0 0 0 0 0 0 42
10 63 163 135 0 0 0 0
PROFSKIP 2EABA34
62 199 0 54 0 51 9 59
0 53 4 0 59 0 80 0
49 7 48 59 105 142 146 0
0 16 56 181 0 0 59 0
41 0 0 0 0 0 0 0
PROFSKIP 2EB2574
0 0 0 0 1 0 0 0
PROFSKIP 2EB2F54
0 0 1 0 0 0 0 0
PROFSKIP 2F283D4
0 0 0 0 0 0 1 0
PROFSKIP 2F29DB4
0 0 0 0 0 1 3 3
0 0 0 4 0 0 0 2
8 0 0 2 0 2 1 55
243 0 11 0 5 0 3 17
0 4 0 1 5 3 0 0
1 0 1 0 0 0 4 0
0 0 0 1 0 0 0 0
0 1 0 3 2 1 0 0
4 0 0 0 0 0 0 0
0 1 0 2 1 0 2 0
0 0 0 0 0 0 2 0
1 2 0 0 0 0 0 0
PROFSKIP 2F29F94
0 0 0 0 0 0 0 1
PROFSKIP 2F2A054
1 0 0 0 0 0 0 0
0 0 0 0 0 0 1 0
1 0 1 0 0 0 0 0
PROFSKIP 2F2A114
0 0 0 0 0 1 0 7
PROFSKIP 2F2A154
0 0 0 3 1 0 0 0
0 0 2 0 0 0 0 0
PROFSKIP 2F2A3D4
0 0 0 0 0 0 1 0
0 1 0 0 0 0 0 0
PROFSKIP 2F4B2B4
0 0 0 0 1 0 0 0
PROFSKIP 2F564F4
0 0 0 0 10 0 4 0
PROFSKIP 2F56F74
0 1 0 0 0 0 0 0
0 3 0 0 1 0 0 0
PROFSKIP 2F5A914
0 0 0 0 0 0 0 1
PROFSKIP 2F5AF94
0 0 0 0 0 0 0 1
0 0 0 1 2 0 0 0
0 2 0 0 0 0 2 0
0 0 0 3 0 0 0 0
2 0 0 1 0 1 0 0
0 0 2 0 0 0 0 0
PROFSKIP 2FAD134
0 0 0 2 0 1 0 0
PROFSKIP 2FD7854
0 0 0 0 1 52 25 0
PROFSKIP 2FEEC34
0 0 0 0 0 0 23 7
22 0 0 0 0 0 0 0

System Total = 000148325
Interrupt Total = 000087588 (59 percent)
Sched Total = 000056761 (38 percent)

Interrupt [00] = 000087588 (59 percent)

If you want a CPU profile analysed you need a Cisco maintenance contract.  You need to open a case with Cisco TAC, and ask them to put it into their profiling tool to get the results.

I think your router is working normally.  I think you have just reached its limits.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco