CISCO 6509 high CPU

[andresdavid]

Hello everybody, I have this problem for 2 days and I don't know what to do, 

bb01.network.ro>sh processes cpu | exclude 0.00%__0.00%__0.00%

CPU utilization for five seconds: 76%/67%; one minute: 84%; five minutes: 91%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
2 56640 661700 85 0.00% 0.01% 0.00% 0 Load Meter
8 34163576 1740542 19628 0.00% 0.88% 0.67% 0 Check heaps
11 4609760 7352294 626 0.15% 0.12% 0.13% 0 ARP Input
20 469000 3278408 143 0.00% 0.01% 0.00% 0 IPC Periodic Tim
23 26914948 162975847 165 0.00% 0.45% 0.51% 0 IPC Seat Manager
50 1405336 3451726 407 0.07% 0.07% 0.07% 0 Per-Second Jobs
51 2748004 116538 23580 0.87% 0.10% 0.06% 0 Per-minute Jobs
63 3541328 146553080 24 0.07% 0.07% 0.07% 0 Net Input
65 4048 463 8742 0.31% 0.19% 0.26% 2 SSH Process
84 2107256 5299555 397 0.07% 0.05% 0.05% 0 DHCP Snooping
218 659728 743227 887 0.00% 0.03% 0.02% 0 Compute load avg
241 526228 93192878 5 0.00% 0.03% 0.02% 0 ACE Tunnel Task
242 297764 6528667 45 0.07% 0.01% 0.00% 0 ACE Config Prop
253 1524904 3120700 488 0.00% 0.02% 0.01% 0 esw_vlan_stat_pr
266 1424060 2545041 559 0.07% 0.08% 0.08% 0 CDP Protocol
272 425015252 275710734 1541 2.63% 5.69% 8.45% 0 IP Input
298 98168 20468998 4 0.07% 0.00% 0.00% 0 Ethernet Timer C
299 1543800 374615844 4 0.23% 0.17% 0.16% 0 Ethernet Msec Ti
315 3613924 2274463 1588 0.15% 0.10% 0.11% 0 QOS Stats Gather
322 690244 120644 5721 0.00% 0.01% 0.00% 0 CEF background p
327 443864 77336 5739 0.07% 0.00% 0.00% 0 IP Background
333 526848 11680137 45 0.00% 0.03% 0.04% 0 TCP Timer
363 2090900 4989795 419 0.00% 0.06% 0.07% 0 CEF: IPv4 proces
376 95343748 35597873 2678 0.71% 0.78% 1.05% 0 FM core
384 1511928 1655151 913 0.00% 0.04% 0.05% 0 HIDDEN VLAN Proc
403 517800 93032054 5 0.00% 0.03% 0.02% 0 RADIUS
495 27500908 125788459 218 1.19% 0.70% 0.72% 0 Port manager per
555 153215372 12309297 12447 1.35% 2.20% 1.76% 0 IP NAT Ager
557 525376 32595667 16 0.00% 0.02% 0.01% 0 IGMP Input
558 302768 3315067 91 0.00% 0.01% 0.00% 0 PIM Process
559 344156 32519050 10 0.00% 0.01% 0.00% 0 Mwheel Process
562 4821188 317657 15177 0.00% 0.06% 0.09% 0 BGP Scanner
570 283060 13253757 21 0.00% 0.01% 0.00% 0 MLD

[Joseph W. Doherty]

Using the sorted option, for your show CPU, might be helpful.

That aside, it looks like you have a surprisingly high "interrupt" CPU consumption, which is "optimal" data traffic forwarding, in software, which is NOT "optimal" on switches.

Start here: https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/63992-6k-high-cpu.html

[Leo Laohoo]

Post the complete output to the following commands: 

1. sh version

2. sh proc cpu sort | ex 0.00

[andresdavid]

>sh version
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Sat 23-Oct-10 01:30 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)

bb01.network.ro uptime is 5 weeks, 3 days, 13 hours, 44 minutes
Uptime for this control processor is 5 weeks, 3 days, 13 hours, 45 minutes
Time since bb01.network.ro switched to active is 5 weeks, 3 days, 13 hours, 44 minutes
System returned to ROM by power cycle at 18:11:38 GMT Wed Mar 8 2023 (SP by power on)
System restarted at 18:36:03 GMT Wed Mar 8 2023
System image file is "disk0:s72033-advipservicesk9_wan-mz.122-33.SXI5.bin"

[andresdavid]

sh proc cpu sort | ex 0.00
CPU utilization for five seconds: 99%/82%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
272 433763408 276822503 1566 9.83% 9.47% 11.03% 0 IP Input
555 155027180 12409985 12492 3.11% 3.43% 3.28% 0 IP NAT Ager
376 96627752 35821374 2697 2.39% 2.63% 2.40% 0 FM core
495 28073976 126514150 221 0.63% 0.58% 0.53% 0 Port manager per
23 27579572 163602723 168 0.55% 0.54% 0.55% 0 IPC Seat Manager
299 1590412 376479296 4 0.15% 0.13% 0.13% 0 Ethernet Msec Ti
65 1188 89 13348 0.15% 0.05% 0.06% 2 SSH Process
8 34601540 1762287 19634 0.15% 0.20% 0.22% 0 Check heaps
50 1460588 3475613 420 0.07% 0.07% 0.07% 0 Per-Second Jobs
384 1547164 1666384 928 0.07% 0.04% 0.05% 0 HIDDEN VLAN Proc
11 4721992 7407789 637 0.07% 0.12% 0.13% 0 ARP Input
84 2144500 5331072 402 0.07% 0.03% 0.02% 0 DHCP Snooping
63 3579596 147425806 24 0.07% 0.07% 0.07% 0 Net Input
218 690968 749072 922 0.07% 0.03% 0.02% 0 Compute load avg

[Leo Laohoo]

All those values do not add up. 

This could mean there is a memory leak somewhere.  Please post the "first page" of the command

sh process memory sort

[andresdavid]

[Leo Laohoo]

Upgrade the firmware of the switch to something more recent but stay on the SXI or SXJ train.

[Joseph W. Doherty]

On general principle, I endorse @Leo Laohoo's recommendation for updating your IOS release.

Before doing that, and to perhaps help determine if root of the problem is a memory leak, reboot the 6500 and see if the problem doesn't immediately reoccur - such can be a sign of a slow memory leak.

BTW, if you have dual sups, you should, I believe, be able to reboot the standby, flip over to it, and then reboot the former primary; all to minimize any outage.

Since this problem just arose suddenly, you might try to determine if there's any "new" kind of traffic on the network passing through the 6500.

As promised in my PM reply here are 3 more Cisco troubleshooting links that may (or possibly not) help your investigation of high CPU interrupt.

https://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41120-highcpu-interrupts.html

https://www.cisco.com/c/en/us/support/docs/routers/7500-series-routers/41160-highcpu-ip-input.html

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xe-16/213549-troubleshoot-high-cpu-usage-in-catalyst.html

[andresdavid]

Can you help me please with the firmware because I can't find anywhere to download it

[andresdavid]

Thank you very much for all the answers

[MHM Cisco World]

Is this issue solved?

[andresdavid]

not for now, I'm waiting for someone to answer me where I can download the latest upgrade

[MHM Cisco World]

272 433763408 276822503 1566 9.83% 9.47% 11.03% 0 IP Input 

this can be issue, 
check if there is MTU mismatch 

[andresdavid]

I upgraded to version 12.2(33)SXJ10, I see that the CPU is now at 38  %, it's great, I hope I'm not mistaken, that there is a client on the network that has gone to sleep. I have to check in the morning to know everything is perfect