6509 switch is getting high utilization

[Muthukumar P]

HI Team ,

  we are getting high utilization on 6509 switch and could someone help on this?

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2020.06.28 19:21:02 =~=~=~=~=~=~=~=~=~=~=~=
sh module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1343222E
2 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAD114400Z6
3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1702WLBT
4 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL10436CEM
5 2 Supervisor Engine 720 (Hot) WS-SUP720-3B SAL12394GMJ
6 2 Supervisor Engine 720 (Active) WS-SUP720-3B SAD0922023R
7 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SAL1204EBBU
8 24 CEF720 24 port 1000mb SFP WS-X6724-SFP SAL161171LF

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0023.eba7.6ed0 to 0023.eba7.6eff 3.4 12.2(18r)S1 12.2(33)SXI1 Ok
2 001e.1356.0a04 to 001e.1356.0a33 2.11 12.2(14r)S5 12.2(33)SXI1 Ok
3 fc99.475d.3f0c to fc99.475d.3f3b 4.2 12.2(18r)S1 12.2(33)SXI1 Ok
4 0019.56f0.068c to 0019.56f0.06bb 2.9 12.2(14r)S5 12.2(33)SXI1 Ok
5 001a.a22d.3f90 to 001a.a22d.3f93 5.7 8.5(2) 12.2(33)SXI1 Ok
6 0011.9202.0514 to 0011.9202.0517 4.4 8.1(3) 12.2(33)SXI1 Ok
7 001c.584c.0654 to 001c.584c.0683 2.9 12.2(14r)S5 12.2(33)SXI1 Ok
8 c464.13c6.e7f0 to c464.13c6.e807 5.1 12.2(18r)S1 12.2(33)SXI1 Ok

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Centralized Forwarding Card WS-F6700-CFC SAL134324NG 4.1 Ok
2 Centralized Forwarding Card WS-F6700-CFC SAL1521E1JD 4.1 Ok
3 Centralized Forwarding Card WS-F6700-CFC SAL1703WW9V 4.1 Ok
4 Centralized Forwarding Card WS-F6700-CFC SAD11080AEB 3.1 Ok
5 Policy Feature Card 3 WS-F6K-PFC3B SAL123948Q4 2.4 Ok
5 MSFC3 Daughterboard WS-SUP720 SAL1239487N 3.2 Ok
6 Policy Feature Card 3 WS-F6K-PFC3B SAD092103M1 2.1 Ok
6 MSFC3 Daughterboard WS-SUP720 SAD092005N5 2.3 Ok
7 Centralized Forwarding Card WS-F6700-CFC SAL1205EJBK 4.1 Ok
8 Centralized Forwarding Card WS-F6700-CFC SAL161175T3 4.1 Ok

Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
4 Pass
5 Pass
6 Pass
--More--   7 Pass
8 Pass
IILINMUMCSW01#sh plat
IILINMUMCSW01#sh platform netint
Network IO Interrupt Throttling:
throttle count=141941 (0 in multicast fs), timer count=141941
active=0, configured=1
netint usec=4000, netint mask usec=800
resource netint mask usec=400
inband_throttle_mask_hi = 0x0
inband_throttle_mask_lo = 0x800000
SPD-drop triggered throttles=201
SPD triggered unthrottles=19
Max SPD throttle time=4000 usecs
Buffer depletion triggered throttles=0
ARP queue triggered throttles=0
Low inband activity:throttles=133892, active=0, min=5 pkts/6000 bytes, 1000 usec
Low activity threshold 1500 pps (192)
Current rx packet rate 226 pps (29)
Idle hook unthrottles=1852

[Muthukumar P]

Hi team,

                      Memory utilization details 

Process sbin/iprouting.iosproc, type IOS, PID = 16431

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 1040058 6396760 3750952 2645808 2645808 2633096

Process sbin/cdp2.iosproc, type IOS, PID = 16432

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 1040058 914996 877528 37468 37468 26532

------------------ show memory detailed all failures alloc ------------------

System Memory: 524288K total, 421657K used, 102631K free, 1000K kernel reserved
Lowest(b) : 105000960

Process sbin/ios-base, type IOS, PID = 16406
205320K total, 106556K text, 4K data, 92K stack, 98668K dynamic

Caller Pool Size Alignment When

Process sbin/iprouting.iosproc, type IOS, PID = 16431

Caller Pool Size Alignment When

Process sbin/cdp2.iosproc, type IOS, PID = 16432

Caller Pool Size Alignment When

------------------ show process memory detailed ------------------

System Memory : 524288K total, 421657K used, 102631K free, 1000K kernel reserved
Lowest(b) : 105000960
PID Text Data Stack Dynamic Total Process
1 0 0 0 0 0 kernel
16386 2168 128 12 192 2500 sbin/chkptd.proc
3 644 28 4 192 868 sbin/devc-pty
4 772 32 12 112 928 sbin/devc-mistral.proc
5 604 24 12 96 736 sbin/pipe
6 2384 176 32 284 2876 sbin/dumper.proc
4103 2596 328 32 488 3444 sbin/pcmcia_driver.proc
4104 2596 348 24 408 3376 sbin/bflash_driver.proc
12297 2448 184 20 232 2884 sbin/flashfs_hes.proc
12298 604 24 4 96 728 sbin/mqueue
12299 2796 340 12 372 3520 sbin/dfs_bootdisk.proc
12300 2704 184 28 232 3148 sbin/ldcache.proc
12301 2084 124 8 176 2392 sbin/watchdog.proc
12302 2156 132 20 244 2552 sbin/syslogd.proc
12303 2096 124 24 1136 3380 sbin/name_svr.proc
12304 3112 204 40 440 3796 sbin/wdsysmon.proc
12305 3004 244 76 480 3804 sbin/sysmgr.proc
16402 3004 244 12 240 3500 sbin/sysmgr.proc
16403 2784 172 8 208 3172 sbin/syslog_dev.proc
16404 2108 128 16 192 2444 sbin/itrace_exec.proc
16405 2220 136 20 232 2608 sbin/packet.proc
16406 106556 4 92 98668 205320 sbin/ios-base

Process sbin/ios-base, type IOS, PID = 16406
205320K total, 106556K text, 4K data, 92K stack, 98668K dynamic
Heap : 103588096 total, 99321584 used, 4266512 free
Task TTY Allocated Freed Holding Getbufs Retbufs TaskName
0 0 97472464 18752 97064016 0 0 *Init*
0 0 620344096 605514512 8666400 6951228 0 *Dead*
563 0 522146992 520217144 1980864 0 0 IP SNMP
238 0 7244824 5098064 1784152 0 0 RPC idprom-MP
305 0 3200538992 3199238848 1247488 0 0 QOS Stats Gathe
9 1 4355040 3280280 1074928 0 0 SSH Process
377 0 2054128 1203472 872104 0 0 FM core
42 0 9469000 8585760 732752 0 0 Entity MIB API
34 0 1005384 330472 650032 0 0 IPC Seat Manage
356 0 1714600 1096544 569728 0 0 IPC LC Message
138 0 631440 12488 526560 0 0 PF_Init Process
45 0 517344 432 517352 231552 0 EEM ED Syslog
349 0 695528 334088 360248 0 0 IPRIB DIST
2 4 864240 618184 246056 0 0 Hot Service Tas
360 0 227256 1248 224632 0 0 CEF: IPv4 proce
25 0 333992 81680808 223712 0 0 ARP Input
55 0 575432 332088 213512 0 0 rf proxy rp age
145 4 29591416 29671880 189288 0 0 SSH Process
354 0 219416 7166376 181272 0 0 XDR mcast
17 0 143200 0 144080 118100 118100 Pool Manager
303 0 135320 192 136008 0 0 SSM connection
371 0 817808 332920 116136 0 0 L3 Manager
503 0 141120 27864 112568 0 0 Entity MIB C6k
299 0 81677296 0 106752 0 0 ADJ resolve pro
282 0 1416240 1410056 96104 0 0 SW VLAN RP
562 0 83840 192 82608 0 0 DHCPD Receive
69 0 224984 113432 78008 0 0 Net Background
4 0 510232824 509594416 77752 0 0 Service Task
559 0 61692704 51651688 71344 0 0 PIM Process
479 0 49976680 49886304 66472 0 0 RPC online_diag
246 0 63400 4520 60608 0 0 OBFL Cfg Dispat
64 0 109376 51408 58848 0 0 NSP:IPC Task
92 0 169152 159912 57856 0 0 Env action
7 0 56868432 56816568 54304 0 0 Chunk Manager
59 0 58224 6080 53024 0 0 RPCP Slave
0 0 4023941824 2694860048 52128 0 0 *Neutrino*
433 0 50368 192 51056 0 0 RADIUS
5 0 18531128 18524192 46072 0 0 Service Task
446 0 193640 149688 44832 0 0 Crypto ACL
545 0 61328 15032 44792 0 0 IGMP Input
517 0 39464 696 39648 0 0 MLD
560 0 5678256 5637512 39440 0 0 MLSM Process
50 0 123688 41416 38912 0 0 Cat6k SNMP
53 0 69816 364624 34032 0 0 IF-MGR control
73 0 1769624 1736400 34024 0 0 Per-Second Jobs
293 0 7045816 5508104 31848 0 0 IP Input
469 0 807991504 807963304 29080 0 0 Port manager pe
481 0 131664 101936 27192 0 0 RPC pagp_switch
269 0 91632 49464 27176 0 0 CMFI RP process
523 0 25072 192 22392 0 0 HSRP IPv4
1 4 599912 577920 21992 0 0 Hot Service Tas
504 0 18872 0 19752 0 0 RP L2 Applicati
260 0 19088 384 19584 0 0 ACE_REQUESTS ha
261 0 19800 1280 18400 0 0 ACE Tunnel Task
568 0 1160688 1144952 16616 0 0 SNMP Traps
3 4 650296 634688 15416 0 0 Hot Service Tas
317 0 33416 18824 15280 0 0 Const Rate Limi
318 0 393280 460608 14768 0 0 HWIF QoS Proces
418 0 13336 0 14216 0 0 RSVP
369 0 14192 520 13760 0 0 CWAN-RP SCP Inp
566 0 11416 192 12104 0 0 IP SNMPV6
104 0 183568 169864 11624 0 0 cpf_msg_rcvq_pr
20 0 18503080 18492608 11352 0 0 EnvMon
236 0 116424 107688 9600 0 0 PF_Split Sync P
508 0 10927208 10922600 9528 0 0 crypto sw pk pr
440 0 9472 1864 8488 0 0 Crypto CA
404 0 17696 10848 7728 0 0 MRIB RP Proxy
82 0 55664 0 7544 0 0 ICC Nego
364 0 8008 1432 7456 0 0 LSD HA Proc
242 0 6832 2104 6152 0 0 HSRP Common
444 0 5168 192 5856 0 0 Lawful Intercep
405 0 6208 192 5856 0 0 PIM HA
87 0 82824 69976 5816 0 0 draco-oir-proce
191 0 4784 0 5664 0 0 IPC ISSU Versio
521 0 4784 0 5664 0 0 PAgP-MP Process
542 0 5032 248 5664 0 0 CWAN Bridge tas
150 0 4784 0 5664 0 0 IPC ISSU Versio
232 0 4784 0 5664 0 0 IPC ISSU Versio
450 0 22256 12080 5440 0 0 IPSEC key engin
74 0 4544 26256 5424 2544 0 Per-minute Jobs
141 0 11032 5808 5208 0 0 SSH Event handl
541 0 2220784 2218016 3648 0 0 Syslog
289 0 48120 0 3624 0 0 network_out Svc
401 0 2832 192 3520 0 0 L2TRACE SERVER
400 0 7024 4408 3496 0 0 LDP HA
75 0 19808 0 2864 0 0 Cat6k NTI ICC p
76 0 68920 57384 2632 0 0 Collection proc
432 0 8840 0 2336 0 0 CEF RP IPC Back
390 0 5296 3248 1888 0 0 PPP HA
120 0 1904 0 1800 0 0 CWAN APS HA Pro
128 0 5008 7192 1784 0 0 ARP HA
124 0 3856 1976 1776 0 0 RF CWAN HA Proc
343 0 1080 8152 1768 0 0 SXP CORE
123 0 1880 0 1760 0 0 CWAN HA VLAN SY
554 0 1040 192 1728 0 0 CTS credentials
117 0 1040 192 1728 0 0 CWRP SPA TSM HA
535 0 4088 3080 1728 0 0 CTS HA
428 0 920 192 1608 0 0 TPLUS
367 0 728 0 1608 0 0 HTTP CORE
334 0 1112 384 1608 0 0 CTS CORE
406 0 912 192 1600 0 0 IPHC RP Process
449 0 616 192 1304 0 0 Crypto IKMP
569 0 26248 25872 1256 0 0 NTP
454 0 344 0 1224 0 0 CCSWVOICE
348 0 94936 95080 1184 0 0 CEF background
281 0 3752 3992 1176 0 0 Gemini L3 forwa
443 0 3368 1824 1160 0 0 crypto engine p
530 0 3416520 3441040 1144 0 0 Online Diag EEM
351 0 192 0 1136 0 0 DHCPD Database
44 0 2048 752 1120 0 0 ifIndex Receive
556 0 192 0 1072 0 0 Online Diag CNS
296 0 384 192 1072 0 0 Spanning Tree

[balaji.bandi]

Can you post below output :

#show processes cpu | ex 0.0%

#sh processes cpu history

# show version

[Muthukumar P]

------------------ show version ------------------

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9_WAN-VM), Version 12.2(33)SXI11, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Fri 15-Feb-13 13:58 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)

IILINMUMCSW01 uptime is 1 day, 10 hours, 43 minutes
Uptime for this control processor is 1 day, 10 hours, 41 minutes
Time since IILINMUMCSW01 switched to active is 1 day, 10 hours, 41 minutes
System returned to ROM by power cycle at 06:25:01 UTC Sat Jul 8 2000 (SP by power on)
System restarted at 08:40:55 IST Sat Jun 27 2020
System image file is "disk0:s72033-adventerprisek9_wan-vz.122-33.SXI11.bin"
Last reload reason: reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C6509-E (R7000) processor (revision 1.3) with 516096K/8192K bytes of memory.
Processor board ID SMG1124N5WW
SR71000 CPU at 600Mhz, Implementation 1284, Rev 1.2, 512KB L2 Cache
Last reset from power-on
33 Virtual Ethernet interfaces
268 Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

------------------ show process cpu detailed ------------------

CPU utilization for five seconds: 89%/11%; one minute: 59%; five minutes: 44%
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
1 15.0% 1.8% 0.5% kernel 5m50s
1 11.0% 41.5% 55.6% [idle thread] 0 Ready 1d03h
2 0.0% 0.0% 0.0% 63 Receive 0.000
3 0.0% 0.0% 0.0% 10 Receive 0.000
4 7.4% 0.8% 0.1% 10 Receive 2m25s
5 0.0% 0.0% 0.0% 63 Receive 0.000
6 0.0% 0.0% 0.0% 63 Receive 0.000
7 0.0% 0.0% 0.0% 63 Receive 0.000
8 0.0% 0.0% 0.0% 63 Receive 0.000
9 0.0% 0.0% 0.0% 63 Receive 0.000
10 0.0% 0.0% 0.0% 63 Receive 0.000
11 0.0% 0.0% 0.0% 63 Receive 0.000
12 0.0% 0.0% 0.0% 10 Receive 0.000
13 0.0% 0.0% 0.0% 10 Receive 0.000
14 0.0% 0.0% 0.0% 10 Receive 1.263
15 7.6% 1.0% 0.4% 10 Ready 3m20s
16 0.0% 0.0% 0.0% 10 Receive 0.000
17 0.0% 0.0% 0.0% 10 Receive 2.236
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
3 0.0% 0.0% 0.0% devc-pty 0.014
1 0.0% 0.0% 0.0% 10 Receive 0.014
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
4 0.0% 0.0% 0.0% devc-mistral.proc 33.574
1 0.0% 0.0% 0.0% 10 Receive 22.449
2 0.0% 0.0% 0.0% 20 Intr 11.125
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
5 0.0% 0.0% 0.0% pipe 0.006
2 0.0% 0.0% 0.0% 10 Receive 0.000
3 0.0% 0.0% 0.0% 10 Receive 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
6 0.0% 0.0% 0.0% dumper.proc 0.432
1 0.0% 0.0% 0.0% 10 Receive 0.046
2 0.0% 0.0% 0.0% 10 Receive 0.001
3 0.0% 0.0% 0.0% 10 Receive 0.279
4 0.0% 0.0% 0.0% 10 Receive 0.026
5 0.0% 0.0% 0.0% 10 Receive 0.077
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
4103 0.0% 0.0% 0.0% pcmcia_driver.proc 8.968
1 0.0% 0.0% 0.0% 10 Receive 0.000
2 0.0% 0.0% 0.0% 10 Sigwaitin 0.000
3 0.0% 0.0% 0.0% 10 Nanosleep 8.667
4 0.0% 0.0% 0.0% 10 Receive 0.246
5 0.0% 0.0% 0.0% 10 Receive 0.001
6 0.0% 0.0% 0.0% 10 Receive 0.008
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
4104 0.0% 0.0% 0.0% bflash_driver.proc 10.394
1 0.0% 0.0% 0.0% 10 Receive 0.043
2 0.0% 0.0% 0.0% 10 Sigwaitin 0.000
3 0.0% 0.0% 0.0% 10 Receive 0.662
5 0.0% 0.0% 0.0% 10 Receive 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12297 0.0% 0.0% 0.0% flashfs_hes.proc 3.976
1 0.0% 0.0% 0.0% 10 Receive 0.000
3 0.0% 0.0% 0.0% 10 Receive 0.254
4 0.0% 0.0% 0.0% 10 Receive 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12298 0.0% 0.0% 0.0% mqueue 0.006
1 0.0% 0.0% 0.0% 10 Receive 0.006
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12299 0.0% 0.0% 0.0% dfs_bootdisk.proc 0.294
1 0.0% 0.0% 0.0% 10 Receive 0.294
2 0.0% 0.0% 0.0% 10 Receive 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12300 0.0% 0.0% 0.0% ldcache.proc 0.534
1 0.0% 0.0% 0.0% 10 Receive 0.001
2 0.0% 0.0% 0.0% 10 Sigwaitin 0.000
5 0.0% 0.0% 0.0% 10 Receive 0.228
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12301 0.0% 0.0% 0.0% watchdog.proc 0.165
1 0.0% 0.0% 0.0% 10 Nanosleep 0.165
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12302 0.0% 0.0% 0.0% syslogd.proc 0.216
1 0.0% 0.0% 0.0% 10 Receive 0.216
2 0.0% 0.0% 0.0% 10 Receive 0.000
3 0.0% 0.0% 0.0% 10 Sigwaitin 0.000
4 0.0% 0.0% 0.0% 10 Condvar 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12303 0.0% 0.0% 0.0% name_svr.proc 14.400
1 0.0% 0.0% 0.0% 10 Receive 0.057
2 0.0% 0.0% 0.0% 10 Receive 0.000
4 0.0% 0.0% 0.0% 10 Receive 4.178
5 0.0% 0.0% 0.0% 10 Receive 9.979
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12304 0.0% 0.0% 0.0% wdsysmon.proc 79.826
1 0.0% 0.0% 0.0% 10 Receive 0.065
2 0.0% 0.0% 0.0% 10 Receive 57.863
3 0.0% 0.0% 0.0% 63 Nanosleep 0.142
4 0.0% 0.0% 0.0% 10 Receive 20.948
5 0.0% 0.0% 0.0% 10 Receive 0.005
6 0.0% 0.0% 0.0% 10 Receive 0.002
7 0.0% 0.0% 0.0% 10 Receive 0.000
8 0.0% 0.0% 0.0% 10 Receive 0.799
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
12305 0.0% 0.0% 0.0% sysmgr.proc 0.560
1 0.0% 0.0% 0.0% 10 Receive 0.097
2 0.0% 0.0% 0.0% 10 Sigwaitin 0.000
3 0.0% 0.0% 0.0% 10 Receive 0.005
4 0.0% 0.0% 0.0% 10 Intr 0.000
5 0.0% 0.0% 0.0% 10 Receive 0.000
6 0.0% 0.0% 0.0% 10 Receive 0.000
7 0.0% 0.0% 0.0% 10 Receive 0.000
8 0.0% 0.0% 0.0% 10 Receive 0.000
9 0.0% 0.0% 0.0% 10 Receive 0.000
10 0.0% 0.0% 0.0% 10 Receive 0.000
11 0.0% 0.0% 0.0% 10 Receive 0.000
12 0.0% 0.0% 0.0% 10 Receive 0.000
13 0.0% 0.0% 0.0% 10 Receive 0.000
14 0.0% 0.0% 0.0% 10 Receive 0.004
15 0.0% 0.0% 0.0% 10 Receive 0.423
16 0.0% 0.0% 0.0% 23 Receive 0.024
17 0.0% 0.0% 0.0% 10 Receive 0.003
18 0.0% 0.0% 0.0% 10 Receive 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16386 0.0% 0.0% 0.0% chkptd.proc 0.257
1 0.0% 0.0% 0.0% 10 Receive 0.257
2 0.0% 0.0% 0.0% 10 Receive 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16402 0.0% 0.0% 0.0% sysmgr.proc 0.287
1 0.0% 0.0% 0.0% 10 Receive 0.287
2 0.0% 0.0% 0.0% 10 Sigwaitin 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16403 0.0% 0.0% 0.0% syslog_dev.proc 0.250
1 0.0% 0.0% 0.0% 10 Receive 0.250
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16404 0.0% 0.0% 0.0% itrace_exec.proc 0.230
1 0.0% 0.0% 0.0% 10 Receive 0.028
2 0.0% 0.0% 0.0% 10 Receive 0.201
3 0.0% 0.0% 0.0% 10 Receive 0.000
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16405 0.0% 0.0% 0.0% packet.proc 16.503
1 0.0% 0.0% 0.0% 10 Receive 0.261
2 0.0% 0.0% 0.0% 10 Sigwaitin 16.085
3 0.0% 0.0% 0.0% 10 Nanosleep 0.152
5 0.0% 0.0% 0.0% 10 Receive 0.003
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16406 65.0% 44.2% 31.2% ios-base 5h49m
1 1.6% 2.9% 1.7% 10 Receive 23m59s
2 0.0% 0.0% 0.0% 5 Condvar 3.377
3 7.3% 2.6% 2.9% 10 Receive 28m48s
4 8.6% 2.1% 2.6% 10 Reply 28m24s
5 0.0% 0.0% 0.0% 11 Nanosleep 3.697
6 3.9% 3.3% 3.2% 21 Intr 42m23s
7 5.8% 5.6% 4.0% 22 Intr 55m12s
8 0.0% 0.0% 0.0% 23 Intr 6.341
9 0.0% 0.0% 0.0% 25 Intr 0.000
10 1.8% 3.5% 1.3% 10 Receive 29m28s
11 0.2% 0.1% 0.1% 10 Receive 2m28s
12 0.0% 0.0% 0.0% 10 Condvar 0.001
13 0.0% 0.0% 0.0% 20 Sigwaitin 1.451
14 9.8% 6.4% 2.0% 10 Receive 22m53s
15 4.0% 3.0% 2.6% 10 Reply 27m41s
16 0.0% 0.0% 0.0% 10 Reply 0.173
17 3.0% 4.7% 3.4% 10 Receive 25m45s
18 1.3% 2.8% 1.0% 10 Receive 30m45s
19 12.6% 4.4% 3.4% 10 Receive 21m14s
20 5.1% 2.8% 3.0% 10 Receive 9m39s

Process sbin/ios-base, type IOS, PID = 16406
CPU utilization for five seconds: 49%/11%; one minute: 28%; five minutes: 15%
Task Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Prio Task Name
1 148 4775 30 0.00% 0.00% 0.00% 4 P Hot Service Task
2 172 4768 36 0.00% 0.00% 0.00% 4 P Hot Service Task
3 143 4763 30 0.00% 0.00% 0.00% 4 P Hot Service Task
4 1138274 6109877 186 0.89% 1.30% 1.37% 0 M Service Task
5 7763 529294 14 0.22% 0.03% 0.00% 0 M Service Task
6 759 95411 7 0.00% 0.00% 0.00% 0 M Service Task
7 500 11344 44 0.00% 0.00% 0.00% 0 C Chunk Manager
8 0 1 0 0.00% 0.00% 0.00% 0 H Connection Mgr
9 7024 4306 1631 0.00% 3.07% 1.79% 1 M SSH Process
10 0 1 0 0.00% 0.00% 0.00% 0 M EM Action CNS
11 0 1 0 0.00% 0.00% 0.00% 0 M PF Redun ICC Req
12 0 1 0 0.00% 0.00% 0.00% 0 M PF Redun ICC Req
13 0 149 0 0.00% 0.00% 0.00% 0 M Retransmission o
14 1 11 90 0.00% 0.00% 0.00% 0 M IPC ISSU Dispatc
15 1785 24933 71 0.00% 0.00% 0.00% 0 C Load Meter
16 451042 27137 16620 0.00% 0.24% 0.31% 0 L Check heaps
17 1 26 38 0.00% 0.00% 0.00% 0 C Pool Manager
18 0 1 0 0.00% 0.00% 0.00% 0 M ext_log_pak Svc
19 0 2 0 0.00% 0.00% 0.00% 0 M Timers
20 22765 94669 240 0.00% 0.02% 0.00% 0 M EnvMon
21 0 1 0 0.00% 0.00% 0.00% 0 L AAA_SERVER_DEADT
22 0 2 0 0.00% 0.00% 0.00% 0 M AAA high-capacit
23 0 1 0 0.00% 0.00% 0.00% 0 M Policy Manager
24 0 1 0 0.00% 0.00% 0.00% 0 M Crash writer
25 3557555 6004383 592 4.05% 4.34% 4.14% 0 M ARP Input
26 0 1 0 0.00% 0.00% 0.00% 0 M raw_enqueue Svc
27 2 2082 0 0.00% 0.00% 0.00% 0 M IPC Dynamic Cach
28 0 1 0 0.00% 0.00% 0.00% 0 M NTI Example Proc
29 0 1 0 0.00% 0.00% 0.00% 0 M ipc_client_ Svc
30 0 1 0 0.00% 0.00% 0.00% 0 H IPC Punt Process
31 88653 124546 711 0.07% 0.06% 0.06% 0 M IPC Periodic Tim
32 325 124579 2 0.00% 0.00% 0.00% 0 M IPC Deferred Por
33 0 1 0 0.00% 0.00% 0.00% 0 M IPC Process leve
34 1075325 6962337 154 2.21% 1.32% 1.05% 0 M IPC Seat Manager
35 0 1 0 0.00% 0.00% 0.00% 0 M IPC Session Serv
36 0 1 0 0.00% 0.00% 0.00% 0 M IPC Stdby Update
37 0 1 0 0.00% 0.00% 0.00% 0 M IPC ISSU Version
38 0 2 0 0.00% 0.00% 0.00% 0 M Serial Backgroun
39 0 2 0 0.00% 0.00% 0.00% 0 M DDR Timers
40 0 2 0 0.00% 0.00% 0.00% 0 M Dialer event
41 33 12486 2 0.00% 0.00% 0.00% 0 H Compute SRP rate
42 776 519 1495 0.00% 0.00% 0.00% 0 L Entity MIB API
43 0 1 0 0.00% 0.00% 0.00% 0 M IFS Agent Manage
44 1 17 58 0.00% 0.00% 0.00% 0 M ifIndex Receive
45 482 2424 198 0.00% 0.00% 0.00% 0 M EEM ED Syslog

[Muthukumar P]

------------------ show process cpu history ------------------

7898889999988899888887776544433333333333333333334333333334
4419795431041611975218403560243434545671223235792442246990
100 *
90 ********* ******
80 *********************
70 ************************
60 **************************
50 ***************************
40 ***************************** * *** **** ****
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
* = maximum CPU% # = average CPU%

8744455545565455566555554445455444545544544557565445484577
4714733693222916311249703420400149081579067546094225140379
100
90
80 ** * * **
70 ** * * * **
60 ** * * * ** ** * * * * * * **
50 #* ******************** * ** ************#*#* * * **#
40 ##**####################*######**###############***#*#*###
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

8999997674746769747976811111111199
5546254548746361182401465634444301
100 * * *
90 ****** * * **
80 ****** * * * * **
70 ********* * ***** *** * **
60 ********* * ***** ***** **
50 *********** *********** **
40 ####******************* **
30 #####################** **
20 ######################**** **
10 #######################*********##
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

------------------ show file systems ------------------

File Systems:

Size(b) Free(b) Type Flags Prefixes
- - ram rw tmp:
- - opaque rw system:
- - opaque ro flexwan-fpd:
65536000 60077696 flash rw bootflash:
* 256503808 72962048 disk rw disk0:#
- - disk rw disk1:
65536000 8894076 flash rw sup-bootflash:
147993198 0 opaque ro sup-microcode:
0 81817428 opaque wo sup-image:
129004 125496 nvram rw const_nvram:
1964024 1733146 nvram rw nvram:
- - opaque rw null:
- - opaque ro tar:
- - network rw rcp:
- - network rw tftp:
- - network rw http:
- - network rw scp:
- - network rw ftp:
- - network rw https:
- - opaque ro cns:
15990784 15990784 flash rw dfc#8-bootflash:
15990784 15990784 flash rw dfc#7-bootflash:
15990784 15990784 flash rw dfc#4-bootflash:
15990784 15990784 flash rw dfc#3-bootflash:
15990784 15990784 flash rw dfc#2-bootflash:
15990784 15990784 flash rw dfc#1-bootflash:
1964024 1732122 nvram rw slavenvram:
129004 125496 nvram rw slaveconst_nvram:
256016384 73211904 disk rw slavedisk0:#
- - disk rw slavedisk1:
65536000 13238884 flash rw slavesup-bootflash:
65536000 61883044 flash rw slavebootflash:
- - nvram rw slavercsf:
- - opaque rw slavesystem:
- - opaque rw revrcsf:

[haulaian]

You are running a modular code for the 6500 called "ION" which is outdated.

ION is an early concept for 6500 for IOS modularity.

looking at your outputs we can see:

PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16406 65.0% 44.2% 31.2% ios-base 5h49m <<<<=====

Next you would want to look at:
show process cpu detailed <PID> sorted | ex 0.00 

PID here is going to be 16406.

Then you will see subtasks under ios-base which are the subprocesses.
When you find which subprocess is high, you can inject it in:

show process cpu detailed 16406 task <#>
<#> will be the task id that is high.

-Further more, you can still use debug netdr tool tool to find the top talkers.
debug netdr capture rx 

show netdr captured-packets (make sure you are not using console access, as it will print 4096 packets in the output, use SSH instead)

-This tool will display the packets punted to the cpu (since we selected rx)

[balaji.bandi]

sometimes hard to read full output (maybe missing small information)

can you post below output :

#show processes cpu | ex 0.0

[Muthukumar P]

IILINMUMCSW01#sh processes cpu | ex 0.0
CPU utilization for five seconds: 33%/16%; one minute: 33%; five minutes: 34%
PID 5Sec 1Min 5Min Process
16406 24.1% 23.5% 23.8% ios-base
16427 0.5% 0.7% 0.8% raw_ip.proc
16429 3.5% 3.6% 3.4% tcp.proc
16430 1.0% 1.0% 1.0% udp.proc
16431 2.6% 2.9% 3.0% iprouting.iosproc
16432 1.1% 1.2% 1.2% cdp2.iosproc

[balaji.bandi]

what is the main function of this switch in the network as is see ios-base punting the CPU.

[Muthukumar P]

mean , you are saying firmware version issue..

[haulaian]

Not Necessarily. as i posted earlier you can use those commands to find the subprocess and what packets are being punted to the cpu.

i also suggest that you check out https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/63992-6k-high-cpu.html for more context of how to identify a cause for high cpu utilization.

[Muthukumar P]

Hi,

 Really i am not understand , and could i know that which command output required? Please suggest..

[balaji.bandi]

Maybe at this stage, we do not know, you have not provided that information I have requested, what is the device function in the network. 

[Muthukumar P]

HI,

         This is core switch.. and we are in process to upgrade the version.. 

[Leo Laohoo]

If you can upgrade the firmware, use the latest SXJ.