High CPU on 6509e-VSS

[whitegrape]

Hi Guys,

Im facing high RP CPU but im unable to tell which traffic is causing.

sh proc cpu 

CPU utilization for five seconds: 95%/74%; one minute: 93%; five minutes: 92%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
260 18198348683633142802 0 5.27% 1.95% 2.15% 0 SNMP ENGINE
12 20179438563103711806 0 3.91% 4.07% 4.04% 0 ARP Input
24 12651289442771501964 0 2.07% 1.78% 1.74% 0 IPC Seat Manager
560 1282542540 373138232 3437 1.91% 2.41% 2.31% 0 Port manager per
328 9746311203977962122 0 1.59% 1.75% 1.77% 0 IP Input
330 10350723522267427981 0 1.11% 1.26% 1.28% 0 ADJ resolve proc
410 333079676 107170941 3107 0.71% 0.65% 0.65% 0 CEF: IPv4 proces
322 89262516 215671860 413 0.39% 0.63% 0.64% 0 CDP Protocol
616 227367481726024960 13 0.39% 0.39% 0.39% 0 VRRP
254 3137825001970405200 159 0.31% 0.29% 0.29% 0 IP SNMP
368 126109100 44009810 2865 0.31% 0.18% 0.17% 0 QOS Stats Gather
352 54758083677544205 0 0.23% 0.22% 0.23% 0 Ethernet Msec Ti
170 35404 6798 5208 0.23% 0.23% 0.22% 1 SSH Process
61 960399082178699930 0 0.23% 0.20% 0.20% 0 VSIBC process
504 122968883383422803 0 0.23% 0.18% 0.18% 0 IP SLAs Event Pr
88 119296624 207453362 575 0.15% 0.18% 0.16% 0 ARP HA
617 42620283812256236 1 0.15% 0.14% 0.15% 0 HSRP Common
52 16900792 65833906 256 0.15% 0.13% 0.15% 0 Per-Second Jobs
2 215832 12657854 17 0.07% 0.01% 0.00% 0 Load Meter

#sh mls rate-limit
Sharing Codes: S - static, D - dynamic
Codes dynamic sharing: H - owner (head) of the group, g - guest of the group

Rate Limiter Type Status Packets/s Burst Sharing
--------------------- ---------- --------- ----- -------
MCAST NON RPF Off - - -
MCAST DFLT ADJ On 100000 100 Not sharing
MCAST DIRECT CON Off - - -
ACL BRIDGED IN Off - - -
ACL BRIDGED OUT Off - - -
IP FEATURES Off - - -
ACL VACL LOG On 5000 1 Not sharing
MAC PBF IN Off - - -
CEF RECEIVE Off - - -
CEF GLEAN On 200 50 Not sharing
MCAST PARTIAL SC On 100000 100 Not sharing
IP RPF FAILURE On 1000 20 Group:0 S
TTL FAILURE On 15 10 Not sharing
ICMP UNREAC. NO-ROUTE On 1000 20 Group:0 S
ICMP UNREAC. ACL-DROP On 1000 20 Group:0 S
ICMP REDIRECT On 20000 20 Not sharing
MTU FAILURE On 100 10 Not sharing
MCAST IP OPTION Off - - -
UCAST IP OPTION Off - - -
LAYER_2 PDU Off - - -
LAYER_2 PT Off - - -
LAYER_2 PORTSEC Off - - -
LAYER_2 MiniProto Off - - -
DHCP Snooping IN Off - - -
DHCP Snooping OUT Off - - -
ARP Inspection Off - - -
IP ERRORS On 1000 20 Group:0 S
CAPTURE PKT Off - - -
MCAST IGMP On 20000 40 Not sharing
MCAST IPv6 DIRECT CON Off - - -
MCAST IPv6 ROUTE CNTL On 1000 20 Group:8 D/g
MCAST IPv6 *G M BRIDG Off - - -
MCAST IPv6 SG BRIDGE Off - - -
MCAST IPv6 DFLT DROP On 1000 20 Group:8 D/H
MCAST IPv6 SECOND. DR Off - - -
MCAST IPv6 *G BRIDGE Off - - -
MCAST IPv6 MLD Off - - -
IP ADMIS. ON L2 PORT Off - - -
UCAST IP TINY FRAG Off - - -
MCAST IP TINY FRAG Off - - -
UCAST UNKNOWN FLOOD Off - - -
MCAST IPv4 PIM Off - - -
IPV6 BRIDGE Off - - -

Thanks in advance

[whitegrape]

SP switch is ok.

#remote command switch sh proc cpu

CPU utilization for five seconds: 17%/4%; one minute: 19%; five minutes: 20%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 156 2115 73 0.00% 0.00% 0.00% 0 Chunk Manager
2 300688 12652675 23 0.00% 0.00% 0.00% 0 Load Meter

[Leo Laohoo]

Post the complete output the the command "sh proc cpu sort | ex 0.00" and "sh version".

[whitegrape]

Hi!

Thanks for your help.

VSS#sh proc cpu sort | ex 0.00
CPU utilization for five seconds: 96%/76%; one minute: 95%; five minutes: 95%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
260 18201234683633322894 0 4.79% 3.11% 2.54% 0 SNMP ENGINE
12 20185258003103866704 0 4.07% 4.18% 4.09% 0 ARP Input
330 10352666282267620370 0 1.51% 1.45% 1.42% 0 ADJ resolve proc
328 9748778283978140081 0 1.51% 1.58% 1.63% 0 IP Input
560 1282866816 373419606 3435 1.27% 2.27% 2.31% 0 Port manager per
322 89364712 215685973 414 0.63% 0.69% 0.65% 0 CDP Protocol
24 12654220762771756467 0 0.55% 1.84% 1.76% 0 IPC Seat Manager
397 27132876 49726110 545 0.55% 0.68% 0.61% 0 XDR mcast
616 227985841726106414 13 0.39% 0.39% 0.39% 0 VRRP
368 126134588 44012091 2865 0.31% 0.18% 0.17% 0 QOS Stats Gather
254 3138226241970462838 159 0.31% 0.37% 0.32% 0 IP SNMP
238 6840672 258038737 26 0.23% 0.26% 0.24% 0 IPv6 ND
421 25475340 78059125 326 0.23% 0.22% 0.21% 0 L3 Manager
398 125191488 59353908 2109 0.23% 0.18% 0.18% 0 IPC LC Message H
352 54911563677949895 0 0.23% 0.21% 0.22% 0 Ethernet Msec Ti
504 123232763383728169 0 0.23% 0.19% 0.18% 0 IP SLAs Event Pr
88 119325956 207467056 575 0.15% 0.17% 0.16% 0 ARP HA
271 21766516 281773852 77 0.15% 0.12% 0.13% 0 cpf_process_tpQ
617 42732643812457122 1 0.15% 0.14% 0.15% 0 HSRP Common
61 960569602178851315 0 0.15% 0.22% 0.22% 0 VSIBC process
52 16920964 65838113 257 0.15% 0.13% 0.14% 0 Per-Second Jobs
21 9656236 63223969 152 0.07% 0.07% 0.07% 0 IPC Periodic Tim
430 18876868 213123970 88 0.07% 0.07% 0.07% 0 ISIS Upd sparks
423 10098632 122060816 82 0.07% 0.10% 0.08% 0 FM core
259 824088321003662711 82 0.07% 0.08% 0.07% 0 PDU DISPATCHER
361 50476012 317639422 158 0.07% 0.10% 0.09% 0 ISIS Adj sparks
297 33704281958220946 1 0.07% 0.05% 0.06% 0 ACE Tunnel Task
278 1863604 250899673 7 0.07% 0.04% 0.05% 0 Transport Port A
VSS#


#sh version
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 12.2(33)SXJ9, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Mon 16-Mar-15 14:07 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)SX7, RELEASE SOFTWARE (fc1)

VSS uptime is 2 years, 2 days, 14 hours, 59 minutes
Uptime for this control processor is 2 years, 2 days, 14 hours, 9 minutes
Time since VSS switched to active is 2 years, 2 days, 14 hours, 10 minutes
System returned to ROM by reload at 01:54:46 UTC Tue Jan 25 2000 (SP by reload)
System restarted at 00:49:15 SGT Sat Feb 3 2018
System image file is "sup-bootdisk:s72033-adventerprisek9-mz.122-33.SXJ9.bin"
Last reload reason: Reason unspecified



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C6509-E (R7000) processor (revision 1.4) with 983008K/65536K bytes of memory.
Processor board ID SMG1133N63V
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
37 Virtual Ethernet interfaces
246 Gigabit Ethernet interfaces
36 Ten Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

[whitegrape]

VSS#sh proc cpu sort | ex 0.00
CPU utilization for five seconds: 96%/76%; one minute: 95%; five minutes: 95%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
260 18201234683633322894 0 4.79% 3.11% 2.54% 0 SNMP ENGINE
12 20185258003103866704 0 4.07% 4.18% 4.09% 0 ARP Input
330 10352666282267620370 0 1.51% 1.45% 1.42% 0 ADJ resolve proc
328 9748778283978140081 0 1.51% 1.58% 1.63% 0 IP Input
560 1282866816 373419606 3435 1.27% 2.27% 2.31% 0 Port manager per
322 89364712 215685973 414 0.63% 0.69% 0.65% 0 CDP Protocol
24 12654220762771756467 0 0.55% 1.84% 1.76% 0 IPC Seat Manager
397 27132876 49726110 545 0.55% 0.68% 0.61% 0 XDR mcast
616 227985841726106414 13 0.39% 0.39% 0.39% 0 VRRP
368 126134588 44012091 2865 0.31% 0.18% 0.17% 0 QOS Stats Gather
254 3138226241970462838 159 0.31% 0.37% 0.32% 0 IP SNMP
238 6840672 258038737 26 0.23% 0.26% 0.24% 0 IPv6 ND
421 25475340 78059125 326 0.23% 0.22% 0.21% 0 L3 Manager
398 125191488 59353908 2109 0.23% 0.18% 0.18% 0 IPC LC Message H
352 54911563677949895 0 0.23% 0.21% 0.22% 0 Ethernet Msec Ti
504 123232763383728169 0 0.23% 0.19% 0.18% 0 IP SLAs Event Pr
88 119325956 207467056 575 0.15% 0.17% 0.16% 0 ARP HA
271 21766516 281773852 77 0.15% 0.12% 0.13% 0 cpf_process_tpQ
617 42732643812457122 1 0.15% 0.14% 0.15% 0 HSRP Common
61 960569602178851315 0 0.15% 0.22% 0.22% 0 VSIBC process
52 16920964 65838113 257 0.15% 0.13% 0.14% 0 Per-Second Jobs
21 9656236 63223969 152 0.07% 0.07% 0.07% 0 IPC Periodic Tim
430 18876868 213123970 88 0.07% 0.07% 0.07% 0 ISIS Upd sparks
423 10098632 122060816 82 0.07% 0.10% 0.08% 0 FM core
259 824088321003662711 82 0.07% 0.08% 0.07% 0 PDU DISPATCHER
361 50476012 317639422 158 0.07% 0.10% 0.09% 0 ISIS Adj sparks
297 33704281958220946 1 0.07% 0.05% 0.06% 0 ACE Tunnel Task
278 1863604 250899673 7 0.07% 0.04% 0.05% 0 Transport Port A

#sh version
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 12.2(33)SXJ9, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Mon 16-Mar-15 14:07 by prod_rel_team

ROM: System Bootstrap, Version 12.2(17r)SX7, RELEASE SOFTWARE (fc1)

VSS uptime is 2 years, 2 days, 14 hours, 59 minutes
Uptime for this control processor is 2 years, 2 days, 14 hours, 9 minutes
Time since VSS switched to active is 2 years, 2 days, 14 hours, 10 minutes
System returned to ROM by reload at 01:54:46 UTC Tue Jan 25 2000 (SP by reload)
System restarted at 00:49:15 SGT Sat Feb 3 2018
System image file is "sup-bootdisk:s72033-adventerprisek9-mz.122-33.SXJ9.bin"
Last reload reason: Reason unspecified



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C6509-E (R7000) processor (revision 1.4) with 983008K/65536K bytes of memory.
Processor board ID SMG1133N63V
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
37 Virtual Ethernet interfaces
246 Gigabit Ethernet interfaces
36 Ten Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

[Leo Laohoo]

Read the Release Notes for SXJ release.
I suspect you're seeing a cosmetic bug.
Otherwise, contact TAC to get this analyzed.

[whitegrape · Accepted solution]

Hello,

Thanks for your suggestion.

I have found the root cause, it was due to one of the internal ips attacking 100+ plus of icmp packets to outside. 

Thanks.