04-25-2017 07:00 AM - edited 03-08-2019 10:19 AM
Hello,
Our switch(3560) is having high CPU utilization. I would appreciate if someone could help resolve this issue.
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
97 6141904 28793325 213 46.18% 45.98% 45.86% 0 hpm main process
43 769359957 788656 975750 23.46% 23.00% 22.91% 0 SFF8472
133 371595715 46875686 7927 4.81% 4.30% 4.29% 0 Hulc LED Process
100 41374376 2356017 17561 1.35% 1.44% 1.45% 0 hpm counter proc
136 396 123 3219 0.45% 0.40% 0.11% 1 SSH Process
141 10850144 585806 18521 0.30% 0.34% 0.35% 0 HQM Stack Pro
Solved! Go to Solution.
04-25-2017 08:34 AM
Hi
Any change or behavior that you have noted? Please check these links:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtl04815/?referring_site=bugquickviewredir
https://supportforums.cisco.com/discussion/11176171/hpm-main-process-high-cpu
If the case persists, you could open a ticket with the Cisco TAC.
Hope it is useful
:-)
04-25-2017 08:34 AM
Hi
Any change or behavior that you have noted? Please check these links:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtl04815/?referring_site=bugquickviewredir
https://supportforums.cisco.com/discussion/11176171/hpm-main-process-high-cpu
If the case persists, you could open a ticket with the Cisco TAC.
Hope it is useful
:-)
04-25-2017 09:55 AM
Hello,
Here's the output from my switch, not sure what it means from command "sh controllers cpu-int"
I would appreciate if you guys can interpret the output of this command.
ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain
-------------------------------------------------------------------------
ASIC0 0 0 0 0 0 0
ASIC1 0 0 0 0 0 0
ASIC2 0 0 0 0 0 0
HOL Fix Counts
--------------
No Fixes: 0 Added: 0 In Use: 0 Both: 0
CPU Heartbeat Statistics
Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
63129775 0 0 0 0 1
Rx Delay
0 1 2 3 4
---------- ---------- ---------- ---------- ----------
63129775 0 0 0 0
AddlDelay AdvanceCnt
---------- ----------
0 0
Rx Retries by RetryCount
0 1 2 3 4 5 6
---------- ---------- ---------- ---------- ---------- ---------- ----------
63129906 0 0 0 0 0 0
7 8 9
---------- ---------- ----------
0 0 0
AddlRetry
----------
0
cpu-queue-frames retrieved dropped invalid hol-block stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc 1 0 0 0 0
stp 104537717 0 0 0 0
ipc 1 0 0 0 0
routing protocol 260328862 0 0 0 0
L2 protocol 1851288 0 0 0 0
remote console 0 0 0 0 0
sw forwarding 0 0 0 0 0
host 304813 0 0 0 0
broadcast 168902820 0 0 0 0
cbt-to-spt 0 0 0 0 0
igmp snooping 26328729 0 0 0 0
icmp 0 0 0 0 0
logging 0 0 0 0 0
rpf-fail 0 0 0 0 0
dstats 0 0 0 0 0
cpu heartbeat 63129995 0 0 0 0
cpu-queue static inuse static added
----------------- ------------ ------------
rpc 0 0
stp 0 0
ipc 0 0
routing protocol 0 0
L2 protocol 0 0
remote console 0 0
sw forwarding 0 0
host 0 0
broadcast 0 0
cbt-to-spt 0 0
igmp snooping 0 0
icmp 0 0
logging 0 0
rpf-fail 0 0
dstats 0 0
cpu heartbeat 0 0
Supervisor ASIC receive-queue parameters
----------------------------------------
queue 0 maxrecevsize 0 pakhead 0 paktail 0
queue 1 maxrecevsize 7E0 pakhead 4704894 paktail 41EED64
queue 2 maxrecevsize 0 pakhead 0 paktail 0
queue 3 maxrecevsize 7E0 pakhead 48E93EC paktail 49020F4
queue 4 maxrecevsize 7E0 pakhead 47113F8 paktail 4709208
queue 5 maxrecevsize 7E0 pakhead 0 paktail 0
queue 6 maxrecevsize 7E0 pakhead 48C874C paktail 48E51B0
queue 7 maxrecevsize 7E0 pakhead 485F0E4 paktail 48684E0
queue 8 maxrecevsize 7E0 pakhead 48765EC paktail 48828D4
queue 9 maxrecevsize 7E0 pakhead 47D455C paktail 47D455C
queue A maxrecevsize 7E0 pakhead 47BF8F4 paktail 40DBB14
queue B maxrecevsize 7E0 pakhead 4902D0C paktail 4906330
queue C maxrecevsize 7E0 pakhead 47DBF4C paktail 47DF570
queue D maxrecevsize 7E0 pakhead 47D0A58 paktail 47D407C
queue E maxrecevsize 0 pakhead 0 paktail 0
queue F maxrecevsize 7E0 pakhead 40D85BC paktail 40D8220
Supervisor ASIC exception status
--------------------------------
Receive overrun 00000000 Transmit overrun 00000000
FrameSignatureErr 00000000 MicInitialize 00000002
BadFrameErr 00000000 LenExceededErr 00000000
BadJumboSegments 00000000
04-25-2017 09:56 AM
And this is from the command "sh ip traffic"
IP statistics:
Rcvd: 3008199 total, 87381 local destination
0 format errors, 0 checksum errors, 0 bad hop count
0 unknown protocol, 2920818 not a gateway
0 security failures, 0 bad options, 89598 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 89598 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
Bcast: 0 received, 3 sent
Mcast: 0 received, 0 sent
Sent: 88143 generated, 0 forwarded
Drop: 1 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied, 0 source IP address zero
ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 1 redirects, 2 unreachable
57527 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
Sent: 0 redirects, 8 unreachable, 0 echo, 57527 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 info reply, 0 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
TCP statistics:
Rcvd: 8702 total, 0 checksum errors, 22 no port
Sent: 8162 total
PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
State-Refresh: 0/0
IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
UDP statistics:
Rcvd: 21152 total, 0 checksum errors, 37 no port
Sent: 22449 total, 0 forwarded broadcasts
ARP statistics:
Rcvd: 218218 requests, 353 replies, 0 reverse, 0 other
Sent: 91 requests, 217389 replies (0 proxy), 0 reverse
Drop due to input queue full: 0
04-25-2017 10:01 AM
Hi Julio,
Thanks for the the info. Please note that no links are flapping and we have around 20+ vlans created.
Below are some of the output of sh controllers cpu-int and sh ip traffic commands. Hopefully you guys can see the real problem that is causing this hign cpu utilization.
04-27-2017 05:50 AM
it's working now. thanks.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide