HIGH CPU utilization on cisco 35 60

[orthicon2009]

Hello,

Our switch(3560) is having high CPU utilization. I would appreciate if someone could help resolve this issue.

 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  97     6141904  28793325        213 46.18% 45.98% 45.86%   0 hpm main process
  43   769359957    788656     975750 23.46% 23.00% 22.91%   0 SFF8472
 133   371595715  46875686       7927  4.81%  4.30%  4.29%   0 Hulc LED Process
 100    41374376   2356017      17561  1.35%  1.44%  1.45%   0 hpm counter proc
 136         396       123       3219  0.45%  0.40%  0.11%   1 SSH Process
 141    10850144    585806      18521  0.30%  0.34%  0.35%   0 HQM Stack Pro

[Julio E. Moisa · Accepted solution]

Hi

Any change or behavior that you have noted? Please check these links:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtl04815/?referring_site=bugquickviewredir

https://supportforums.cisco.com/discussion/11176171/hpm-main-process-high-cpu

If the case persists, you could open a ticket with the Cisco TAC.

Hope it is useful

:-)

[orthicon2009]

Hello,

Here's the output from my switch, not sure what it means from command "sh controllers cpu-int"

I would appreciate if you guys can interpret the output of this command.

ASIC    Rxbiterr   Rxunder    Fwdctfix   Txbuflos   Rxbufloc   Rxbufdrain
-------------------------------------------------------------------------
ASIC0     0          0          0          0          0          0
ASIC1     0          0          0          0          0          0
ASIC2     0          0          0          0          0          0

HOL Fix Counts
--------------
No Fixes:          0 Added:          0 In Use:          0 Both:          0

CPU Heartbeat Statistics

Tx Success Tx Fail    1st Thr    2nd Thr    Unthr      RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
  63129775          0          0          0          0          1

Rx Delay
         0          1          2          3          4
---------- ---------- ---------- ---------- ----------
  63129775          0          0          0          0

AddlDelay AdvanceCnt
---------- ----------
         0          0

Rx Retries by RetryCount
         0          1          2          3          4          5          6
---------- ---------- ---------- ---------- ---------- ---------- ----------
  63129906          0          0          0          0          0          0

         7          8          9
---------- ---------- ----------
         0          0          0

AddlRetry
----------
         0


cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc               1          0          0          0          0
stp               104537717  0          0          0          0
ipc               1          0          0          0          0
routing protocol  260328862  0          0          0          0
L2 protocol       1851288    0          0          0          0
remote console    0          0          0          0          0
sw forwarding     0          0          0          0          0
host              304813     0          0          0          0
broadcast         168902820  0          0          0          0
cbt-to-spt        0          0          0          0          0
igmp snooping     26328729   0          0          0          0
icmp              0          0          0          0          0
logging           0          0          0          0          0
rpf-fail          0          0          0          0          0
dstats            0          0          0          0          0
cpu heartbeat     63129995   0          0          0          0

cpu-queue         static inuse static added
----------------- ------------ ------------
rpc               0            0
stp               0            0
ipc               0            0
routing protocol  0            0
L2 protocol       0            0
remote console    0            0
sw forwarding     0            0
host              0            0
broadcast         0            0
cbt-to-spt        0            0
igmp snooping     0            0
icmp              0            0
logging           0            0
rpf-fail          0            0
dstats            0            0
cpu heartbeat     0            0

Supervisor ASIC receive-queue parameters
----------------------------------------
 queue 0 maxrecevsize 0 pakhead 0 paktail 0
 queue 1 maxrecevsize 7E0 pakhead 4704894 paktail 41EED64
 queue 2 maxrecevsize 0 pakhead 0 paktail 0
 queue 3 maxrecevsize 7E0 pakhead 48E93EC paktail 49020F4
 queue 4 maxrecevsize 7E0 pakhead 47113F8 paktail 4709208
 queue 5 maxrecevsize 7E0 pakhead 0 paktail 0
 queue 6 maxrecevsize 7E0 pakhead 48C874C paktail 48E51B0
 queue 7 maxrecevsize 7E0 pakhead 485F0E4 paktail 48684E0
 queue 8 maxrecevsize 7E0 pakhead 48765EC paktail 48828D4
 queue 9 maxrecevsize 7E0 pakhead 47D455C paktail 47D455C
 queue A maxrecevsize 7E0 pakhead 47BF8F4 paktail 40DBB14
 queue B maxrecevsize 7E0 pakhead 4902D0C paktail 4906330
 queue C maxrecevsize 7E0 pakhead 47DBF4C paktail 47DF570
 queue D maxrecevsize 7E0 pakhead 47D0A58 paktail 47D407C
 queue E maxrecevsize 0 pakhead 0 paktail 0
 queue F maxrecevsize 7E0 pakhead 40D85BC paktail 40D8220
Supervisor ASIC exception status
--------------------------------
Receive overrun    00000000   Transmit overrun 00000000
FrameSignatureErr  00000000   MicInitialize    00000002
BadFrameErr        00000000   LenExceededErr   00000000
BadJumboSegments   00000000

[orthicon2009]

And this is from the command "sh ip traffic"

IP statistics:
  Rcvd:  3008199 total, 87381 local destination
         0 format errors, 0 checksum errors, 0 bad hop count
         0 unknown protocol, 2920818 not a gateway
         0 security failures, 0 bad options, 89598 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 89598 alert, 0 cipso, 0 ump
         0 other
  Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
         0 fragmented, 0 couldn't fragment
  Bcast: 0 received, 3 sent
  Mcast: 0 received, 0 sent
  Sent:  88143 generated, 0 forwarded
  Drop:  1 encapsulation failed, 0 unresolved, 0 no adjacency
         0 no route, 0 unicast RPF, 0 forced drop
         0 options denied, 0 source IP address zero

ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 1 redirects, 2 unreachable
        57527 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
  Sent: 0 redirects, 8 unreachable, 0 echo, 57527 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp
        0 info reply, 0 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

TCP statistics:
  Rcvd: 8702 total, 0 checksum errors, 22 no port
  Sent: 8162 total

PIMv2 statistics: Sent/Received
  Total: 0/0, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0

UDP statistics:
  Rcvd: 21152 total, 0 checksum errors, 37 no port
  Sent: 22449 total, 0 forwarded broadcasts

ARP statistics:
  Rcvd: 218218 requests, 353 replies, 0 reverse, 0 other
  Sent: 91 requests, 217389 replies (0 proxy), 0 reverse
  Drop due to input queue full: 0

[orthicon2009]

Hi Julio,

Thanks for the the info.  Please note that no links are flapping and we have around 20+ vlans created.

Below are some of the output of sh controllers cpu-int and sh ip traffic commands. Hopefully you guys can see the real problem that is causing this hign cpu utilization.

[orthicon2009]

it's working now. thanks.