High CPU utilization due to hpm main process in Cisco-3750E

[Athul Chandran]

Hi All,

I'm getting high CPU utilization in Cisco-3750E because of hpm main process. Please share your suggestions for decreasing the same.

SECXI-DC-SWT-DS#sh processes cpu sor
CPU utilization for five seconds: 63%/18%; one minute: 57%; five minutes: 58%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
124   104126484   163508169        636  9.95%  8.56%  8.49%   0 hpm main process
  12    11695460    11909185        982  7.06%  6.28%  6.44%   0 ARP Input
231    74992684    24451179       3067  6.26%  6.45%  6.47%   0 Spanning Tree
168    23363773    94354693        247  3.04%  2.24%  2.43%   0 Hulc LED Process
  10     7243208       63482     114098  2.08%  0.26%  0.18%   0 Licensing Auto U
220    23404940     9076656       2578  1.92%  1.92%  1.88%   0 ADJ resolve proc
213     4551461     8621761        527  1.76%  1.79%  1.69%   0 IP Input
375        7393        2147       3443  1.60%  0.78%  0.49%   1 SSH Process
128    35630805     3816377       9336  1.28%  1.04%  1.09%   0 hpm counter proc
199     1454235      831191       1749  1.28%  0.18%  0.13%   0 CDP Protocol
  85    21241925    13886806       1529  0.80%  0.76%  0.74%   0 RedEarth Tx Mana
  91     8292337      327240      25340  0.64%  0.79%  0.79%   0 Adjust Regions
278     1068054     8044622        132  0.32%  0.23%  0.23%   0 DHCPD Receive
169     3156326     2787493       1132  0.16%  0.23%  0.23%   0 HL3U bkgrd proce
264     1266902     5878321        215  0.16%  0.20%  0.24%   0 CEF: IPv4 proces
244      664383     3784626        175  0.16%  0.10%  0.11%   0 PI MATM Aging Pr
  84     2452628    18812107        130  0.16%  0.26%  0.26%   0 RedEarth I2C dri
338      198653    33043561          6  0.16%  0.01%  0.00%   0 OBFL VOLT obfl0
107     1356612   110468241         12  0.16%  0.12%  0.15%   0 HLFM address lea
181     1658884     1522165       1089  0.16%  0.06%  0.04%   0 HRPC qos request
180     7796779      761092      10244  0.16%  0.25%  0.25%   0 HQM Stack Proces
  57      249691     1097286        227  0.16%  0.04%  0.00%   0 HC Counter Timer
120      255378      253921       1005  0.16%  0.01%  0.00%   0 Strider Tcam Mem
  24       27235     3702774          7  0.00%  0.00%  0.00%   0 IPC Deferred Por
  23       30113     3702783          8  0.00%  0.00%  0.00%   0 IPC Periodic Tim
  26       19155      253923         75  0.00%  0.00%  0.00%   0 IPC Seat Man

Regards,

AC

[maxrodri]

Hello Athul,

With just this output it's not possible to confirm a high CPU issue on the switch. Can you take another "show processs cpu sorted | exclude 0.00" and a "show process cpu history"?

This can help to determine if a high CPU condition is present.

If you wanna check out what the "hpm main process" works for, maybe a TAC case will be needed (I tried to look for the process but I couldn't find anything about it)

You can also check this link to find out what queue it's receiving more packets on the CPU:

Monitoring Packet Counts for CPU Receive Queues

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1000983

[Athul Chandran]

Hello Max,

Pls find the outputs

SECXI-DC-SWT-DS#sh processes cpu sor | exc 0.00%
CPU utilization for five seconds: 52%/16%; one minute: 53%; five minutes: 55%
PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
124   112080284   166710608        672  8.95%  8.07%  8.03%   0 hpm main process
231    80885210    26717224       3027  6.23%  6.18%  6.35%   0 Spanning Tree
  12    13999986    14350612        975  4.63%  4.62%  4.99%   0 ARP Input
  10     7438424       64834     114730  2.07%  0.25%  0.17%   0 Licensing Auto U
168    26026174    96351912        270  1.59%  2.01%  2.28%   0 Hulc LED Process
220    26272102    10308226       2548  1.59%  1.82%  1.82%   0 ADJ resolve proc
199     1573820      872282       1804  1.27%  0.23%  0.14%   0 CDP Protocol
128    36690066     3897701       9413  1.11%  1.10%  1.10%   0 hpm counter proc
213     5194355     9987013        520  0.63%  1.24%  1.32%   0 IP Input
  85    21986245    14186714       1549  0.47%  0.71%  0.72%   0 RedEarth Tx Mana
  91     9341026      359274      25999  0.47%  0.66%  0.68%   0 Adjust Regions
278     1184233     8280372        143  0.31%  0.15%  0.16%   0 DHCPD Receive
  84     2632451    19213259        137  0.31%  0.32%  0.25%   0 RedEarth I2C dri
180     8029088      777251      10330  0.31%  0.22%  0.22%   0 HQM Stack Proces
244      742619     3864818        192  0.15%  0.09%  0.07%   0 PI MATM Aging Pr

SECXI-DC-SWT-DS#sh processes cpu hi

SECXI-DC-SWT-DS#sh processes cpu history

      666665555555555666665555555555666664444455555444445555555555

      111113333366666222224444466666000009999911111555554444400000

  100

   90

   80

   70

   60 *****     **********     **********

   50 **********************************************************

   40 **********************************************************

   30 **********************************************************

   20 **********************************************************

   10 **********************************************************

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per second (last 60 seconds)

      666666668777777777778877777777767766777787777777777777777787

      859413346491236299750078883679294669622394535413704027040105

  100

   90         *                               *

   80         * *   * ********** ***   *  *   * * *   *    *

   70 ***     **************************************************

   60 ******#*##################*#######*#####################*#

   50 ##########################################################

   40 ##########################################################

   30 ##########################################################

   20 ##########################################################

   10 ##########################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

      899989897799999999998899999999999888998877989898989999899888669888999999

      097329697903496770804779526877887030936118268977895898293730962428690888

  100  **  * *     **** *   *** *******   *       * * * **** *          ** *

   90  *** ***  ********** ************   ***   ************ ***    *  *****

   80 **************************************** *******************  ********

   70 **************************#********************##*********************

   60 ######**************##########**************##########**************##

   50 #########*#######################*####################################

   40 ######################################################################

   30 ######################################################################

   20 ######################################################################

   10 ######################################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

               0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

Regards,

AC