The commands dont work on my - Page 2

jarar_m01 · ‎01-23-2017

Hi guys... please help me fix this issue. I get High CPU utilization on my

C3750E Software (C3750E-IPBASEK9-M), Version 15.0(2)SE10a, RELEASE SOFTWARE (fc3)

Can any one help me with the solution. I tried to check on support forums but its still not resolved. I had the same issue on 3550 so Replaced it with 3750 but still the same.

Any help will be appreciated.

jarar_m01 · ‎01-26-2017

The commands dont work on my device....

action 0.8 cli command "show platform health | append flash:HIGH_CPU.txt "
action 0.9 cli command "show platform cpu packet statistics all | append flash:HIGH_CPU.txt "
action 1.0 cli command "show platform cpu packet buffered | append flash:HIGH_CPU.txt "

I have collected information manually as my CPU now is always on 99%

Please find the attached file below. May be you find out smthing that will help me fix this issue. I would really appreciate it.

Mark Malone · ‎01-26-2017

Hi

None of your interfaces are under pressure when that was taken even though cpu is being hammered , buffers are free as well
You did hit a bug in the last crashfile which still is in your ios version so i would definitly move off that version, cpu hog bug

The analyzer did pick up on the cpu and provided a specific doc for 3750 tshoot with cpu issues but nothing else really
are you using eigrp or ospf as stub shouldnt be set on a core router usually its for an edge device with nothing behind it
another thing in your config you have a lot of policy maps in place redirecting traffic these can cause cpu issues

analyzer doc
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

Dec 22 00:10:41.090: %SYS-3-CPUHOG: Task is running for (4195)msecs, more than (2000)msecs (7/6),process = SNMP ENGINE.
-Traceback= 7F9C44 7F9C38 78B050 19D4444 19A82F0 19989E8 19C37B0 CD7088 CCDB58
DXB_CSW_HO#show process cpu history | e 0.00

Found this in another doc
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/68461-high-cpu-utilization-cat3750.html
High CPU Due to Policy Based Routing

Policy Based Routing (PBR) implementation in Cisco Catalyst 3750 switches has some limitations. If these restrictions are not followed, it can cause high CPU utilization.

•You can enable PBR on a routed port or an SVI.

•The switch does not support route-map deny statements for PBR.

•Multicast traffic is not policy-routed. PBR applies only to unicast traffic.

•Do not match ACLs that permit packets destined for a local address. PBR forwards these packets, which can cause ping or Telnet failure or route protocol flapping.

•Do not match ACLs with deny ACEs. Packets that match a deny ACE are sent to the CPU, which can cause high CPU utilization.

•In order to use PBR, you must first enable the routing template with the sdm prefer routing global configuration command. PBR is not supported with the VLAN or default template.

For a complete list, refer to the PBR Configuration Guidelines.

      999999999999999999999999999999999999999999999999999999999999
      888899999888888888899999888889999988888888888888899999888888
100 **********************************************************
   90 **********************************************************
   80 **********************************************************
   70 **********************************************************
   60 **********************************************************
   50 **********************************************************
   40 **********************************************************
   30 **********************************************************
   20 **********************************************************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)

      999999999999999999999999999999999999999999999999999999999999
      999999999999999999999999999998999999999999999999999999999999
100 ##########################################################
   90 ##########################################################
   80 ##########################################################
   70 ##########################################################
   60 ##########################################################
   50 ##########################################################
   40 ##########################################################
   30 ##########################################################
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

      1                            11                 11                     1
      099999996967869767799999999990088459869756589999009999969457549754376990
      099999867782779495198999999990032992229273159999009999257034149476960990
100 ******** *    *    ************       *     ******#*** *     *      *
   90 ###*#*** * * *    ****##******    * *    ***#*#*##*** *     *      *
   80 #####*** * * * * **#######***** ** *    ***########* *     *    * *
   70 ######***************#########*** ** **   ***########*** * **   * *
   60 ######***************#########*** ******** ***########*** * *** ***
   50 ######***************#########***************#########*** *** **** ***
   40 ######***************##########**************#########***************#
   30 #######**************##########*************###########**************#
   20 ########***********############*************###########*#************#
   10 ######################################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

1st thing I would do as cpuhog is in that version is try another one

jarar_m01 · ‎01-26-2017

Allright Mark, You have convinced me to change the version... Lets remove this CPU HOG from my switch... can you suggest the most stable version in IP_BASE for this switch ?

I will also delete few Policy Maps as I believe they are not in USE .. I am new here so would study before doing any thing... The previous engineer has really messed up things here....

Mark Malone · ‎01-26-2017

So this is the alert seen described , if its nit a software issue its most likely the switch under too much pressure causing it as below , your on the suggested ios versions you could roll back to c3750e-ipbasek9-mz.150-2.SE10.bin and see if its stil the same , you can have multiple ios versions in your flash once theres room so you can easily revert back to 10a if it does nothing

the 3750 cpu tshoot doc above there's some good things to check in that as well

....................................

To reduce the impact of runaway processes, Cisco IOS® software uses a process watchdog timer that allows the scheduler to periodically poll the currently active process. This feature is not the same as preemption. Instead, it is a fail-safe mechanism, which ensures that the system does not become unresponsive or completely lock up due to the total consumption of the CPU by any process.

If a process appears to hang (for example, if it continues to run for a long time), the scheduler can force the process to terminate.

Every time the scheduler allows a process to run on the CPU, it starts a watchdog timer for that process. After a preset period, if the process continues to run, the watchdog process generates an interrupt and causes a router restart by a "software forced crash" (the stack trace shows a watchdog process as the trigger of the crash).

The first time the watchdog expires, the scheduler prints a warning message such as:

%SYS-3-CPUHOG: Task ran for 2148 msec (20/13), Process = IP Input, PC = 3199482    
-Traceback= 314B5E6 319948A

This message indicates a process has held up the CPU. Here, it is the "IP Input" process. This message usually appears during transient circumstances, such as an Online Insertion and Removal (OIR) when the router boots up, or under heavy traffic conditions. The "%SYS-3-CPUHOG" messages must not appear during normal operation of the router.

If the router is busy at interrupt level after a process was scheduled to run, the accounting of the duration for which the process ran can be inaccurate. This is because, the CPUHOG only tracks process level tasks. It does not track interrupt level tasks that are permitted to interrupt and gain control of the CPU.

The typical process to run at interrupt level is packet switching.

High CPU utilization due to IP Input