Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
6799
Views
0
Helpful
8
Replies

EEM Script for high CPU utilization problem

lap
Level 2
Level 2

‎01-31-2011 05:48 AM

Hi,

I have configured the following script (received from TAC) to catch the process that was causing high CPU utilization on some C4510R:

event manager applet cpu_stats
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op ge entry-val "70" exit-op lt exit-val "70" poll-interval 2
action 1.01 syslog msg "------HIGH CPU DETECTED----, CPU: $_snmp_oid_val %"
action 1.02 cli command "enable"
action 1.03 cli command "show clock | append bootflash:/cpu_stats"
action 1.04 cli command "show proc cpu sort | append bootflash:/cpu_stats"
action 1.05 cli command "Show proc cpu | exc 0.00% | append bootflash:/cpu_stats"
action 1.06 cli command "Show proc cpu history | append bootflash:/cpu_stats"
action 1.07 cli command "debug platform packet all receive buffer| append bootflash:/cpu_stats"
action 1.08 cli command "show platform cpu packet buffered | append bootflash:/cpu_stats"
action 1.09 cli command "show platform cpu packet buffered | in  timestamp|destmac | append bootflash:/cpu_stats"
action 1.10 cli command "show ibc | append bootflash:/cpu_stats"
action 1.11 cli command "show proc cpu sort | append bootflash:/cpu_stats"
action 1.12 cli command "Show proc cpu | exc 0.00% | append bootflash:/cpu_stats"
action 1.13 cli command "Show proc cpu history | append bootflash:/cpu_stats"
action 1.14 cli command "debug platform packet all receive buffer | append bootflash:/cpu_stats"
action 1.15 cli command " show platform cpu packet buffered | append bootflash:/cpu_stats"
action 1.16 cli command " show platform cpu packet buffered | in  timestamp|destmac | append bootflash:/cpu_stats"
action 1.17 cli command "sho log | append bootflash:/cpu_stats"
action 1.18 cli command "undebug all"
action 1.19 cli command "show platform hardware capacity ibc | append bootflash:/cpu_stats"
action 1.20 cli command "show tcp brief | append bootflash:/cpu_stats"
!
end

I can see in my log that the script is being excecuted:

Jan 31 14:19:16.247: %HA_EM-6-LOG: cpu_stats: ------HIGH CPU DETECTED----, CPU: 77 %

But I have no cpu_stats in the bootflash:

#sh bootflash:
-#- --length-- -----date/time------ path
1     24462212 Dec 15 2009 01:26:54 cat4500e-ipbasek9-mz.122-53.SG1.bin
2     24936345 Jan 14 2011 10:34:46 cat4500e-ipbasek9-mz.122-54.SG.bin

72318976 bytes available (59117568 bytes used)

Has anyone got an idea why?

Thanks in advance.

Best regards,

Laurent

Labels:
0 Helpful
8 Replies 8

xorrkaz_1306
Level 1
Level 1

‎01-31-2011 11:23 AM

This script will not work as written.  The commands that use double pipes ('|') will not work as you expect.  However, the other commands should work.  If the file is not being created, and you have AAA command authorization enabled, configure the following command:

event manager session cli username USER

Where USER is a AAA user authorized to run all of the CLI commands in your EEM policies.

0 Helpful

lap
Level 2
Level 2
In response to xorrkaz_1306

‎02-01-2011 12:45 AM

Hi Joe,

Thanks for your post.

Alright, I have added the command you told me. I wil let you know if it helps.

Regards,

Laurent

0 Helpful

lap
Level 2
Level 2
In response to lap

‎02-01-2011 01:39 AM

Hi Joe,

Well actually I did some debugging:

debug event manager action cli

And look what I found out:

%HA_EM-6-LOG: cpu_stats: ------HIGH CPU DETECTED----, CPU: 71 %
F%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH>
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH>enable
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH#
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH#show clock | append bootflash:cpu_stats
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : % Appending is not supported in this file system
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : % Incomplete command before pipe.
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH#
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH#show proc cpu sort | append bootflash:cpu_stats
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : % Appending is not supported in this file system
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : % Incomplete command before pipe.
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH#
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH#Show proc cpu | exc 0.00% | append bootflash:cpu_stats
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : CPU utilization for five seconds: 71%/0%; one minute: 37%; five minutes: 34%
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :    7      706152     88897       7943  0.15%  0.08%  0.09%   0 Check heaps     
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :   31    12204888  10461205       1166  3.35%  2.29%  2.15%   0 ARP Snoop       
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :   38     1762708    632553       2786  0.31%  0.31%  0.31%   0 IDB Work        
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :   53    58795496 130924060        449 13.81% 14.60% 15.22%   0 Cat4k Mgmt HiPri
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :   54    92957236  67192183       1383 52.07% 18.10% 15.63%   0 Cat4k Mgmt LoPri
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :   84      545216   1345383        405  0.95%  0.16%  0.09%   0 Port-Security   
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :   88      180824    716579        252  0.15%  0.13%  0.13%   0 UDLD            
%HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT :  174      153240  17431110          8  0.07%

It looks like I cannot use append command?

Regards,

Laurent

0 Helpful

Joe Clarke
Cisco Employee
Cisco Employee
In response to lap

‎02-01-2011 03:56 AM

Correct,.  Your file system doesn;t support append.  If you have a flash disk in this device, you can change the bootflash: path to that disk, and the append will work.

0 Helpful

lap
Level 2
Level 2
In response to Joe Clarke

‎02-01-2011 12:21 PM

Hi Joseph,

Thanks a lot for your post.

I don't have a flash disk in this device. I have tried to use the redirect command instead. It looks like it´s working as I get the cpu_stats file in the bootflash but the content of the file looks like that only:

SWITCH#sh bootflash:

-#- --length-- -----date/time------ path

1           32 Feb 01 2011 15:40:52 cpu_stats

2     24462212 Dec 15 2009 01:26:54 cat4500e-ipbasek9-mz.122-53.SG1.bin

3     24936345 Jan 14 2011 10:34:46 cat4500e-ipbasek9-mz.122-54.SG.bin

72310784 bytes available (59125760 bytes used)

SWITCH#more bootflash:cpu_stats

16:40:53.575 CET Tue Feb 1 2011

So there is only info of the first cli line of the script. I did a debug:

Feb  1 18:44:56.402: %HA_EM-6-LOG: cpu_stats: ------HIGH CPU DETECTED----, CPU: 72 %
Feb  1 18:44:56.402: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : CTL : cli_open called.
Feb  1 18:44:56.402: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH>
Feb  1 18:44:56.402: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH>enable
Feb  1 18:44:56.414: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH#
Feb  1 18:44:56.414: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH#show clock | redirect bootflash:cpu_stats
Feb  1 18:44:58.403: %HA_EM-6-LOG: cpu_stats: ------HIGH CPU DETECTED----, CPU: 72 %
Feb  1 18:44:58.403: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : CTL : cli_open called.
Feb  1 18:44:58.407: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH>
Feb  1 18:44:58.407: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH>enable
Feb  1 18:44:58.419: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH#
Feb  1 18:44:58.419: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH#show clock | redirect bootflash:cpu_stats
Feb  1 18:45:00.403: %HA_EM-6-LOG: cpu_stats: ------HIGH CPU DETECTED----, CPU: 72 %
Feb  1 18:45:00.403: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : CTL : cli_open called.
Feb  1 18:45:00.403: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH>
Feb  1 18:45:00.403: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH>enable
Feb  1 18:45:00.415: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : OUT : SWITCH#
Feb  1 18:45:00.415: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : IN  : SWITCH#show clock | redirect bootflash:cpu_stats
Feb  1 18:45:16.436: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : CTL : cli_close called.
Feb  1 18:45:17: %IP_SOURCE_GUARD-4-DENY_INVALID_PACKET: Detected and dropped illegal traffic on port Fa2/28 and vlan 111, the non-cumulative packet dropped count is 2.
Feb  1 18:45:18.413: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : CTL : cli_close called.
Feb  1 18:45:20.425: %HA_EM-6-LOG: cpu_stats : DEBUG(cli_lib) : : CTL : cli_close called.

Is that caused by the redirect command you think? The next lines in the script should be excecuted no?

Regards,

Laurent

0 Helpful

Joe Clarke
Cisco Employee
Cisco Employee
In response to lap

‎02-02-2011 08:46 AM

Yes, the next line should run (though the output file will be overwritten with the output of the most recently executed command).  Post your current applet.  Note: the reason the applet may not be running to completion is that you need to increase the maxrun timer.  You might try adding this to the event registration line of the applet:

maxrun 300

0 Helpful

lap
Level 2
Level 2
In response to Joe Clarke

‎02-02-2011 10:24 AM

Hi Joseph,

Here is my current applet:

event manager applet cpu_stats
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt entry-val "70" poll-interval 2
action 1.01 syslog msg "------HIGH CPU DETECTED----, CPU: $_snmp_oid_val %"
action 1.02 cli command "enable"
action 1.04 cli command "show proc cpu sort | redirect bootflash:cpu_stats"
action 1.05 cli command "Show proc cpu | exc 0.00% | redirect bootflash:cpu_stats"
action 1.06 cli command "Show proc cpu history | redirect bootflash:cpu_stats"
action 1.07 cli command "debug netdr cap rx | redirect bootflash:cpu_stats"
action 1.08 cli command "show netdr cap | redirect bootflash:cpu_stats"
action 1.09 cli command "show netdr cap | in  timestamp|destmac | redirect bootflash:cpu_stats"
action 1.11 cli command "show proc cpu sort | redirect bootflash:cpu_stats"
action 1.12 cli command "Show proc cpu | exc 0.00% | redirect bootflash:cpu_stats"
action 1.13 cli command "Show proc cpu history | redirect bootflash:cpu_stats"
action 1.14 cli command "debug netdr cap rx | redirect bootflash:cpu_stats"
action 1.15 cli command "show netdr cap | redirect bootflash:cpu_stats"
action 1.16 cli command "show netdr cap | in  timestamp|destmac | redirect bootflash:cpu_stats"
action 1.17 cli command "sho log | redirect bootflash:cpu_stats"
action 1.18 cli command "undebug all"
!

Regards,

Laurent

0 Helpful

Joe Clarke
Cisco Employee
Cisco Employee
In response to lap

‎02-02-2011 11:11 AM

This applet does not line up with the latest debug you have shown.  However, change the event registration line to:

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt entry-val "70" poll-interval 2 maxrun 300

Then see what happens.

0 Helpful
Customers Also Viewed These Support Documents