High CPU Usage on 3750

[SIMMN]

Hi,

I have this 3750 old switch running c3750-ipbase-mz.122-35.SE5.bin (I know it is old...) and it is used for connecting remote offices via 10Mbps circuits. It has been running fine for the last 12-month. But recently, I found the CPU usage is a little bit high, like in screenshot.

I know the average CPU usage is only about 10-20% but it bothers me that it constant has MAX CPU usage around 90+%...I also did show processes cpu sorted and the only thing having 3+% CPU usage is spanning tree...but that won't be enough to cause the CPU peaks... There is no special feature configure on the switch, just VLAN taggings.

Any suggestions on what to check?

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Your stats don't show a constant max CPU of 90+%, they show spikes of 90+%, happening (it appears) once every 30 minutes.

[SIMMN]

"Constant" might sound confusing but every 30-min bothers me really bad as well...

[Sanjay Shaw]

Hi,

Reboot it once and check the status.

[SIMMN]

It is hard to schedule an reboot of that switch but it will be used as the last resort.

[Sanjay Shaw]

Hi,

Please find the EEM script to configure it in the switch. After that please share the output.

event manager session cli username "Privlige 15 leve user"
event manager applet cpuspike
 event snmp oid 1.3.6.1.4.1.9.2.1.56 get-type next entry-op ge entry-val "60" exit-time 20 poll-interval 1
 action 1.0 syslog msg "chinese_ys: high CPU - check nvram:cpuinfo file."
 action 2.0 cli command "enable"
 action 3.0 cli command "show clock | append nvram:cpuinfo"
 action 4.0 cli command "show proc cpu sort | append nvram:cpuinfo"
 action 5.0 cli command "show log | append nvram:cpuinfo"
 action 6.0 cli command "u all"
 action 7.0 cli command "term length 24"

BR// Sanjay

[SIMMN]

Thanks, I will try those later.

But once it is configure, should I also syslog server or just show log will give me the capture?

[Sanjay Shaw]

Hi,

Please configure ur snmp polling server to poll in an interval of 4 hr or 6 hr.

And  remove all the unnecessary snmp trap. 

To troubleshoot step by step:

1) Remove the snmp config from the switch, now monitor.

2)Hope ur cpu is behaving normal, now configure the snmp and enable the required snmp trap, don't enable all and check the cpu .status.

Thanks...

Please rate if the EEM & tshoot has provided solution.

[SIMMN]

I will do this and report back.

[SIMMN]

I pretty much disabled all the health monitors of switch from SNMP server and output of sh processes cpu history is indeed reduced to MAX @ 80-85% from 95%. But still more than my expected 50-60%.

[Sanjay Shaw]

Please re-post your the below output.

show tech

show proc cpu sorted

[Leo Laohoo]

Your IOS is pretty old.  How about you upgrade your IOS to 12.2(55)SE9 and observe for any improvements?

[SIMMN]

it is running 12.2(55)SE9

[SIMMN]

Here is the output file attached. It seems like SNMP Engine is the root cause.

BTW, Why use nvram not flash for storing the file?

[Ahmed Muhi]

Hello chinese_ys,

Do you have logging enabled ? can you share "show logging" output, maybe it will give us some indication about what the problem is.