Cisco 3750 High CPU Load

[andreribeiro]

Hello,

I'm facing a big issue with my stack of two members os 3750E.

The CPU is so high but there is any process consuming the CPU as you can see bellow.

#sh processes cpu sorted

CPU utilization for five seconds: 96%/84%; one minute: 96%; five minutes: 94%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

90 905211 169354 5345 1.80% 1.54% 1.51% 0 RedEarth Tx Mana

179 1348764 988489 1364 1.47% 1.19% 1.16% 0 Hulc LED Process

134 440167 43981 10008 0.98% 0.90% 0.87% 0 hpm counter proc

13 274848 360295 762 0.82% 0.77% 0.73% 0 ARP Input

229 163220 232279 702 0.32% 0.30% 0.27% 0 IP Input

390 3674 1492 2462 0.32% 0.03% 0.02% 5 SSH Process

247 124654 227296 548 0.32% 0.28% 0.28% 0 Spanning Tree

89 396288 233160 1699 0.32% 0.44% 0.51% 0 RedEarth I2C dri

I checked the CPU queu frames and in 5 minutes the difference doesn't seems unusual (image attached).

On logs nothing special.

Can someone give me an hint here?

Thank you

Model: WS-C3750X-24
Firmware Versoin: 15.0(2)SE11

[andreribeiro]

The missing image

[Diana Karolina Rojas]

Hello!

Are you sure you don't have any Debug running in your device? As a first step I recommend you to use "undebug all", aditionally can you provide me the output of the next commands in a txt file?:

show proc cpu detailed process fed | ex 0.0
show proc cpu sorted | ex 0.0
show log
show ver

-----Do not forget to rate/mark useful answers.----

Regards,

[andreribeiro]

Hello Diana,
Thank you for replying.

The requested commands are on the file attached.

Thank you again for you availability.

[Diana Karolina Rojas]

Hello Andre thanks for the reply,

I also need this outputs:

show process memory

show platform tcam utilization

Best Regards,

[andreribeiro]

Hi Diana,

Thank you.
The requested commands are attached.

[Diana Karolina Rojas]

On the other hand, 

Why do you have flapping in those interfaces? (2/0/16 and 2/0/24)? Is this flapp constant in your network?

Regards,

[andreribeiro]

I really don't know.
On these 2 ports there are servers (in access mode) connected and are managed by other team.

Interface statistics attached.

[Diana Karolina Rojas]

It's strange because the interfaces not seen to have errors, what about the OSPF neighbor (10.42.201.250)? I could not find anything strange besides the logs among all the exits that you sent me. 

[Diana Karolina Rojas]

please provide me the next output:  show controllers cpu-interface

Regards,

[andreribeiro]

Here is the requested output.

An important finding that we got recently is, we have a aggregator switch connected to this (another 3750) that when this switch has a high value of CPU load the other, the aggregator also have the similar CPU load values).

I mean seems that is something that affect both switches and the aggregator switch doesn't "speak" OSPF.

[andreribeiro]

The other neighbor is Firewall PaloAlto and we have around 40 implementations like that without any issue.

[Diana Karolina Rojas]

Good morning Andre,

II know the kind of implementation isn't the problem, but in the logs you provide me there are a lot of logs that indicate flapping with this neighbor and that is not a healty behavior in your network, maybe the constant change in the neighbord ststus is giving you problems. You should review why this is happening.

---Do not forget to rate useful post.---

Regards,

[Yeshwanth Simhadri]

There are 2 versions of IOS that Cisco recommends to all users

Early Deployment (ED)

Software releases that provide new features and new platform support in addition to bug fixes.

　

Maintenance Deployment (MD)

A Cisco software release that provides bug fix support and ongoing software maintenance.

Just download the newest version of IOS for the 3750switch & you should be good to go...

Copy the new IOS file from USB or TFTP to flash...

1 - software clean

2 - software install file flash:new IOS image

3 - reload

[Leo Laohoo]

Post the complete output to the following commands:
1. sh version; and
2. sh proc cpu sort | ex 0.00