topic Re: Cisco 3750 High CPU Load in Switching

Cisco 3750 High CPU Load

andreribeiro — Fri, 08 Mar 2019 22:44:10 GMT

Hello,

I'm facing a big issue with my stack of two members os 3750E.

The CPU is so high but there is any process consuming the CPU as you can see bellow.

#sh processes cpu sorted

CPU utilization for five seconds: 96%/84%; one minute: 96%; five minutes: 94%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

90 905211 169354 5345 1.80% 1.54% 1.51% 0 RedEarth Tx Mana

179 1348764 988489 1364 1.47% 1.19% 1.16% 0 Hulc LED Process

134 440167 43981 10008 0.98% 0.90% 0.87% 0 hpm counter proc

13 274848 360295 762 0.82% 0.77% 0.73% 0 ARP Input

229 163220 232279 702 0.32% 0.30% 0.27% 0 IP Input

390 3674 1492 2462 0.32% 0.03% 0.02% 5 SSH Process

247 124654 227296 548 0.32% 0.28% 0.28% 0 Spanning Tree 

89 396288 233160 1699 0.32% 0.44% 0.51% 0 RedEarth I2C dri

I checked the CPU queu frames and in 5 minutes the difference doesn't seems unusual (image attached).

On logs nothing special.

Can someone give me an hint here?

Thank you

Model: WS-C3750X-24
Firmware Versoin: 15.0(2)SE11

Re: Cisco 3750 High CPU Load

andreribeiro — Fri, 20 Apr 2018 13:13:10 GMT

The missing image

Re: Cisco 3750 High CPU Load

Diana Karolina Rojas — Fri, 20 Apr 2018 17:52:41 GMT

Hello!

Are you sure you don't have any Debug running in your device? As a first step I recommend you to use "undebug all", aditionally can you provide me the output of the next commands in a txt file?:

show proc cpu detailed process fed | ex 0.0
show proc cpu sorted | ex 0.0
show log
show ver

-----Do not forget to rate/mark useful answers.----

Regards,

Re: Cisco 3750 High CPU Load

Yeshwanth Simhadri — Fri, 20 Apr 2018 18:14:37 GMT

There are 2 versions of IOS that Cisco recommends to all users

Early Deployment (ED)

Software releases that provide new features and new platform support in addition to bug fixes.

Maintenance Deployment (MD)

A Cisco software release that provides bug fix support and ongoing software maintenance.

Just download the newest version of IOS for the 3750switch & you should be good to go...

Copy the new IOS file from USB or TFTP to flash...

1 - software clean

2 - software install file flash:new IOS image

3 - reload

Re: Cisco 3750 High CPU Load

Leo Laohoo — Sat, 21 Apr 2018 01:12:16 GMT

Post the complete output to the following commands:
1. sh version; and
2. sh proc cpu sort | ex 0.00

Re: Cisco 3750 High CPU Load

andreribeiro — Tue, 24 Apr 2018 12:36:54 GMT

Hello Diana,
Thank you for replying.

The requested commands are on the file attached.

Thank you again for you availability.

Re: Cisco 3750 High CPU Load

andreribeiro — Tue, 24 Apr 2018 12:40:25 GMT

Thank you Leo,

The requested command output is attached.
Sh version is on a reply to Diana above.

As you can see processor now is ok, but during the morning it was like hisotry file attached.

Thank you for your help.

Re: Cisco 3750 High CPU Load

andreribeiro — Tue, 24 Apr 2018 12:41:27 GMT

Additional information.

As you can see the history of CPU load is so high.

CPU history is attached.

Re: Cisco 3750 High CPU Load

Diana Karolina Rojas — Tue, 24 Apr 2018 13:47:00 GMT

Hello Andre thanks for the reply,

I also need this outputs:

show process memory

show platform tcam utilization

Best Regards,

Re: Cisco 3750 High CPU Load

andreribeiro — Tue, 24 Apr 2018 13:58:06 GMT

Hi Diana,

Thank you.
The requested commands are attached.

Re: Cisco 3750 High CPU Load

Diana Karolina Rojas — Tue, 24 Apr 2018 13:58:50 GMT

On the other hand,

Why do you have flapping in those interfaces? (2/0/16 and 2/0/24)? Is this flapp constant in your network?

Regards,

Re: Cisco 3750 High CPU Load

andreribeiro — Tue, 24 Apr 2018 14:20:46 GMT

I really don't know.
On these 2 ports there are servers (in access mode) connected and are managed by other team.

Interface statistics attached.

Re: Cisco 3750 High CPU Load

Diana Karolina Rojas — Tue, 24 Apr 2018 14:29:51 GMT

It's strange because the interfaces not seen to have errors, what about the OSPF neighbor (10.42.201.250)? I could not find anything strange besides the logs among all the exits that you sent me.

Re: Cisco 3750 High CPU Load

ITexpert — Tue, 24 Apr 2018 14:30:37 GMT

Did you try restarting switch ?

Re: Cisco 3750 High CPU Load

Diana Karolina Rojas — Tue, 24 Apr 2018 14:38:31 GMT

please provide me the next output: show controllers cpu-interface

Regards,

Re: Cisco 3750 High CPU Load

andreribeiro — Tue, 24 Apr 2018 14:40:50 GMT

The other neighbor is Firewall PaloAlto and we have around 40 implementations like that without any issue.

Re: Cisco 3750 High CPU Load

andreribeiro — Tue, 24 Apr 2018 14:42:42 GMT

Here is the requested output.

An important finding that we got recently is, we have a aggregator switch connected to this (another 3750) that when this switch has a high value of CPU load the other, the aggregator also have the similar CPU load values).

I mean seems that is something that affect both switches and the aggregator switch doesn't "speak" OSPF.

Re: Cisco 3750 High CPU Load

Leo Laohoo — Tue, 24 Apr 2018 23:51:00 GMT

I just checked the output I've requested and it looks like OSPF flapping could be causing the issue of high CPU.

Re: Cisco 3750 High CPU Load

Diana Karolina Rojas — Wed, 25 Apr 2018 12:40:36 GMT

Good morning Andre,

II know the kind of implementation isn't the problem, but in the logs you provide me there are a lot of logs that indicate flapping with this neighbor and that is not a healty behavior in your network, maybe the constant change in the neighbord ststus is giving you problems. You should review why this is happening.

---Do not forget to rate useful post.---

Regards,

Re: Cisco 3750 High CPU Load

Yeshwanth Simhadri — Thu, 03 May 2018 09:34:51 GMT

did you try to upgrade the IOS - as advised below?