Cisco 3850x Very High CPU - Stack-mgr Process

[TOM FRANCHINA]

We are getting high cpu on at least 2 of our 3850x Stacks. Spoke to TAC he noted:

CSCuo14511    fed and stack-mgr causing High CPU on 3850

Suggesting to upgrade to 03.03.04SE

Has anyone seen this and solved it without an upgrade.

Thanks,

Tom

ST3-Stack1-3850#sho proc cpu sort | e 0.00
Core 0: CPU utilization for five seconds: 94%; one minute: 92%;  five minutes: 90%
Core 1: CPU utilization for five seconds: 94%; one minute: 93%;  five minutes: 94%
Core 2: CPU utilization for five seconds: 98%; one minute: 96%;  five minutes: 93%
Core 3: CPU utilization for five seconds: 80%; one minute: 83%;  five minutes: 89%
PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process
5719   3399767     25647721 923    49.61    49.34    49.07    0     stack-mgr
5717   2287217     89195731 434    27.47    27.21    26.97    0     fed
10243  733637      16770188 371    13.71    13.54    13.70    34816 iosd
6250   40832       82186167 665    0.78     0.60     0.59     0     pdsd
10239  1299808     36806083 15     0.24     0.12     0.11     0     wcm
6261   692231      40238409 871    0.10     0.06     0.05     0     cpumemd
19     974400      53966893 33     0.05     0.03     0.04     0     sirq-net-rx/1
43     2452848     54120184 28     0.05     0.04     0.05     0     sirq-net-rx/3
5718   115760      32235590 20     0.05     0.09     0.10     0     platform_mgr
10240  1899130     59100638 3      0.05     0.04     0.02     0     table_mgr

ST3-Stack1-3850#sho ver
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.01SE RELEASE SOFTWARE (fc1)

Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     2 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     3 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     4 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     5 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     6 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     7 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     8 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     9 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL

[InayathUlla Sharieff]

I dont see any other way to bring the cpu to lower other than the upgrade.

HTH

[bruno.fernandes]

Hi guys,

I'm hitting the same problem after upgrading to 03.03.04SE :(

​Apparently this only affects groups of stacked switches …. standalones are not affected

Any news on this issue ?

Regards,

Bruno

[Ton V Engelen]

So v3.3.4 also has this?

It seems like something that comes back.

Here s what we saw:

v.3.3.0

- CPU is about 31%

- Reboot the stack

- CPU is about 9%

- After 6 weeks or so it goes up to 31%

- Upgrade to v3.3.3 and reload

- CPU is about 9%

- After 6 weeks or so it goes up to 31% again.

It doesn't seem to affect performance though, so i dont bother to look at it untill we try a new release.

[TOM FRANCHINA]

All,

We found that the Cisco 6509 is configured with:

spanning-tree mode pvst

The Cisco 3850x is configured with:

spanning-tree mode rapid-pvst

We are going to do some test lab changes to see if we can change this without downtime. Our CE suggests changing to  Multiple spanning tree mode (mst)

Any thought out there,

Tom

[Kanes Ramasamy]

Hi Tom,

I am having the same issue and I can see that its being caused by the process below, was just wondering if you have managed to settle the issue that you were facing.

Thanks and Regards,

Kanes.R

5705 L 2 6162 4205548 117931076 0 24.7 24.6 24.6 0 Replenish OOB
5705 L 0 6163 1588138 120072772 0 24.7 24.4 24.3 0 OOBnd RX

[Aaron Leong]

Hi Guys,

We are hit this bug on V03.07.00E .

Symptoms are similar, CPU usage in stack-mgr  .

Any lead thus far for this bug?

[tfetzer]

I have been working with TAC for almost a year on this issue.  We have (11) 3850's; 2 are stacked and 9 standalone.  Current code is 03.07.01.  I have 3 of the 9 standalone switches still reporting high cpu and I am receiving alerts from Cisco Prime.  I have sent EMM scripts and captures to TAC.  No solution as of this time.  Also, we are not sure why only these 3 switches are still having the issue and not the other 4.  Please update if anyone has any solutions.  Thanks.

[TOM FRANCHINA]

We just install 2 new stacks of 3850x's... the CPU utilization is good

JH3-3850  uptime is 2 weeks, 3 days, 16 minutes
CPU utilization for five seconds: 11%; one minute: 12%;  five minutes: 12%
Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     2 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     3 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     4 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
================================================================================
PhelpsB-3850 uptime is 3 weeks, 5 days, 6 minutes
CPU utilization for five seconds: 14%; one minute: 13%;  five minutes: 12%
Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     2 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     3 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     4 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     5 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL

Here is the switch stack that has been up for a year and a half:

Dec3-3850 uptime is 1 year, 24 weeks, 5 days, 3 hours, 31 minutes
CPU utilization for five seconds: 94%; one minute: 94%;  five minutes: 94%
Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     2 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL
     3 56    WS-C3850-48P       03.03.01SE        cat3k_caa-universalk9 INSTALL

NO solution !!!!!!!!!!!!!!!!!!

[John Vincent]

We have only started to see this recently on one of our (15) 3850 stacks...  Almost done upgrading them all to 03.06.03 for this,and other bugs..  No mention of a fix in this release, but it is the current gold star version from Cisco.

The stack that we are seeing this high CPU from stack-mgr gets upgraded early monday morning...  Hopefully this goes away....

[Brett Erickson]

I can confirm this from my experience. I have a bunch of 3850 stacks running on version 03.03.02 and 03.03.03 and they all have issues with their CPU spiking. However in another location my 3850s are running on 03.06.03E and they don't exhibit this CPU spiking.

I will be upgrading all of my switches to 03.06.03E in my next maintenance window.

[sunbocsg1]

Hi, Brett, 

are your switches running fine now?

Thanks!

[kingston01]

Referring back to this bug as i am noticing the slow creep up of CPU after 1 year of up time,   one stack has been upgraded to 03.03.05SE  

Anyone seeing gradual memory increase over IOS process stack across stacked and non stacked  3650/3850   (03.03.05SE  / 03.03.03SE) ?

If Frequent mac flapping is a trigger, how would this be handled in a WLAN environment as this event would be expected

[Brett Erickson]

You should try upgrading to 03.06.03E, the ones I have recently upgraded don't have a year of uptime yet however I do have one stack that has been running on 03.06.03E with an uptime of 50 weeks, 1 day, 16 hours, 28 minutes and the cpu looks like this;

Core 0: CPU utilization for five seconds: 9%; one minute: 10%; five minutes: 11%
Core 1: CPU utilization for five seconds: 4%; one minute: 5%; five minutes: 6%
Core 2: CPU utilization for five seconds: 4%; one minute: 25%; five minutes: 31%
Core 3: CPU utilization for five seconds: 95%; one minute: 73%; five minutes: 64%

I guess that does support your argument though seeing how one of my Cores is at 95%.

[Ali Aljuraib]

Hi Btrett,

I am just wondering if the high CPU issue has been solved after upgrading into 03.06.03E ?? please feedback as we are experience the same issue ?

Core 0: CPU utilization for five seconds: 98%; one minute: 98%;  five minutes: 97%
Core 1: CPU utilization for five seconds: 97%; one minute: 98%;  five minutes: 98%
Core 2: CPU utilization for five seconds: 100%; one minute: 98%;  five minutes: 98%
Core 3: CPU utilization for five seconds: 97%; one minute: 98%;  five minutes: 98%
PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process
5642   2734308     30851062 2427   49.36    48.77    48.82    0     stack-mgr
5640   4166095     17345165 779    43.64    44.62    45.08    1088  fed
9223   2881219     28810212 260    4.88     3.63     3.54     0     iosd
6305   1057746     20350254 1229   0.15     0.15     0.15     0     cpumemd
6294   3028210     78879150 1556   0.10     0.12     0.14     0     pdsd
5641   3567933     42488656 16     0.05     0.04     0.05     0     platform_mgr
9219   3376277     73361020 22     0.05     0.05     0.05     0     wcm