Cisco 3850 High CPU due to fed

[johnlloyd_13]

hi,

we got a 3850 that's reporting to have constant high CPU.

our team is considering an IOS upgrade but i ran into this site below and suggesting to enable this command ipv6 mld snooping.

my question is, does someone ran into the same issue and solved it by applying the said command?

is this safe to enable this command in a production switch?

https://www.theroutingtable.com/cisco-3850-high-cpu-usage-fed-and-iosd-processes/

#sh ve
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.02.03.SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Mon 23-Sep-13 18:24 by prod_rel_team

<SNIP>

Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
     1 56    WS-C3850-48P       03.02.03.SE       cat3k_caa-universalk9 INSTALL

#show processes cpu sorted
Core 0: CPU utilization for five seconds: 77%; one minute: 71%;  five minutes: 71%
Core 1: CPU utilization for five seconds: 28%; one minute: 71%;  five minutes: 78%
Core 2: CPU utilization for five seconds: 96%; one minute: 72%;  five minutes: 67%
Core 3: CPU utilization for five seconds: 99%; one minute: 90%;  five minutes: 91%
PID    Runtime(ms) Invoked  uSecs  5Sec     1Min     5Min     TTY   Process
5632   547159      38565649 240    73.91    73.47    73.38    1088  fed                
9223   2869678     17955697 98     0.97     1.17     1.15     0     iosd               
6297   1592710     28366670 1186   0.14     0.15     0.14     0     cpumemd            
6278   403412      5609218  837    0.10     0.01     0.03     0     oom_poll.sh        
9219   3878616     89615676 25     0.10     0.10     0.10     0     wcm                
5633   2355637     33902453 13     0.05     0.14     0.12     0     platform_mgr       
5634   918091      19813894 41     0.05     0.06     0.05     0     stack-mgr          
6286   605408      34263960 50     0.05     0.09     0.10     0     pdsd               
6290   905582      23382817 22     0.05     0.01     0.01     0     netd               
1      1650        1056     1562   0.00     0.00     0.00     0     init               
2      0           126      0      0.00     0.00     0.00     0     kthreadd           
3      9620        1171075  8      0.00     0.00     0.00     0     migration/0        
4      0           3        0      0.00     0.00     0.00     0     sirq-high/0        
5      1139468     45134130 2      0.00     0.00     0.00     0     sirq-timer/0       
6      0           6        0      0.00     0.00     0.00     0     sirq-net-tx/0      
7      10          9636     1      0.00     0.00     0.00     0     sirq-net-rx/0      
8      1180        2734379  0      0.00     0.00     0.00     0     sirq-block/0       
9      0           8        0      0.00     0.00     0.00     0     sirq-block-iopo    
10     20          3286     6      0.00     0.00     0.00     0     sirq-tasklet/0

#show process cpu detail process fed sorted         
Core 0: CPU utilization for five seconds: 67%; one minute: 61%; five minutes: 63%
Core 1: CPU utilization for five seconds: 90%; one minute: 63%; five minutes: 68%
Core 2: CPU utilization for five seconds: 44%; one minute: 77%; five minutes: 74%
Core 3: CPU utilization for five seconds: 99%; one minute: 98%; five minutes: 96%
PID    T C  TID    Runtime(ms) Invoked uSecs  5Sec      1Min     5Min     TTY   Process
                                               (%)       (%)      (%)                  
5632   L           1730899     3856815 240    73.05     73.39   73.34   1088  fed                
5632   L 1  9959   3858823     1188680 0      24.13     24.15   24.13   0     PunjectRx          
5632   L 3  9957   1862104     1126280 0      24.08     24.16   24.15   0     Replenish          
5632   L 2  9934   3063734     1230300 0      18.38     18.48   18.28   0     Xcvr               
5632   L 0  10126  1153414     2162711 0      5.74      5.73    5.90    0     XcvrPoll           
5632   L 0  6089   719166      5508445 0      0.43      0.42    0.43    0     fed-ots-nfl        
5632   L 0  6086   381850      7773271 0      0.10      0.21    0.19    0     fed-ots-main       
5632   L 0  6117   1693000     1885126 0      0.10      0.14    0.14    0     IntrDrv            
5632   L 0  9960   3774407     6739445 0      0.10      0.08    0.08    0     PunjectTx          
5632   L 1  5632   31910       7488    0      0.00      0.00    0.00    1088  fed                
5632   L 2  6073   3951458     5714392 0      0.00      0.04    0.02    1088  fed                
5632   L 3  6074   30          1482    0      0.00      0.00    0.00    1088  fed                
5632   L 2  6075   3178510     2822516 0      0.00      0.03    0.04    1088  fed      

[Leo Laohoo]

John, 

3.2.3 is a very, very old IOS.  I think it's best to upgrade to 3.7.X or Denali.  

[johnlloyd_13]

leo,

pre, what's Denali?

how about the command mentioned? will it solve the high CPU issue without IOS update?

[Leo Laohoo · Accepted solution]

John, 

What is the uptime of this switch and stack?

Denali is the "new" IOS for the 3650/3850, version 16.X.  

I understand the reluctance to make a decision to bounce the switch, however, the have been known vulnerabilities and, possibly, crashes from this version. 

I didn't bother chasing up BTK if the high CPU is, in any way, related to the version currently running on this switch.

[johnlloyd_13]

thanks! will go the IOS upgrade route.

[Leo Laohoo]

Thanks for taking the time to rate our post(s), pre.  :)

[johnlloyd_13]

hi leo,

sorry just got another question that pop up.

i'll be doing the 3850 switch upgrade remotely.

can you execute the software install command remotely or need this to be via console?

i saw there's some macros executed and then asked to reboot.

[1 ]: Finished installing software.  New software will load on reboot.
[1 ]: Committing provisioning file
[1 ]: Do you want to proceed with reload? [yes/no]: yes    <<< WILL I STILL HAVE REMOTE SSH ACCESS  UP TO THIS POINT?

[Leo Laohoo]

can you execute the software install command remotely or need this to be via console?

Can be done remotely. 

Once you answer "y" the stack will reload. 

[johnlloyd_13]

pre,

referring to link below about the 3850 IOS upgrade using the software install command:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117552-technote-cat3850-00.html

do i need to put a boot sequence command?

boot system switch all flash:<NEW IOS>.bin

the 3850 switch that i'll be upgrading has or running on .pkg file (no .bin file).

[Leo Laohoo · Accepted solution]

Pre, 

There is no need to change the boot variable statement if you're using the "software install" command.  

Please don't forget to run the "software clean" so that the contents of the flash doesn't look like a Chinese take-away menu.  

[napdelcaribe]

Greetins,

We have the same issue.

We actived the IPv6 mld snooping command and nothing happenen; the high cpu consumption due to fed proccess still persist.

We have being searching for almost two days  and haven't anything to resolve this.

Is there any other way out of this? update? power cicle the equipment??

Regards,