Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
12473
Views
0
Helpful
10
Replies

High CPU Utilization on 3850 stack

Go to solution
avinash.bolla
Level 1
Level 1

‎02-22-2018 07:47 AM - edited ‎03-08-2019 01:59 PM

Hello,

 

I need help on understanding the problem we are facing with one of our 3850 stack. The stack is currently supporting 4 switches. 

 

We have a high CPU utilization for the past one week and I am trying to find the reason. I couldn't get any solution so far. The best one I got was rebooting the entire stack. 

 

I have provided below the CPU utilization for all 4 cores its running on. And also the logs I have. 

History information for system:
 
                                                                
    555555555555555555555555555555555555555555555555555555555555
    555333333333344444444444444444444333333333333333444445555544
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60 ***                                                  *****  
 50 ************************************************************
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)
 
                                                                
    556785665565555655565565565555555565555555655555565556565555
    570905025445657245499706537545455455569797344744687775705455
100                                                             
 90                                                             
 80    **                                                       
 70    **              *              *              *   *      
 60 ***##**#* ****** * #******#* * ** #********  *  *#**##*** **
 50 ############################################################
 40 ############################################################
 30 ############################################################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
 
                                                                            
    676666668886676767777668667766666687676678767676667667666676676668786667
    907878860078909182002982864088999711829912282808971990988688807972018982
100                                                                         
 90           *                                                             
 80         ***            *          *      *                *      * *    
 70 ************************************************************************
 60 ***********************#************************************************
 50 ########################################################################
 40 ########################################################################
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0 
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%
 
 
History information for core 0:
          
                                                                
    111               111111111111111     1111111111111111111111
    333777777777799999222222222222222999990000000000222224444422
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30                                                             
 20                                                             
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)
 
                             11                     1   111     
    112591191141911911195129100919311121199921311911099900021112
    460531373272934912398438300929244443398928310953099700033234
100        *    *  *   *   * *** *       ***     *  ****##*     
 90     *  *    *  *   *   * *** *       ***     *  ****##*     
 80     *  *    *  *   *   * *## *       ***     *  ****##*     
 70     *  *    *  *   *   * *## *       **#     *  #***##*     
 60    **  *    *  *   **  * *## *       **#     *  #*#*##*     
 50    **  *  * *  *   **  * *## *       ###     *  ###*##*     
 40    **  *  * *  *   **  * *## *       ###     *  ###*###     
 30    **  *  * #  #   #*  # *## #*      ###  *  #  ###*###     
 20  **##  #  * #  #   ## *# ### #*   *  ###***  #* #######*   *
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
 
    111111111111111111111111111111111111111111111111111111111 11
    000000000000000000000000000000000000000000000000000000000900
    000000000000000000000000000000000000000000000000000000000900
100 ************************************************************************
 90 ************************************************************************
 80 ************************************************************************
 70 ************************************************************************
 60 ************************************************************************
 50 ******#***************************************#*************************
 40 ******#***##***********#****#****#**##********##*##*#*********#*****#***
 30 ##**########*#####*###*##*#################*#######*#*###*##############
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0 
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%
 
 
History information for core 1:
 
                                                                
                                                                
    999777777777799999666665555555555555556666666666888889999944
100                                                             
 90                                                             
 80                                                             
 70                                                             
 60                                                             
 50                                                             
 40                                                             
 30                                                             
 20                                                             
 10 **********************************************************  
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)
 
        11111  11                     11             11111111111
     19900000 9009921 199999932199  1900999211299919900000000000
    969900000990099919199998572489881900987057399918900000000000
100   ***###* *#**#    ***#**   **   *###**    *#* ****#########
 90   #*####* *#*##    ***#**   **   *###**    *#* #***#########
 80   #*####* *#*##    ***#**   *#   *###**    *#* #***#########
 70   ######* *#*##    ***#**   ##   *###**    *#* #***#########
 60   ####### *#*##    ***#**   ##   *###*#    *## #**##########
 50   ####### *#*##    ***#**   ##   ######    *## #############
 40   ####### #####    *#*#***  ##   ######    ### #############
 30   ####### #####*   #####**  ##   ######    ### #############
 20  *####### #####*   ######** ##   ######****### #############
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
 
    111111111111111111111111111111111111111111111111111111111111
    000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000
100 **************************##********************************************
 90 **********#***************###******#*****#*********#********************
 80 **********#*#*******##****###******#*#***#***#***###********#****#******
 70 ***#**#***#*###****###*#*####****#*####**##**#*#*###***#***##*####*#****
 60 ***##*##*#######***#####*####*#**#*####*######*#########**##########**#*
 50 #*######################*#########*###########*######################*##
 40 ########################################################################
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0 
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%
 
 
History information for core 2:
 
                                                                
    999999999999999999999999999999999999999999999999999999999999
    888888888888888888888889999988888999998888899999999998888899
100 ************************************************************
 90 ************************************************************
 80 ************************************************************
 70 ************************************************************
 60 ************************************************************
 50 ************************************************************
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)
 
        1   1111111111111111111             1                11 
    99990 39000000000000000000099 99994119290999 999 91123999009
    999909090000000000000000000969998810160909889997927806999009
100 ##***  **##################** ###*   * *###* *#*      *####*
 90 ##***  **##################** ###*   * *###* *#* *    *####*
 80 ##***  **##################** ###*   * *###* *#* *    *#####
 70 ##***  **##################** ###*   * *###* *#* *    ######
 60 ##***  **##################** ###*   * *#### *#* *    ######
 50 ##***  *###################** ####   * *#### *#* *    ######
 40 ##**#  *####################* ####*  * ##### *#* *   *######
 30 ##*## *###################### ####*  * ##### ##* *   *######
 20 ##### *###################### ####*  #*##### ### #**########
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
 
    1111111111111111111111111111111111111111111111111111111111 1
    000000000000000000000000000000000000000000000000000000000090
    000000000000000000000000000000000000000000000000000000000090
100 *##****###********#*****#******##*#************************************#
 90 *##****###*****#*##*****#****#*##*#*****************#*************#****#
 80 *##****###*****#*###****#****#*##*#*******##********#***#*********#***##
 70 *##**#####*#***#*###**#*#****#*#####**#***##***#**#*#*#*#***##****#***##
 60 ###**#####*#####*###*##*#***##*########*####**##*##*#*#*#**###****#*#*##
 50 ###*######*#####*######*#**###*#############**##*####*###**###*#*##*#*##
 40 ############################################*###*#############*#########
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0 
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%
 
 
History information for core 3:
 
                                     11111                      
    999999999999999999999999999999999000009999999999999999999999
    999999999999999999999999999999999000009999999999999999999999
100 ************************************************************
 90 ************************************************************
 80 ************************************************************
 70 ************************************************************
 60 ************************************************************
 50 ************************************************************
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per second (last 60 seconds)
 
    1111   1             1     11111111111 111 11111111         
    000099909991981999999029999000000000009000900000000912121 19
    000099909990925999899078998000000000009000900000000917000909
100 ####*##*##* *  *###*** *****######################**       *
 90 #######*##* *  *###*** **#**########################       *
 80 #######*##* ** *###**# *##*#########################       *
 70 ##########* ** ####**# *##*#########################       *
 60 ########### ** ####### ###*#########################       *
 50 ########### ** ####### ###*#########################       *
 40 ########### #* ####### ###*#########################       *
 30 ########### #* #######*############################# *     *
 20 ########### ##*#######*############################# * *   #
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5    
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
 
    1111111 1 1111111111111111111111 111111111111111111111111111
    000000090900000000000000000000009000000000000000000000000000
    000000090900000000000000000000009000000000000000000000000000
100 ****************#***************************#************##**********#**
 90 ***#************#***************************#***#****#***##**********#**
 80 ***##***********#********#****#********#****#***#****#***##*****#**#*#**
 70 #**##***********#******#*#****#********##***#*#*#****#*#*##***#*#**###**
 60 #**###*******##*#*****##*#****#********##***###*#**#*###*###*#####*###**
 50 ##*###****#*###*###*#*#####**##*****#*####*######**#*#######*#####*###**
 40 ######****#########################*##############*#*###################
 30 #######*#*##############################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0 
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%
 
Logs: 
Feb 21 10:51:42.385 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to down
Feb 21 10:51:43.385 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to down
Feb 21 10:53:05.266 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to up
Feb 21 10:53:06.264 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to up
Feb 21 10:53:27.444 Central: %DOT1X-5-FAIL: Authentication failed for client (009c.0206.13a9) on Interface Gi3/0/45 AuditSessionID AC14FFA10000483281A4389C
Feb 21 10:54:20.956 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to down
Feb 21 10:54:21.966 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to down
Feb 21 10:55:38.908 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to up
Feb 21 10:55:39.906 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to up
Feb 21 10:56:00.998 Central: %DOT1X-5-FAIL: Authentication failed for client (009c.0206.13a9) on Interface Gi3/0/45 AuditSessionID AC14FFA10000483381A69074
Feb 21 11:11:13.485 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Feb 21 11:11:14.499 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Feb 21 11:11:21.740 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Feb 21 11:11:22.740 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Feb 21 11:11:35.017 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to down
Feb 21 11:11:36.015 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to down
Feb 21 11:11:39.109 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/45, changed state to up
Feb 21 11:11:40.111 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/45, changed state to up
Feb 21 11:11:57.764 Central: %DOT1X-5-FAIL: Authentication failed for client (34e6.d70a.745f) on Interface Gi1/0/45 AuditSessionID AC14FFA10000483481B529E0
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 11:11:58.178 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 21 12:04:38.105 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to down
Feb 21 12:04:39.109 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to down
Feb 21 12:23:54.454 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to down
Feb 21 12:23:55.465 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to down
Feb 21 12:25:20.854 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/45, changed state to up
Feb 21 12:25:21.860 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/45, changed state to up
Feb 21 12:25:40.468 Central: %DOT1X-5-FAIL: Authentication failed for client (009c.0206.13a9) on Interface Gi3/0/45 AuditSessionID AC14FFA10000483581F8A616
Feb 21 12:58:17.604 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to down
Feb 21 12:58:18.608 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/40, changed state to down
Feb 21 12:58:26.195 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/40, changed state to up
Feb 21 12:58:27.194 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to up
Feb 21 12:58:34.974 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to down
Feb 21 12:58:36.976 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to up
Feb 21 12:58:57.157 Central: %DOT1X-5-FAIL: Authentication failed for client (0030.c15e.b980) on Interface Gi2/0/40 AuditSessionID AC14FFA10000483782171D9E
Feb 21 14:56:03.675 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to down
Feb 21 14:56:04.676 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to down
Feb 21 16:19:34.198 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to down
Feb 21 16:19:35.208 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to down
Feb 21 16:19:39.377 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to up
Feb 21 16:19:40.379 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Feb 21 16:25:07.438 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to down
Feb 21 16:25:08.439 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to down
Feb 21 16:25:11.127 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to up
Feb 21 16:25:12.126 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to up
Feb 21 16:25:45.796 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to down
Feb 21 16:25:46.798 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to down
Feb 21 16:51:31.824 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to down
Feb 21 16:51:32.825 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to down
Feb 22 03:04:53.506 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to down
Feb 22 03:04:54.518 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to down
Feb 22 03:05:01.208 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to up
Feb 22 03:05:02.210 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to up
Feb 22 03:17:25.455 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to down
Feb 22 03:17:26.454 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to down
Feb 22 03:17:33.189 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to up
Feb 22 03:17:34.189 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to up
Feb 22 03:19:18.285 Central: %DOT1X-5-FAIL: Authentication failed for client (9c93.4e5c.9aa2) on Interface Gi2/0/14 AuditSessionID AC14FFA100004846852ACA9E
Feb 22 06:25:53.567 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/41, changed state to down
Feb 22 06:25:56.685 Central: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/41, changed state to down
Feb 22 06:25:59.887 Central: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/41, changed state to up
Feb 22 06:26:00.888 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/41, changed state to up
Feb 22 06:26:24.076 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/41, changed state to down
Feb 22 06:26:25.086 Central: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/41, changed state to down
Feb 22 06:26:28.258 Central: %LINK-3-UPDOWN: Interface GigabitEthernet4/0/41, changed state to up
Feb 22 06:26:29.256 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/41, changed state to up
Feb 22 06:50:30.516 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to up
Feb 22 06:50:33.067 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to down
Feb 22 06:50:36.216 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to up
Feb 22 06:50:37.214 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to up
Feb 22 06:51:13.149 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to down
Feb 22 06:51:14.144 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to down
Feb 22 06:51:17.276 Central: %LINK-3-UPDOWN: Interface GigabitEthernet3/0/19, changed state to up
Feb 22 06:51:18.277 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/19, changed state to up
Feb 22 07:05:48.257 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to down
Feb 22 07:05:51.385 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to down
Feb 22 07:05:54.680 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to up
Feb 22 07:05:55.678 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Feb 22 07:06:33.215 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to down
Feb 22 07:06:34.225 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to down
Feb 22 07:06:37.387 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to up
Feb 22 07:06:38.384 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Feb 22 07:06:56.236 Central: %DOT1X-5-FAIL: Authentication failed for client (ecf4.bb4d.aebc) on Interface Gi2/0/7 AuditSessionID AC14FFA10000484A85FB3206
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:06:56.506 Central: %IOSXE-5-PLATFORM: 2 process fed: Logging is not supported for Client/Group ACL
Feb 22 07:20:25.847 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to up
Feb 22 07:20:26.846 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to up
Feb 22 07:20:43.426 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to down
Feb 22 07:20:44.428 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to down
Feb 22 07:20:47.505 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to up
Feb 22 07:20:48.510 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/15, changed state to up
Feb 22 08:26:52.275 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to up
Feb 22 08:26:53.280 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to up
Feb 22 08:27:26.394 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to down
Feb 22 08:27:27.398 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to down
Feb 22 08:27:30.644 Central: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/38, changed state to up
Feb 22 08:27:31.648 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/38, changed state to up
Feb 22 09:01:49.866 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:01:50.867 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:02:11.567 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:02:12.565 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:02:19.786 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:02:20.788 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:02:22.016 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:02:23.015 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:02:26.194 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:02:27.198 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:03:32.608 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:03:33.608 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to down
Feb 22 09:03:36.767 Central: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:03:37.773 Central: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/33, changed state to up
Feb 22 09:03:55.747 Central: %DOT1X-5-FAIL: Authentication failed for client (f01f.af21.905e) on Interface Gi1/0/33 AuditSessionID AC14FFA10000485186664DF2
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
 

 

Some of the logs are repeating at irregular interval. is this something to do with this high CPU utilization. Couple the cores are at 100% almost all the time. 

 

Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.914 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL
Feb 22 09:03:55.915 Central: %IOSXE-5-PLATFORM: STANDBY:1 process fed: Logging is not supported for Client/Group ACL

 

Any help on this issue is much appreciated. 

4 people had this problem
Labels:
0 Helpful
2 Accepted Solutions

Accepted Solutions

Go to solution
Mark Malone
VIP Alumni
VIP Alumni
In response to avinash.bolla

‎02-22-2018 08:09 AM

actually there in the link i posted on how to tshoot high cpu on a 3850 , and also stk manager

 

  • Forwarding Engine Driver (FED): This is the heart of the Cisco Catalyst 3850 Series Switch and is responsible for all hardware programming/forwarding.
  • IOSd: This is the Cisco IOS daemon that runs on the Linux kernel. It is run as a software process within the kernel.

This is what i would do , take the show tech run it through Cisco tool cli analyzer and see if you get a bug id back or if your completely over utilizing the switch

I would also roll the switch back rather than upgrade to the more stable ios-xe version 3.6.6E and see if that fixes the issue

 

View solution in original post

0 Helpful

Go to solution
Leo Laohoo
Hall of Fame
Hall of Fame
In response to avinash.bolla

‎02-22-2018 11:20 AM

@avinash.bolla wrote:

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5881 3303778 46124849 472 49.41 49.35 49.26 0 stack-mgr 
13631 3774917 28728769 167 2.01 2.66 2.69 34817 iosd 

There are several bugs involving high CPUs and "stack-mgr" and "iosd" processes and  CSCuo14511 is one of them.

View solution in original post

0 Helpful
10 Replies 10

Go to solution
Mark Malone
VIP Alumni
VIP Alumni

‎02-22-2018 07:52 AM

Whats the high cpu users when its maxed out
sh proc cpu sorted | exec 0.00

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117594-technote-hicpu3850-00.html
0 Helpful

Go to solution
avinash.bolla
Level 1
Level 1
In response to Mark Malone

‎02-22-2018 07:56 AM

This is the output for "sh proc cpu sorted | exclude 0.00"

 

Core 0: CPU utilization for five seconds: 10%; one minute: 45%; five minutes: 46%
Core 1: CPU utilization for five seconds: 99%; one minute: 66%; five minutes: 56%
Core 2: CPU utilization for five seconds: 99%; one minute: 95%; five minutes: 97%
Core 3: CPU utilization for five seconds: 6%; one minute: 15%; five minutes: 21%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5881 3303778 46124849 472 49.41 49.35 49.26 0 stack-mgr
13631 3774917 28728769 167 2.01 2.66 2.69 34817 iosd
5879 1258585 29330092 62 1.62 2.06 2.09 1088 fed
6484 2399245 18480507 238 0.54 0.59 0.59 0 ffm
13626 4118936 23572236 12 0.20 0.14 0.12 0 wcm
5880 2146464 30879302 14 0.15 0.08 0.09 0 platform_mgr
5608 1464328 90388706 158 0.05 0.01 0.04 0 system_mgr
6469 28782 20308219 21 0.05 0.02 0.05 0 obfld
6487 2055026 94832542 972 0.05 0.05 0.05 0 cpumemd

 

As you can see couple of cores are always at near 100%. 

0 Helpful

Go to solution
Mark Malone
VIP Alumni
VIP Alumni
In response to avinash.bolla

‎02-22-2018 08:00 AM

there the platform and forwarder processes so either you have hit a bug or your hammering the switch , what ios-xe version are you running currently ?
0 Helpful

Go to solution
avinash.bolla
Level 1
Level 1
In response to Mark Malone

‎02-22-2018 08:05 AM

Currently, the version its running is "IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.01E".

 

Can you please describe, which one are the platform and forwarder processes again?

0 Helpful

Go to solution
Mark Malone
VIP Alumni
VIP Alumni
In response to avinash.bolla

‎02-22-2018 08:09 AM

actually there in the link i posted on how to tshoot high cpu on a 3850 , and also stk manager

 

  • Forwarding Engine Driver (FED): This is the heart of the Cisco Catalyst 3850 Series Switch and is responsible for all hardware programming/forwarding.
  • IOSd: This is the Cisco IOS daemon that runs on the Linux kernel. It is run as a software process within the kernel.

This is what i would do , take the show tech run it through Cisco tool cli analyzer and see if you get a bug id back or if your completely over utilizing the switch

I would also roll the switch back rather than upgrade to the more stable ios-xe version 3.6.6E and see if that fixes the issue

 

0 Helpful

Go to solution
avinash.bolla
Level 1
Level 1
In response to Mark Malone

‎02-22-2018 08:13 AM

Thank you very much for your quick response. Will try the those options and will update here. 

0 Helpful

Go to solution
Leo Laohoo
Hall of Fame
Hall of Fame
In response to avinash.bolla

‎02-22-2018 11:20 AM

@avinash.bolla wrote:

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5881 3303778 46124849 472 49.41 49.35 49.26 0 stack-mgr 
13631 3774917 28728769 167 2.01 2.66 2.69 34817 iosd 

There are several bugs involving high CPUs and "stack-mgr" and "iosd" processes and  CSCuo14511 is one of them.

0 Helpful

Go to solution
neutron
Level 1
Level 1

‎02-22-2018 11:27 AM

right, major problem here is that this switch stack is the distribution switch for most of the main building. But luckily this building is not 24/7.

 

So a reboot or os upgrade or downgrade would be possible.

It's interesting, the only real problem i've had with 3850s before is that crazy ip device-tracking bug that messes up VMware (or that VMware messes up, depending on who you ask!)

 

otherwise they have been pretty rock solid.

 

Hi Avi.

-Conan
"It's probably NOT the network."
0 Helpful

Go to solution
Leo Laohoo
Hall of Fame
Hall of Fame
In response to neutron

‎02-22-2018 11:28 AM

Do you have IPv6 enabled?

0 Helpful

Go to solution
avinash.bolla
Level 1
Level 1
In response to Leo Laohoo

‎02-22-2018 12:03 PM

No, we didn't enable IPV6.  And also, I analysed the "Show Tech-support" in Cisco CLI analyzer and the result was a bug "CSCuo14511" for "stack-mgr" high CPU utilization. 

We are going to upgrade the stack to rectify this. 

 

 

0 Helpful
Customers Also Viewed These Support Documents