%SW_MATM-4-MACFLAP_NOTIF: Host in vlan 105 is flapping between port Po2 and port Gi1/0/40 HSRP

[anilkumar.cisco]

Hello Team,

I have Core A and Core B running in HSRP and both core are in Stack.

Recording server is connected to port gig 1/0/40 (access vlan 105) on both Core switches, shows the below continuous flapping

Apr 5 13:52:37: %SW_MATM-4-MACFLAP_NOTIF: Host 98f2.b3ef.ac38 (recording server mac)  in vlan 105 is flapping between port Po2 and port Gi1/0/40

configuration:-

Core A and B are connected via port channel 2

interface Port-channel2
description 3750-CORE A to 3750-COREB trunk for HSRP
switchport trunk allowed vlan 101,102,105,204,205
switchport mode trunk

interface GigabitEthernet2/1/2
description 3750-CORE A to 3750-COREB trunk for HSRP
switchport trunk allowed vlan 101,102,105,204,205
switchport mode trunk
channel-group 2 mode on

interface GigabitEthernet3/1/2
description 3750-CORE A to 3750-COREB trunk for HSRP
switchport trunk allowed vlan 101,102,105,204,205
switchport mode trunk
channel-group 2 mode on

Core A..

switch 1 provision ws-c3650-48ts
switch 2 provision ws-c3650-48ts
switch 3 provision ws-c3650-48ts
switch 4 provision ws-c3650-48ts

sh ver

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 52 WS-C3650-48TS 03.06.04.E cat3k_caa-universalk9 INSTALL
2 52 WS-C3650-48TS 03.06.04.E cat3k_caa-universalk9 INSTALL
3 52 WS-C3650-48TS 03.06.04.E cat3k_caa-universalk9 INSTALL
4 52 WS-C3650-48TS 03.06.04.E cat3k_caa-universalk9 INSTALL

Core B also have the same configuration.

Port channel 2 is up.. and no issue..

Core A is root bridge for vlan 102.102,105,204 and 205..

but at every 15 minute I am getting the flapping error..

Now i am bothering/worries, if I will connect one Stack Access switch for Vlan 204 and 205 and will uplink to CORE A and Core B via switch port mod trunk command.. the STP will run and complete network will be down..

Also, now sure.. what will happen by this flapping.. we have got complain from the customer yesterday that recording was not working.. when we visited the site today  then came to know.. if is because port channel 2 between Core switch were down..

Not sure why, it is because of 204 and 205 vlan we have added to port-channel 2 yesterday thats why or what could be exact reason..

[Leo Laohoo]

---

@anilkumar.cisco wrote:

Recording server is connected to port gig 1/0/40 (access vlan 105) on both Core switches, shows the below continuous flapping

---

If the recording server is in Gi 1/0/40 then try this: 
1.  Ping the IP address of the recording server; 
2.  Shut/disable the Gi 1/0/40
Is the recording server still reachable?

NOTE:  Please read FN - 64191 - Catalyst 3850/3650 Switches That Run Cisco IOS XE Releases 3.6.5E/3.7.4E or Earlier Might Exhibit High CPU 

[anilkumar.cisco]

Thanks , Leo Laohoo for your prompt response..
I will try this tomorrow once will be onsite and then confirm you..
As per the customer, there were two recording server IP address and must be connected to both core switch via different link.. will confirm by tomorrow..

May I know the reason, why you have given high CPU link?
Once again thanks for your reply..
Best Regards
Anil Singh

[Leo Laohoo]

---

@anilkumar.cisco wrote:
May I know the reason, why you have given high CPU link?

---

Are you serious about this question or are you just trying to be funny?

[anilkumar.cisco]

Hello Leo,

the CPU utilization of both Core A and B is going continuously 80% and sometimes spikes at 100%.. Similarly we are able to observe 100% spike for  individual cores 1, 2, and 3 , below is the command output..

and yes.. I am not funny as not able to related the provided link with mine issue.. so need your expert advise..

B#sh process cpu hist

History information for system:

                                                               
    777777777777777777777777777777777777777777777777777777777777
    555566666666666666655555444445555555555555555555555555666667
100                                                            
 90                                                            
 80 ************************     *******************************
 70 ************************************************************
 60 ************************************************************
 50 ************************************************************
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per second (last 60 seconds)

                                                               
    777777777777777777777777777777777777777777777979777779777777
    766668667677766666666676666687666666677777777987776768767667
100                                              * *     *     
 90                                              * *     *     
 80 ###**#############*####****#################################
 70 ############################################################
 60 ############################################################
 50 ############################################################
 40 ############################################################
 30 ############################################################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

                                   1                                       
    987778777777778777778777797999807777778797887777878999988888888878888888
    818983888888980988882989878687108888890988039998395799916240754696245135
100 *                        * *** *        *          ****                
 90 *                        * *** *        *         ***** *   ** * *  *  *
 80 ########################################################################
 70 ########################################################################
 60 ########################################################################
 50 ########################################################################
 40 ########################################################################
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

History information for core 0:

                  11111                                   11111
    222299999999990000077777222227777799999999997777799999000009
    666666666999990000055555111117777799999666665555599999000009
100     ***************               **********     ***********
 90     ***************               **********     ***********
 80     ********************     *******************************
 70     ********************     *******************************
 60     ********************     *******************************
 50     ********************     *******************************
 40     ********************     *******************************
 30 ************************     *******************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per second (last 60 seconds)

    111  1 11 111 11  11111  1 11111   1 111111111111111  1 1111
    000990900900090099000009909000009990900000000000000099090000
    000990900900090099000009904000009990900000000000000099090000
100 #****#****###*##***##***** **#*##**##****#******#******##***
 90 ##***##*#*######*######****#*####*#####*###**####****#####*#
 80 ####*##################****######*#####*###########*######*#
 70 ####*####################*##################################
 60 ############################################################
 50 ############################################################
 40 ############################################################
 30 ############################################################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    111111111111111111111111111111111111111111111111111111111111111111111111
    000000000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000000000
100 ************************************************************************
 90 ####*##*#########***#######*#*########*##*#######*#*####################
 80 ########################################################################
 70 ########################################################################
 60 ########################################################################
 50 ########################################################################
 40 ########################################################################
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

History information for core 1:

                                                               
    999944444444448888888888999994444477777888884444422222222223
    888833333111117777755555666666666611111666668888844444555550
100 ****                    *****                              
 90 ****          ***************          *****               
 80 ****          ***************          *****               
 70 ****          ***************     **********               
 60 ****          ***************     **********               
 50 ****          ***********************************          
 40 *************************************************          
 30 *************************************************     ******
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per second (last 60 seconds)

               11             1          1 1     1 1           
    999996999990098899959899990992888996909099889090999999894999
    598999949970097899959989990989688948409090989080999999670998
100 ***** * #***#*  *** * *******    *   ****   ***#*#**** * ***
 90 ***** **#**###***** ****#**** ***** ***********###****** ***
 80 *#*#* #*##*###*##** ****###** **##* #*####***#####**#*** ***
 70 *#*##*#*##*######## #**####*# *###**######*#######*###** *#*
 60 *####*#*##*########*#**###### *####*##################*# *##
 50 #####*#############*#**###### #####*##################*# ###
 40 #####*#############*#*####### ##########################*###
 30 #############################*##############################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    111111111111111111111111111111111111111111111111111111111111111111111111
    000000000000000000000000000000000000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000000000000000
100 ************************************************************************
 90 *************************#*****************#************#***************
 80 ###**#***#**#####***##*#*##******#***#****##****#****#*##*#*########*#*#
 70 ##########*#########################*#*##*###*##########################
 60 ########################################################################
 50 ########################################################################
 40 ########################################################################
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

History information for core 2:

                                                               
    777766666666662222244444888888888833333222227777788888888888
    888899999777770000044444333331111133333111119999922222222223
100                                                            
 90                                                            
 80 ****                    **********          ****************
 70 **************          **********          ****************
 60 **************          **********          ****************
 50 **************          **********          ****************
 40 **************     ***************          ****************
 30 **************     ********************     ****************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per second (last 60 seconds)

               1                                11111    1     
    879998898990828898888989899888887988899999990000099890888898
    569794398830933294144435279449328932399999990000018580823334
100   ***  * * *    *      * **      *   *#########** * **     
 90 * ***  ****#*   *    * * **  *   *   *#########********   *
 80 *****#*****## *****#**#******#********#########******##*#***
 70 ****##*#**### *****#*##******#*****#*###########**#**##*#*#*
 60 #*#*##*###### *****#*###*#***##****#*###############*##*####
 50 #*#*##*###### ***#*#*###*######**###*##################*####
 40 #*########### #*################*###*#######################
 30 ############# ##############################################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

    11111111111111111111111111111111111111 11 111111111111111111111111111111
    000000000000000000000000000000000000009009000000000000000000000000000000
    000000000000000000000000000000000000009009000000000000000000000000000000
100 **************************************************************#*********
 90 ******#*******#*******************************#***************#**#***#**
 80 **#*#*#*##*##*#******#****#**#*#***#*#*##*****#*##*##*****#***##*##**#**
 70 ###################*####*##**#*#######*##*###*#*######*#*##***#####*##*#
 60 ###################################################################*####
 50 ########################################################################
 40 ########################################################################
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

History information for core 3:

                                                               
    999999999999999999999999999999999999999999999999999999999999
    888888888888887777788888888888888888888888888888888888888889
100 ************************************************************
 90 ************************************************************
 80 ************************************************************
 70 ************************************************************
 60 ************************************************************
 50 ************************************************************
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per second (last 60 seconds)

                                                 1             
    999999998994899999999999999999999999998988998089899999999999
    989899994999298989999998999899989989894943986029389999999999
100 ######## **  ########################* *  ** * * *###*######
 90 ######## *#  ########################* *  **** * *###*######
 80 ########*## *########################*************##########
 70 ########*## *#########################************##########
 60 ########*## *#########################*#**#*#**#*###########
 50 ###########**#########################*#**####*#############
 40 ###########**############################*####*#############
 30 ###########*################################################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

            1111     1 111 1 1   1 1 1     111  1 1  111111 1   11     111 1
    999999990000999990900090909990909099999000990909900000090999009999900090
    999999990000999990900090909990909099999000990909900000090999009999900090
100 *****************************************#******************************
 90 **********#********#****#**##*#*******#**#***#*#******#************#**#*
 80 *******#**#******###***##**##*#*#*#*#*#**#***#*#******#**#*#*******##*#*
 70 ##*###*##*#**#*#####**###**##########*#####*##*#*#########*###*##**##*##
 60 ##############*###############################################*##*###*##
 50 ##############################################################*#########
 40 ########################################################################
 30 ########################################################################
 20 ########################################################################
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                       CPU% per hour (last 72 hours)
                    * = maximum CPU%   # = average CPU%

[Leo Laohoo]

Read the Field Notice and understand what it means and the workaround.

[anilkumar.cisco]

Thanks.. I understood and recommended the customer for upgrade..



Pls note: As of now.. the flapping is not there since yesterday.. so I am still monitoring this case..



Once again thanks for your help..

[paul driver]

Hello

Does that server have two links into either core if so are they teamed?
Do you have stp portfast enabled on these ports?

sh int trunk
sh spanning-tree detail | i Port|VLAN|tra|BPDU

[anilkumar.cisco]

Hello Paul,

thanks for your reply..

As per Recording vendor, they have configured teaming in Active/standby mode..
and yes , port fast has been configured on these port..
since yesterday 5:30 PM Singapore time, I did not see any flapping..
below is the interface configuration output..

interface GigabitEthernet1/0/40
switchport access vlan 105
switchport mode access
switchport nonegotiate
spanning-tree portfast
will share the requested output very soon...

[anilkumar.cisco · Accepted solution]

Hello Paul,
pls find the below response..

SW#show spanning-tree detail | i Port|VLAN|tra|BPDU
VLAN0204 is executing the rstp compatible Spanning Tree protocol
Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
from StackPort1
Port 49 (GigabitEthernet1/0/49) of VLAN0204 is designated forwarding
Port path cost 4, Port priority 128, Port Identifier 128.49.
Number of transitions to forwarding state: 1
BPDU: sent 122, received 0
Port 105 (GigabitEthernet2/0/49) of VLAN0204 is designated blocking
Port path cost 4, Port priority 128, Port Identifier 128.105.
Number of transitions to forwarding state: 0
BPDU: sent 4, received 0
VLAN0205 is executing the rstp compatible Spanning Tree protocol
Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
from StackPort1
Port 49 (GigabitEthernet1/0/49) of VLAN0205 is designated forwarding
Port path cost 4, Port priority 128, Port Identifier 128.49.
Number of transitions to forwarding state: 1
BPDU: sent 122, received 0
Port 105 (GigabitEthernet2/0/49) of VLAN0205 is designated blocking
Port path cost 4, Port priority 128, Port Identifier 128.105.
Number of transitions to forwarding state: 0
BPDU: sent 4, received 0
ITSAccess-SW#sh int trun

Port Mode Encapsulation Status Native vlan
Gi1/0/49 on 802.1q trunking 1
Gi2/0/49 on 802.1q trunking 1

Port Vlans allowed on trunk
Gi1/0/49 204-205
Gi2/0/49 204-205

Port Vlans allowed and active in management domain
Gi1/0/49 204-205
Gi2/0/49 204-205

Port Vlans in spanning tree forwarding state and not pruned
Gi1/0/49 204-205
Gi2/0/49 none

[anilkumar.cisco]

Thanks team, issue got resolved... seems issue were there at Server side.. and during new vlan addition on Core switch followed by new switch addition on the core switch, the port channel must have hang..
Thanks for help

[anilkumar.cisco]

Hello Team,

Today again flapping happened.. with below logs..
on Core A for very few seconds only..
Apr 14 10:23:52: %SW_MATM-4-MACFLAP_NOTIF: Host a03d.6f74.4e5a in vlan 204 is flapping between port Gi1/1/2 and port Po2
Apr 14 10:23:52: %SW_MATM-4-MACFLAP_NOTIF: Host a03d.6f74.4e4c in vlan 205 is flapping between port Gi1/1/2 and port Po2
Apr 14 10:23:52: %SW_MATM-4-MACFLAP_NOTIF: Host a03d.6f74.4e5a in vlan 204 is flapping between port Gi1/1/2 and port Po2 (ITScore_A-2)
Apr 14 10:23:53: %SW_MATM-4-MACFLAP_NOTIF: Host a03d.6f74.4e4c in vlan 205 is flapping between port Gi1/1/2 and port Po2 (ITScore_A-2)

but continuously on Core B
Apr 15 10:33:51: %SW_MATM-4-MACFLAP_NOTIF: Host 98f2.b3ef.ac38 in vlan 105 is flapping between port Po2 and port Gi1/0/40
Apr 15 10:33:51: %SW_MATM-4-MACFLAP_NOTIF: Host 98f2.b3ef.ac38 in vlan 105 is flapping between port Po2 and port Gi1/0/40 (ITScore_B-2)
Apr 15 10:38:50: %SW_MATM-4-MACFLAP_NOTIF: Host 98f2.b3ef.ac38 in vlan 105 is flapping between port Po2 and port Gi1/0/40
Apr 15 10:38:50: %SW_MATM-4-MACFLAP_NOTIF: Host 98f2.b3ef.ac38 in vlan 105 is flapping between port Po2 and port Gi1/0/40 (ITScore_B-2)
Apr 15 10:43:51: %SW_MATM-4-MACFLAP_NOTIF: Host 98f2.b3ef.ac38 in vlan 105 is flapping between port Po2 and port Gi1/0/40
Apr 15 10:43:51: %SW_MATM-4-MACFLAP_NOTIF: Host 98f2.b3ef.ac38 in vlan 105 is flapping between port Po2 and port Gi1/0/40 (ITScore_B-2)
Apr 15 10:48:51: %SW_MATM-4-MACFLAP_NOTIF: Host 98f2.b3ef.ac38 in vlan 105 is flapping between port Po2 and port Gi1/0/40

and because of this.. now Turrets phone are not getting IP via DHCP address if we are rebooting them.. but recording is working perfectly.

As interim we had shut down Gi1/0/40 in Core B and now no more flapping.. not able to correlate why this flapping because of problem of Server side or Cisco IOS bug or configuration issue..

I am seeing OSPF configuration different between Core A and B.. but I don't thing that can create this issue..
for your reference below is the OSPF configuration of both Core Switches..

Core A.
router ospf 1
router-id X.X.246.231
ispf
log-adjacency-changes detail
auto-cost reference-bandwidth 100000
timers throttle spf 5 100 5000
timers throttle lsa 0 100 5000
timers lsa arrival 50
redistribute connected subnets
redistribute static subnets
passive-interface default
no passive-interface Vlan103
no passive-interface GigabitEthernet2/0/2
no passive-interface GigabitEthernet3/0/2
no passive-interface Port-channel3
no passive-interface Port-channel4
network X.0.0.0 0.255.255.255 area 0.0.0.2

Core B,
router ospf 1
router-id x.x.246.232
ispf
log-adjacency-changes detail
auto-cost reference-bandwidth 100000
timers throttle spf 5 100 5000
timers throttle lsa 0 100 5000
timers lsa arrival 50
redistribute static subnets
passive-interface default
no passive-interface Vlan101
no passive-interface Vlan102
no passive-interface Vlan105
no passive-interface GigabitEthernet2/0/2
no passive-interface GigabitEthernet3/0/2
no passive-interface Port-channel3
no passive-interface Port-channel4
network x.0.0.0 0.255.255.255 area 0.0.0.2

only no passive interface command in Core B for vlan 101, 102 and 105.. but only no passive interface command in Core A..  i don't thing this could be an issue..