Solved: HSRP State flapping

mamadou.barry · ‎08-08-2017

Hello

I've just migrate my core switch from Cisco 3560 to Cisco WS-C3850-48T-L (IOS-XE Software, (CAT3K_CAA-UNIVERSALK9-M), Version 03.06.06E).

The issue I meet is that HSRP state is flapping continiously when I check my log, I have no physical link failure between two switches and STP is configured according to HSRP priority on SVI.

Can someone help me find solution of that issue, below some config and log capture

Switch1:

Aug 8 03:58:00: %HSRP-5-STATECHANGE: Vlan12 Grp 5 state Standby -> Active

Aug 8 03:58:00: %HSRP-5-STATECHANGE: Vlan506 Grp 3 state Standby -> Active

Aug 8 03:58:00: %HSRP-5-STATECHANGE: Vlan12 Grp 5 state Active -> Speak

Aug 8 03:58:00: %HSRP-5-STATECHANGE: Vlan506 Grp 3 state Active -> Speak

Aug 8 03:58:03: %HSRP-5-STATECHANGE: Vlan12 Grp 5 state Speak -> Standby

Aug 8 03:58:03: %HSRP-5-STATECHANGE: Vlan506 Grp 3 state Speak -> Standby

Aug 8 04:30:35: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202

Aug 8 04:30:55: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202

Aug 8 04:30:56: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202

Aug 8 04:30:57: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202

Aug 8 04:56:21: %TRACK-6-STATE: 1 ip sla 1 reachability Up -> Down

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan510 Grp 13 state Standby -> Active

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan15 Grp 215 state Standby -> Active

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan219 Grp 219 state Standby -> Active

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan100 Grp 8 state Standby -> Active

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan16 Grp 6 state Standby -> Active

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan504 Grp 1 state Standby -> Active

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan504 Grp 1 state Active -> Speak

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan510 Grp 13 state Active -> Speak

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan15 Grp 215 state Active -> Speak

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan219 Grp 219 state Active -> Speak

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan100 Grp 8 state Active -> Speak

Aug 8 04:56:21: %HSRP-5-STATECHANGE: Vlan16 Grp 6 state Active -> Speak

Aug 8 04:56:24: %HSRP-5-STATECHANGE: Vlan15 Grp 215 state Speak -> Standby

Aug 8 04:56:24: %HSRP-5-STATECHANGE: Vlan100 Grp 8 state Speak -> Standby

Aug 8 04:56:24: %HSRP-5-STATECHANGE: Vlan510 Grp 13 state Speak -> Standby

Aug 8 04:56:24: %HSRP-5-STATECHANGE: Vlan16 Grp 6 state Speak -> Standby

spanning-tree vlan 11,101,103,105,505,509,511 priority 24576
spanning-tree vlan 12,15-16,19,100,102,104,219,504,506,510,512 priority 28672

S3850-Siege-Federateur-1#sh spanning-tree br

Hello Max Fwd
Vlan Bridge ID Time Age Dly Protocol
---------------- --------------------------------- ----- --- --- --------
VLAN0001 32769 (32768, 1) 2c86.d244.4e80 2 20 15 rstp
VLAN0011 24587 (24576, 11) 2c86.d244.4e80 2 20 15 rstp
VLAN0012 28684 (28672, 12) 2c86.d244.4e80 2 20 15 rstp
VLAN0015 28687 (28672, 15) 2c86.d244.4e80 2 20 15 rstp
VLAN0016 28688 (28672, 16) 2c86.d244.4e80 2 20 15 rstp
VLAN0019 28691 (28672, 19) 2c86.d244.4e80 2 20 15 rstp
VLAN0100 28772 (28672, 100) 2c86.d244.4e80 2 20 15 rstp
VLAN0101 24677 (24576, 101) 2c86.d244.4e80 2 20 15 rstp
VLAN0102 28774 (28672, 102) 2c86.d244.4e80 2 20 15 rstp
VLAN0103 24679 (24576, 103) 2c86.d244.4e80 2 20 15 rstp
VLAN0104 28776 (28672, 104) 2c86.d244.4e80 2 20 15 rstp
VLAN0105 24681 (24576, 105) 2c86.d244.4e80 2 20 15 rstp
VLAN0219 28891 (28672, 219) 2c86.d244.4e80 2 20 15 rstp
VLAN0504 29176 (28672, 504) 2c86.d244.4e80 2 20 15 rstp
VLAN0505 25081 (24576, 505) 2c86.d244.4e80 2 20 15 rstp
VLAN0506 29178 (28672, 506) 2c86.d244.4e80 2 20 15 rstp
VLAN0509 25085 (24576, 509) 2c86.d244.4e80 2 20 15 rstp
VLAN0510 29182 (28672, 510) 2c86.d244.4e80 2 20 15 rstp
VLAN0511 25087 (24576, 511) 2c86.d244.4e80 2 20 15 rstp
VLAN0512 29184 (28672, 512) 2c86.d244.4e80 2 20 15 rstp

S3850-Siege-Federateur-1#sh standby br
P indicates configured to preempt.
|
Interface Grp Pri P State Active Standby Virtual IP
Vl11 7 100 P Active local 10.20.3.3 10.20.3.2
Vl12 5 50 Standby 10.20.2.3 local 10.20.2.2
Vl15 215 50 Standby 10.20.1.195 local 10.20.1.194
Vl16 6 50 Standby 10.20.0.232 local 10.20.0.231
Vl19 19 50 Standby 192.168.1.3 local 192.168.1.2
Vl100 8 50 Standby 10.20.100.202 local 10.20.100.200
Vl101 9 100 P Active local 10.20.101.202 10.20.101.200
Vl102 10 50 Standby 10.20.102.202 local 10.20.102.200
Vl104 12 50 Standby 10.20.104.202 local 10.20.104.200
Vl105 16 100 P Active local 10.20.105.202 10.20.105.200
Vl219 219 50 Standby 192.168.2.3 local 192.168.2.2
Vl504 1 50 Standby 10.20.3.195 local 10.20.3.194
Vl505 2 100 P Active local 10.20.3.131 10.20.3.130
Vl506 3 50 Standby 10.20.3.67 local 10.20.3.66
Vl509 4 100 P Active local 10.20.3.35 10.20.3.34
Vl510 13 50 Standby 10.20.0.35 local 10.20.0.34
Vl511 14 100 P Active local 10.20.0.3 10.20.0.2
Vl512 15 50 Standby 10.20.0.67 local 10.20.0.66

S3850-Siege-Federateur-1#sh run int vlan 15
Building configuration...

Current configuration : 275 bytes
!
interface Vlan15
description **** VLAN_Serv_Intern_Yeumbeul ****
ip address 10.20.1.193 255.255.255.192
ip helper-address 10.20.3.5
standby version 2
standby 215 ip 10.20.1.194
standby 215 timers 1 3
standby 215 priority 50
standby 215 authentication xxxxxx!

S3850-Siege-Federateur-1#sh int port-channel 10 summary

*: interface is up
IHQ: pkts in input hold queue IQD: pkts dropped from input queue
OHQ: pkts in output hold queue OQD: pkts dropped from output queue
RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)
TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)
TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
-----------------------------------------------------------------------------------------------------------------
* Port-channel10 0 0 0 0 8358000 2104 39552000 5111 0
S3850-Siege-Federateur-1#

Switch 2:

Aug 8 04:30:55: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 04:30:56: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 04:30:57: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 04:30:57: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 04:36:08: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPINFORM, chaddr: 0000.0000.0000, MAC sa: 0018.8b7b.2df2
Aug 8 04:36:08: %DHCP_SNOOPING-5-DHCP_SNOOPING_UNTRUSTED_PORT: DHCP_SNOOPING drop message on untrusted port, message type: DHCPACK, MAC sa: 0018.8b7b.2df2
Aug 8 05:30:35: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 05:30:54: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 05:30:55: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 05:30:56: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 05:30:57: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 05:36:14: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPINFORM, chaddr: 0000.0000.0000, MAC sa: 0018.8b7b.2df2
Aug 8 05:36:14: %DHCP_SNOOPING-5-DHCP_SNOOPING_UNTRUSTED_PORT: DHCP_SNOOPING drop message on untrusted port, message type: DHCPACK, MAC sa: 0018.8b7b.2df2
Aug 8 06:30:35: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 06:30:54: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 06:30:55: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 06:30:56: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 06:30:57: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 213.154.80.202
Aug 8 06:33:43: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Standby -> Active
Aug 8 06:33:43: %HSRP-5-STATECHANGE: Vlan511 Grp 14 state Standby -> Active
Aug 8 06:33:43: %HSRP-5-STATECHANGE: Vlan505 Grp 2 state Standby -> Active
Aug 8 06:33:43: %HSRP-5-STATECHANGE: Vlan105 Grp 16 state Standby -> Active
Aug 8 06:33:43: %HSRP-5-STATECHANGE: Vlan511 Grp 14 state Active -> Speak
Aug 8 06:33:43: %HSRP-5-STATECHANGE: Vlan505 Grp 2 state Active -> Speak
Aug 8 06:33:44: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Active -> Speak
Aug 8 06:33:44: %HSRP-5-STATECHANGE: Vlan105 Grp 16 state Active -> Speak
Aug 8 06:33:47: %HSRP-5-STATECHANGE: Vlan511 Grp 14 state Speak -> Standby
Aug 8 06:33:47: %HSRP-5-STATECHANGE: Vlan105 Grp 16 state Speak -> Standby
Aug 8 06:33:47: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Speak -> Standby
Aug 8 06:33:47: %HSRP-5-STATECHANGE: Vlan505 Grp 2 state Speak -> Standby
Aug 8 06:36:20: %DHCP_SNOOPING-5-DHCP_SNOOPING_MATCH_MAC_FAIL: DHCP_SNOOPING drop message because the chaddr doesn't match source mac, message type: DHCPINFORM, chaddr: 0000.0000.0000, MAC sa: 0018.8b7b.2df2
Aug 8 06:36:20: %DHCP_SNOOPING-5-DHCP_SNOOPING_UNTRUSTED_PORT: DHCP_SNOOPING drop message on untrusted port, message type: DHCPACK, MAC sa: 0018.8b7b.2df2
Aug 8 06:56:16: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Standby -> Active
Aug 8 06:56:16: %HSRP-5-STATECHANGE: Vlan101 Grp 9 state Standby -> Active
Aug 8 06:56:16: %HSRP-5-STATECHANGE: Vlan101 Grp 9 state Active -> Speak
Aug 8 06:56:16: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Active -> Speak
Aug 8 06:56:19: %HSRP-5-STATECHANGE: Vlan101 Grp 9 state Speak -> Standby
Aug 8 06:56:19: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Speak -> Standby
Aug 8 07:16:16: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Standby -> Active
Aug 8 07:16:16: %HSRP-5-STATECHANGE: Vlan509 Grp 4 state Standby -> Active
Aug 8 07:16:16: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Active -> Speak
Aug 8 07:16:16: %HSRP-5-STATECHANGE: Vlan509 Grp 4 state Active -> Speak
Aug 8 07:16:19: %HSRP-5-STATECHANGE: Vlan509 Grp 4 state Speak -> Standby
Aug 8 07:16:20: %HSRP-5-STATECHANGE: Vlan11 Grp 7 state Speak -> Standby

spanning-tree vlan 11,101,103,105,505,509,511 priority 28672
spanning-tree vlan 12,15-16,19,100,102,104,219,504,506,510,512 priority 24576

S3850-Siege-Federateur-2#sh spanning-tree br

Hello Max Fwd
Vlan Bridge ID Time Age Dly Protocol
---------------- --------------------------------- ----- --- --- --------
VLAN0001 32769 (32768, 1) 00f8.2c68.6c80 2 20 15 rstp
VLAN0011 28683 (28672, 11) 00f8.2c68.6c80 2 20 15 rstp
VLAN0012 24588 (24576, 12) 00f8.2c68.6c80 2 20 15 rstp
VLAN0015 24591 (24576, 15) 00f8.2c68.6c80 2 20 15 rstp
VLAN0016 24592 (24576, 16) 00f8.2c68.6c80 2 20 15 rstp
VLAN0019 24595 (24576, 19) 00f8.2c68.6c80 2 20 15 rstp
VLAN0100 24676 (24576, 100) 00f8.2c68.6c80 2 20 15 rstp
VLAN0101 28773 (28672, 101) 00f8.2c68.6c80 2 20 15 rstp
VLAN0102 24678 (24576, 102) 00f8.2c68.6c80 2 20 15 rstp
VLAN0103 28775 (28672, 103) 00f8.2c68.6c80 2 20 15 rstp
VLAN0104 24680 (24576, 104) 00f8.2c68.6c80 2 20 15 rstp
VLAN0105 28777 (28672, 105) 00f8.2c68.6c80 2 20 15 rstp
VLAN0219 24795 (24576, 219) 00f8.2c68.6c80 2 20 15 rstp
VLAN0504 25080 (24576, 504) 00f8.2c68.6c80 2 20 15 rstp
VLAN0505 29177 (28672, 505) 00f8.2c68.6c80 2 20 15 rstp
VLAN0506 25082 (24576, 506) 00f8.2c68.6c80 2 20 15 rstp
VLAN0509 29181 (28672, 509) 00f8.2c68.6c80 2 20 15 rstp
VLAN0510 25086 (24576, 510) 00f8.2c68.6c80 2 20 15 rstp
VLAN0511 29183 (28672, 511) 00f8.2c68.6c80 2 20 15 rstp
VLAN0512 25088 (24576, 512) 00f8.2c68.6c80 2 20 15 rstp

S3850-Siege-Federateur-2#sh standby br
P indicates configured to preempt.
|
Interface Grp Pri P State Active Standby Virtual IP
Vl11 7 50 Standby 10.20.3.1 local 10.20.3.2
Vl12 5 100 P Active local 10.20.2.1 10.20.2.2
Vl15 215 100 P Active local 10.20.1.193 10.20.1.194
Vl16 6 100 P Active local 10.20.0.230 10.20.0.231
Vl19 19 100 P Active local 192.168.1.1 192.168.1.2
Vl100 8 100 P Active local 10.20.100.201 10.20.100.200
Vl101 9 50 Standby 10.20.101.201 local 10.20.101.200
Vl102 10 100 P Active local 10.20.102.201 10.20.102.200
Vl104 12 100 P Active local 10.20.104.201 10.20.104.200
Vl105 16 50 P Standby 10.20.105.201 local 10.20.105.200
Vl219 219 50 Active local 192.168.2.1 192.168.2.2
Vl504 1 100 P Active local 10.20.3.193 10.20.3.194
Vl505 2 50 Standby 10.20.3.129 local 10.20.3.130
Vl506 3 100 P Active local 10.20.3.65 10.20.3.66
Vl509 4 50 Standby 10.20.3.33 local 10.20.3.34
Vl510 13 100 P Active local 10.20.0.33 10.20.0.34
Vl511 14 50 Standby 10.20.0.4 local 10.20.0.2
Vl512 15 100 P Active local 10.20.0.65 10.20.0.66

S3850-Siege-Federateur-2#sh run int vlan 15
Building configuration...

Current configuration : 271 bytes
!
interface Vlan15
description **** VLAN_Serv_Intern_Yeumbeul ****
ip address 10.20.1.195 255.255.255.192
ip helper-address 10.20.3.5
standby version 2
standby 215 ip 10.20.1.194
standby 215 timers 1 3
standby 215 preempt
standby 215 authentication xxxxx!

S3850-Siege-Federateur-2#sh int port-channel 10 summary

*: interface is up
IHQ: pkts in input hold queue IQD: pkts dropped from input queue
OHQ: pkts in output hold queue OQD: pkts dropped from output queue
RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)
TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)
TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
-----------------------------------------------------------------------------------------------------------------
* Port-channel10 0 0 0 0 36439000 4800 9294000 1884 0
S3850-Siege-Federateur-2#

Thank you for any help

Mark Malone · ‎08-11-2017

ah that's good it stopped , ye I think you might be facing same issue , try below there the default timers , then if you want you can shorten them bit by bit but the frequency 1 is too quick at 1 , threshold and timeout are in miliseconds while freq is seconds

so that's 5 and 5 and 60 seconds

threshold 5000 timeout 5000 frequency 60

View solution in original post

Mark Malone · ‎08-08-2017

Hi

Try increase the HSRP timers up from 1 second that's low maybe casuing an issue

The ports are definitely not flapping with STP at layer 2 or the vlan , you can see it easily with this command

sh spanning-tree detail | i ieee|occur|from|is exec

whats the topology here is it just 2 routers with 1 switch in between and HSRP ran between them ?

mamadou.barry · ‎08-08-2017

Thank Mark

I will increase timers after opened hours.

The topology is just two layer 3 switches cisco 3850 connected together directly with a port channel link and running HSRP.

I will let you know the behavor after applying new timers (I will try "standby xxx timers 3 9", is it correct?)

Mark Malone · ‎08-08-2017

HI yes I would increase it slightly bit by bit see if stabilizes 1 second is low for the hello on hsrp , if it doesn't get seen if theres other traffic impeding slightly it may cause it as it cant see its hsrp neighbour , if theres no stp or physical issue it could be that , the default is 3/10 , so you could just remove the timer command and it wil become 3/10 see if it helps you

mamadou.barry · ‎08-11-2017

Hello Mark!

I applied you advice and HSRP stopped flapping, I made it in default value.

Thank you so much

I have a other problem witch looks like same issue, it's about a link between two offices traking with ip SLA.

The frequency is 1 sec, and usualy the state goes down.

here log

Aug 11 07:10:24: %TRACK-6-STATE: 1 ip sla 1 reachability Up ->
Aug 11 07:10:29: %TRACK-6-STATE: 1 ip sla 1 reachability Down
Aug 11 07:14:54: %TRACK-6-STATE: 1 ip sla 1 reachability Up ->
Aug 11 07:14:59: %TRACK-6-STATE: 1 ip sla 1 reachability Down

Here the Config

ip sla 1
icmp-echo 10.100.10.2 source-ip 10.20.3.1
threshold 500
timeout 500
frequency 1
ip sla schedule 1 life forever start-time now

ip route 10.20.1.0 255.255.255.0 10.20.3.3 track 1

ip route 10.20.1.0 255.255.255.0 10.20.0.1 10

I would like to increase frequency and timeout.

I need your advice for that.

Mark Malone · ‎08-11-2017

ah that's good it stopped , ye I think you might be facing same issue , try below there the default timers , then if you want you can shorten them bit by bit but the frequency 1 is too quick at 1 , threshold and timeout are in miliseconds while freq is seconds

so that's 5 and 5 and 60 seconds

threshold 5000 timeout 5000 frequency 60

mamadou.barry · ‎08-23-2017

Thank you Mark

The tracking by SLA stop flapping when I increase the frequency timer to 30s.

Thank you so much

CSCO11785675 · ‎06-10-2019

It OK, Thanks bro :D ...................

paul driver · ‎08-08-2017

Hello

Add to mark comments,

It looks like you have dhcp snooping active and relay addressing applied, and it seems your dhcp server is ion the same subnet as vlan 11

You dont require relay when the dhcp server is local, so that can be removed, and you dont need dhcp snooping on the l3 core either only on the access closets

res

Paul

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul