Getting continuous HSRP state change error message

amit996699 · ‎07-07-2012

error log for this,

Jul 6 21:34:53: %HSRP-5-STATECHANGE: GigabitEthernet1/46 Grp 95 state Standby -> Active

Jul 6 21:35:52: %HSRP-5-STATECHANGE: GigabitEthernet1/46 Grp 95 state Active -> Speak

Jul 6 21:36:15: %HSRP-5-STATECHANGE: GigabitEthernet1/46 Grp 95 state Standby -> Active

Jul 6 21:36:37: %HSRP-5-STATECHANGE: GigabitEthernet1/46 Grp 95 state Active -> Speak

Jul 6 21:36:47: %HSRP-5-STATECHANGE: GigabitEthernet1/46 Grp 95 state Speak -> Standby

Jul 6 21:36:56: %HSRP-5-STATECHANGE: GigabitEthernet1/46 Grp 95 state Standby -> Active

What could be the possible reason for this , its Cisco 7609 Router pair.

Peter Paluch · ‎07-07-2012

Hello Amit,

To my best knowledge, HSRP speaker transitions from Active to Speak state if it receives a Coup message from another preempt-enabled HSRP speaker trying to become the new Active router. From the transcript you have posted, it seems that for some reason, the another router in the standby group periodically tries to become the Active router and subsequently resigns again.

It is necessary to investigate in closer detail what causes both routers to change their HSRP state. I personally start having a close look on the preempt setting of both routers and what makes each of these routers change their HSRP priority (thereby leading to Active role takeover). Especially, the stability of the tracked interfaces/objects has to be investigated.

Also, I suggest running the debug standby terse command on both routers to see the individual messages and events as they take place.

Would it be possible for you to post the HSRP configuration from both routers, including the output of the debug command?

Best regards,

Peter

amit996699 · ‎07-08-2012

Hi Peter ,

Thanks for your reply ,

Further I can share with you some configurations on these Interfaces,

For Router 1-

interface GigabitEthernet1/46

description TO_SW1_GE-0/5_100MB/S

vrf forwarding mob_r4

ip address 10.153.200.236 255.255.255.0

no ip redirects

speed 100

duplex full

storm-control broadcast level 0.10

storm-control multicast level 0.10

no cdp enable

standby 95 ip 10.153.200.235

standby 95 priority 110

standby 95 preempt

Show Standby-

Interface Grp Pri P State Active Standby Virtual IP

Gi1/46 95 110 P Active local 10.153.200.237 10.153.200.235

For Router 2 -

interface GigabitEthernet1/46

description TO_SW2_GE-0/5_100MB/S

vrf forwarding mob_r4

ip address 10.153.200.237 255.255.255.0

no ip redirects

storm-control broadcast level 0.10

storm-control multicast level 0.10

no cdp enable

standby 95 ip 10.153.200.235

standby 95 priority 80

standby 95 preempt

Show Standby -

Interface Grp Pri P State Active Standby Virtual IP

Gi1/46 95 80 P Standby 10.153.200.236 local 10.153.200.235

Sorry, not possible for me to share debug output right now.

Also, I think the issue could be at the connectivity of these links towards Switches , and right now trying to figure out that only as HSRP configurations seems to be fine.

Thanks

Amit

jimmysands73_2 · ‎07-08-2012

To add:

I believe the issue can also be caused by mis-configured timers (does not appear the case) and layer 1 issues.

Does the issue happen continously? And do you see alot of port transitions or port errors on either physical interface?

amit996699 · ‎07-08-2012

Hi Jimmi ,

Regarding the timers, i too gave a thot and checked but, that seems to be fine,

But yes , Layer 1 issues could be there and other team is working on that ,

Althogh not getting port errors from router end, but it could be there on switch end.

Thanks

Peter Paluch · ‎07-09-2012

Hello Amit,

I reviewed the information you kindly provided me with. There are a few minor things I am surprised to see:

Although preemption is configured, no interface/object tracking seems to be used. Is this observation correct, i.e. do you only use the preempt configuration to make the active router role deterministic, or is there also some tracking configuration that was not made public here?
The storm control level for multicast and broadcast are set to considerably low levels. As HSRP packets are addressed as multicasts to 224.0.0.2, they may also be affected if the storm control kicks in. Is it possible to temporarily remove these protections or set them to higher levels?

There is a very good document explaining the HSRP states and their transitions, including troubleshooting the common HSRP issues here:

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a0080094afd.shtml

Try to check it out - prehaps some of the suggestions will give you a hint. And of course, please keep us informed about anything new you'll find out in the meantime!

Best regards,

Peter

Tharak Abraham · ‎07-09-2012

Amit,

Could you hard code the speed and duplex to 100 at Router 2's Gig interface ?

And also regarding storm control of broadcasts (which also affects multicasts indirectly unless Cisco has implemented in a different way), you may have to increase the threshold. Reading it as 0.1% which may look ok under a Gig but since you have changed the speed to 100 Mbps, this value would eventually be 100Kbps.(Again depends on how the IOS is accounting for this)

Consider a host in your domain is busy sending broadcasts and since you are limiting this with storm control, the HSRP state/coup messages can take a hit which might cause this unwanted transitions or even black holeing of messages.

Note that only the STP messages are an exception to storm control and all other control packets are affected.

Ideally, Storm control should be done at the access switches.

Sandeep Choudhary · ‎07-09-2012

Hi Amit,

You can try this kind of solution,

just check this blog :

http://ccie-in-3-months.blogspot.de/2008/11/solving-routing-asymmetry-due-to-hsrp.html

Regards

please rate if it helps.

Peter Paluch · ‎07-09-2012

Hello Sandeep,

Thank you for joining in! Unfortunately, the link you have posted is focused towards a different problem with asymmetric switching/routing in HSRP-based environments. Amit's problem is concerned about a different and more basic issue: instability in HSRP roles of routers that form the standby group.

Best regards,

Peter

Hari Haran S M · ‎07-09-2012

Hi Amit,

If the configuration looks fine and there are no L1 issue, then there is a possibility of high CPU. When the CPU is high on the router, it will start dropping the HSRP packets and this causing the state changes. Could you please check whether hte CPU is fine using the command "show process cpu sort | ex 0.00" and if the issue is intermittent, please check whether, there were any cpu spikes during the time of HSRP flap. You can check the CPU history using the command "show proccess cpu history".

Regards,

Hari

amit996699 · ‎07-09-2012

Hi All,

Thanks a lot for all your analysis and suggestions , I think I will surely get through the issue now, as the issue is not with me currently, so not able to update with your querries , but as far as i get any update or resolution , I will soon share it with you all.

Thanks & Regards,

Amit Sharma