3750G HSRP issues

[dbuckley77]

We have two 3750Gs setup with HSRP that we're seeing the below message on.  Pretty sure it's because HSRP messages are not getting shared correctly between the two but don't know why.

105331: May 11 16:29:48: %HSRP-6-STATECHANGE: Vlan114 Grp 1 state Standby -> Active
105332: 2y30w: HSRP: Vl114 Grp 1 Redundancy "hsrp-Vl114-1" state Standby -> Active
105333: 2y30w: HSRP: Vl114 Grp 1 Redundancy group hsrp-Vl114-1 state Active -> Active
105334: 2y30w: HSRP: Vl114 Grp 1 Redundancy group hsrp-Vl114-1 state Active -> Active
105335: 2y30w: HSRP: Vl114 API active virtual address 10.100.114.1 found
105336: 2y30w: HSRP: Vl114 Grp 1 Hello in 10.100.x.x Active pri 150 vIP 10.100.x.x
105337: 2y30w: HSRP: Vl114 Grp 1 Active router is 10.100.x.x, was local
105338: 2y30w: HSRP: Vl114 Grp 1 Active: g/Hello rcvd from higher pri Active router (150/10.100.x.x)
105339: 2y30w: HSRP: Vl114 Grp 1 Active -> Speak
105340: May 11 16:30:08: %HSRP-6-STATECHANGE: Vlan114 Grp 1 state Active -> Speak
105341: 2y30w: HSRP: Vl114 Grp 1 Redundancy "hsrp-Vl114-1" state Active -> Speak

[MHM Cisco World]

can you share the log before this Peer be from standby to active ?

[Reza Sharifi]

Is vlan 114 part of the trunk between the 2 switches? Is there a layer-2 connection between the switches?

Can you post relevant configs from both switches?.

HTH

[dbuckley77]

There is a trunk directly between the two switches passing several vlans.  There are also four ports on each switch setup as access vlan that connect to each other through a series of non-Cisco switches not running spanning-tree.  So basically port 5 creates a ring between the two switches  on vlan 114.  So does port 6 and port 7 etc...multiple rings on vlan 114 between the two switches.

[MHM Cisco World]

show spanning tree 
check if trunk VLAN is forward or BLK, 
the spanning tree for VLAN 114 in both SW must forward not BLK.

[SinghRaminder]

Provide Relevant HSRP configuration from both switches and the port configuration on the trunks as well please

[dbuckley77]

These are the configs from both switches:

config

[David Ruess]

Hello,

The configuration looks fine from initial read. As @paul driver mentioned the timestamp...it looks like it happened 2 years and 30 weeks ago if we read that right? Is this a problem you are having currently. Do you have more recent logs?

Can you enter the command:

show standby or show standby brief?

-David

[dbuckley77]

Here is a current log from switch 2 and sh standy from both switches:

[David Ruess]

Thank you for the info. Can you elaborate on what specific issue you are having? The changes/messages you got were a long time ago and even according to the output you just provided the last change was 2-3 years ago depending on the VLAN interface.

HSRP looks to be functioning and acting as normal. If you are having a specific issues can you please provide more detail.

There may be an issue with logging if you still have messages from years ago stuck in your log output and no new ones are being generated.

-David

[MHM Cisco World]

not all VLAN face issue some have 
150020 state changes, last state change 00:30:02

[dbuckley77]

We have a monitoring system losing ping to the first switch routinely even though it's not actually losing connectivity so I looked in the logs and though the output showed something wrong with HSRP.

[David Ruess]

Thank you @MHM Cisco World for pointing the change out as more recent. I missed that. 

@dbuckley77 - you said you have a monitoring system pinging the first switch. Does this mean you're not pinging the HSRP IP and you are pinging the actual SVI IP? If that's the case it may not be an HSRP issue, but a connectivity issue.

-David

[Richard Burts · Accepted solution]

First let me address the time stamp question. David says "it looks like it happened 2 years and 30 weeks ago" Actually that is not how long ago it happened but is how long this switch has been "up" when the event occurred. One of the options in logging messages is to use the up time as the time stamp and the usual option is to use current time. I believe that up time is the default if the device does not have authoritative time. I am puzzled at the mixture in these log messages of time stamps as up time and time stamp as current time. I don't know if that might point at something that would be significant.

Actually I just figured out the inconsistency. Check the configuration of timestamps

service timestamps debug uptime
service timestamps log datetime localtime

Some of these messages are generated by debug and are using uptime while other messages are generated by syslog.

I note that all of these messages refer to vlan 114. There are no messages about vlan 112 or 113. Is there something about vlan 114 that is different?

There is another inconsistency that I notice. There are messages like this

113257: 2y30w: HSRP: Vl114 API active virtual address 10.100.114.231 not found

This address is not in the config that was posted. Where did it come from?

Recognizing that this message was generated by debug I guess the question is really where did debug get this from? It is still not consistent with the config.

I would like to focus on this group of messages

113287: 2y30w: HSRP: Vl114 Grp 1 Standby: c/Active timer expired (10.100.114.2)
113288: 2y30w: HSRP: Vl114 Grp 1 Active router is local, was 10.100.114.2
113289: 2y30w: HSRP: Vl114 Grp 1 Standby router is unknown, was local
113290: 2y30w: HSRP: Vl114 Grp 1 Standby -> Active

The first of these messages seems to indicate that there has been a normal/successful HSRP relationship between the switches and now switch 2 is no longe receiving HSRP messages from switch 1. The other 3 messages describe a typical failover where this switch (now the active member) is not in contact with the peer switch.

The message that follows them is a syslog message with a much more helpful timestamp

113291: May 12 12:29:03: %HSRP-6-STATECHANGE: Vlan114 Grp 1 state Standby -> Active

Skipping over a series of debug messages the next syslog message is this

113299: May 12 12:29:25: %HSRP-6-STATECHANGE: Vlan114 Grp 1 state Active -> Speak

This means that at 12:29:25 switch2 received a HSRP message from switch 1. Switch 1 with higher priority and preempt configured is taking over as the active router. And switch2 is going back to standby.

If we focus on the syslog messages we see that there was a login at May 12 12:22:39. The context suggests that HSRP was operating successfully with switch2 as standby.

At May 12 12:29:03 there was a failover based on not receiving messages from switch1. (note there are no log messages about any interface state changes).

At May 12 12:29:25 switch2 has received a HSRP message from switch1 and switch1 is resuming the role of active.

I would suggest that at this point switch2 appears to be normal and whatever issue there might be is to be found on switch1.

[dbuckley77]

The issue has been found and was on switch one.  The issue was that we had changed the IP address of one of our servers but did not update the ACL in the switch.  I had thought it was an HSRP issue because I was unfamiliar with the entries int he log and they looked ominous so I consulted a Cisco doc that led me to believe that HSRP was the issue.