HSRP Packet Issues

[david-swope]

I am at a site and have an interesting HSRP situation between two 7200 routers. These routers are running v15.0(1)M3 (AdvSecurity) IOS and configured interfaces are both G0/2 on each router.

They are laid out as shown in the attached drawing, nothing out of the ordinary there.

Configs are as follows

R1

interface GigabitEthernet0/2

description Nunya

ip address x.x.x.2 x.x.x.x.x

ip access-group 101 in

ip flow ingress

duplex auto

speed auto

media-type rj45

negotiation auto

standby 100 ip x.x.x.1

standby 100 priority 110

standby 100 preempt delay minimum 30

R2

interface GigabitEthernet0/2

description Nunya

ip address x.x.x.3 x.x.x.x

ip access-group 101 in

duplex auto

speed auto

media-type rj45

negotiation auto

standby 100 x.x.x.1

standby 100 priority 105

standby 100 preempt delay minimum 30

R1#sh standby br

                     P indicates configured to preempt.

                     |

Interface   Grp  Pri   P   State      Active       Standby         Virtual IP

Gi0/2       100  110  P   Active     local          unknown        x.x.x.1

R2#sh standby brief

                     P indicates configured to preempt.

                     |

Interface   Grp  Pri   P     State      Active          Standby         Virtual IP

Gi0/2       100  105  P    Standby   x.x.x.2         local              x.x.x.1

Debug output from R1

Jan 20 2014 09:30:54.178 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:30:59.154 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:01.795 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:04.723 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:07.155 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Debug output from R2

Jan 20 2014 09:31:23.447 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:23.459 CST: HSRP: Gi0/2 Grp 100 Hello  in  x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:25.879 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:25.971 CST: HSRP: Gi0/2 Grp 100 Hello  in  x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:28.451 CST: HSRP: Gi0/2 Grp 100 Hello  in  x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:28.455 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:29.127 CST: HSRP: Gi0/2 Interface adv out, Passive, active 0 passive 1

Here is what I have done. I have specifically added a permit statement to ACL 101 on R1 for 224.0.0.2 port 1985, it still does nothing. I then added the same to R2 just to see the hit count increase, it did of course although the ACL is not needed, more of a visual way for me to track it. On the end of each ACL 101 there is a "permit ip any any"

I made sure both sides had appropriate priorities, preempt statements. The routers have been rebooted and the next thing I could remove HSRP all together from G0/2 on R1 and add it back. It's simply an odd issue, is it buggy IOS perhaps?  Switches are configured the same, can find nothing wrong there.

[david-swope]

Excellent idea actually..so this is obviously a multicast issue as shown below

R1

R1#

Jan 22 2014 09:34:09.864 CST: EIGRP: Neighbor(x.x.x.3) not yet found

R1#

Jan 22 2014 09:34:14.864 CST: EIGRP: Neighbor(x.x.x.3) not yet found

R1#

Jan 22 2014 09:34:19.864 CST: EIGRP: Neighbor(x.x.x.3) not yet found

R2

Jan 22 2014 09:34:00.354 CST: EIGRP: New peer x.x.x.2

R2#

Jan 22 2014 09:34:00.354 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.2 (GigabitEthernet0/2) is up: new adjacency

R2#sh ip eigrp neighbors

EIGRP-IPv4 Neighbors for AS(1)

H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq

                                            (sec)         (ms)       Cnt Num

0   x.x.x.2            Gi0/2             13 00:00:38    1  5000  1  0

[Jon Marshall]

David  / Rick

Great idea Rick. I think this has to be an acl issue because IGMP snooping certainly does not filter link local multicast addresses.

It's actually very hard to filter that specific multicast range

Whether CGMP does or doesn't i can't say but it would have to be quite an old switch to be running that.

Jon

[Richard Burts]

The drawing in the original post shows 2 switches connecting the 2 routers. So I believe that we need to look more closely at the switches to see if one of them is the cause of this strange behavior.

HTH

Rick

[david-swope]

These routers and switches are "External", so the two routers are connected to access ports on the two WS-C2960G-24TC-L via G0/9, switches are running 12.2(46)SE (LANBASE)

VLAN 301 is the access vlan on each port. CDP neighbor shows the other switch, guest router, vpn etc. Etherchannel is used between the two switches

SW1

interface Port-channel1

switchport mode trunk

interface GigabitEthernet0/22

switchport mode trunk

channel-group 1 mode active

end

interface GigabitEthernet0/23

switchport mode trunk

channel-group 1 mode active

SW1#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

SW2

                 Gig 0/23          126           S I      WS-C2960G Gig 0/23

SW2

                 Gig 0/22          129           S I      WS-C2960G Gig 0/22

R1

                 Gig 0/9           173            R       7204VXR   Gig 0/2

SW2

interface Port-channel1

switchport mode trunk

interface GigabitEthernet0/22

switchport mode trunk

channel-group 1 mode active

interface GigabitEthernet0/23

switchport mode trunk

channel-group 1 mode active

SW2#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

SW1

                 Gig 0/23          159           S I      WS-C2960G Gig 0/23

SW1

                 Gig 0/22          159           S I      WS-C2960G Gig 0/22

R2

                 Gig 0/9           175            R       7204VXR   Gig 0/2

[david-swope]

Interesting enough, I can now see HSRP is up and running fine. I did nothing that is not posted here, I simply logged into the routers today to pick up where I left off and was shocked at the out of sh standby br, more like a "wtf?" moment.

Can anyone say odd? Wow!

R1#sh standby br

                     P indicates configured to preempt.

                     |

Interface   Grp  Pri P State   Active          Standby         Virtual IP

Gi0/2       100  110 P Active  local           x.x.x.3            x.x.x.1

R2#sh standby br

                     P indicates configured to preempt.

                     |

Interface   Grp   Pri   P      State           Active          Standby         Virtual IP

Gi0/2       100   105   P     Standby       x.x.x.2         local               x.x.x.1

R2 can now ping multicast as well

R2#ping 224.0.0.2

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 224.0.0.2, timeout is 2 seconds:

Reply to request 0 from x.x.x.2, 1 ms

ACL's show hits now

R1

60 permit udp host x.x.x.3 host 224.0.0.2 eq 1985 (142878 matches)

R2

60 permit udp host x.x.x.2 host 224.0.0.2 eq 1985 (260476 matches)

If you remember, I added these ACL's to get a visual of the hit counts going up as the traffic came in. Had previously added it to R1 but no hits ever came across. Removed the ACL but added it back last week and left it, this one has me folks. Something buggy for sure.

[Richard Burts]

David

That surely is unexpected but is good news 

Is there any chance that the router rebooted, or interface reset, or anything like that which might have re-initialized the multicast processing?

HTH

Rick

[david-swope]

Nope, sure I shut the ports down on the switch and brought them up during the troubleshooting last week but this did not show any immediate resolution to this issue.

Also, on this R2 there is an ARP issue. A few entries have had previously been manually configured. I don't see the x.x.x.1 entry in the table so with that said, R2 cannot ping the vIP address x.x.x.1

[Richard Burts]

David

Probably the router not being able to ping the virtual IP is not much of a problem (in fact I remember when HSRP was a fairly new feature it was normal behavior that the router could not ping the virtual IP - but that changed many releases ago). So I believe that this is one more indicator that there is something quite buggy about this version of code and HSRP.

HTH

Rick

[david-swope]

Agreed! I'm filing this one under "buggy IOS". Thanks for all the insight..this was an interesting one for sure.