Re: HSRP Packet Issues

david-swope · ‎01-20-2014

I am at a site and have an interesting HSRP situation between two 7200 routers. These routers are running v15.0(1)M3 (AdvSecurity) IOS and configured interfaces are both G0/2 on each router.

They are laid out as shown in the attached drawing, nothing out of the ordinary there.

Configs are as follows

R1

interface GigabitEthernet0/2

description Nunya

ip address x.x.x.2 x.x.x.x.x

ip access-group 101 in

ip flow ingress

duplex auto

speed auto

media-type rj45

negotiation auto

standby 100 ip x.x.x.1

standby 100 priority 110

standby 100 preempt delay minimum 30

R2

interface GigabitEthernet0/2

description Nunya

ip address x.x.x.3 x.x.x.x

ip access-group 101 in

duplex auto

speed auto

media-type rj45

negotiation auto

standby 100 x.x.x.1

standby 100 priority 105

standby 100 preempt delay minimum 30

R1#sh standby br

P indicates configured to preempt.

|

Interface Grp Pri P State Active Standby Virtual IP

Gi0/2 100 110 P Active local unknown x.x.x.1

R2#sh standby brief

P indicates configured to preempt.

|

Interface Grp Pri P State Active Standby Virtual IP

Gi0/2 100 105 P Standby x.x.x.2 local x.x.x.1

Debug output from R1

Jan 20 2014 09:30:54.178 CST: HSRP: Gi0/2 Grp 100 Hello out x.x.x.2 Active pri 110 vIP x.x.x.1

Jan 20 2014 09:30:59.154 CST: HSRP: Gi0/2 Grp 100 Hello out x.x.x.2 Active pri 110 vIP x.x.x.1

Jan 20 2014 09:31:01.795 CST: HSRP: Gi0/2 Grp 100 Hello out x.x.x.2 Active pri 110 vIP x.x.x.1

Jan 20 2014 09:31:04.723 CST: HSRP: Gi0/2 Grp 100 Hello out x.x.x.2 Active pri 110 vIP x.x.x.1

Jan 20 2014 09:31:07.155 CST: HSRP: Gi0/2 Grp 100 Hello out x.x.x.2 Active pri 110 vIP x.x.x.1

Debug output from R2

Jan 20 2014 09:31:23.447 CST: HSRP: Gi0/2 Grp 100 Hello out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:23.459 CST: HSRP: Gi0/2 Grp 100 Hello in x.x.x.2 Active pri 110 vIP x.x.x.1

Jan 20 2014 09:31:25.879 CST: HSRP: Gi0/2 Grp 100 Hello out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:25.971 CST: HSRP: Gi0/2 Grp 100 Hello in x.x.x.2 Active pri 110 vIP x.x.x.1

Jan 20 2014 09:31:28.451 CST: HSRP: Gi0/2 Grp 100 Hello in x.x.x.2 Active pri 110 vIP x.x.x.1

Jan 20 2014 09:31:28.455 CST: HSRP: Gi0/2 Grp 100 Hello out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:29.127 CST: HSRP: Gi0/2 Interface adv out, Passive, active 0 passive 1

Here is what I have done. I have specifically added a permit statement to ACL 101 on R1 for 224.0.0.2 port 1985, it still does nothing. I then added the same to R2 just to see the hit count increase, it did of course although the ACL is not needed, more of a visual way for me to track it. On the end of each ACL 101 there is a "permit ip any any"

I made sure both sides had appropriate priorities, preempt statements. The routers have been rebooted and the next thing I could remove HSRP all together from G0/2 on R1 and add it back. It's simply an odd issue, is it buggy IOS perhaps? Switches are configured the same, can find nothing wrong there.

Richard Burts · ‎01-20-2014

The debug is pretty clear that R1 sees outbound HSRP but no inbound. My first question would be what does CDP show on each router? Does R1 see R2 as a neighbor on G0/2? My second question would be whether the routers can traceroute to each other and if so is the response coming back from G0/2?

HTH

Rick

HTH

Rick

Jon Marshall · ‎01-20-2014

David

Apologies for interrupting the thread.

Rick

Could i ask you a favour. I have been involved in a thread where i seem to be going round in circles and cannot understand exactly how things are working.

If possible could you have a look at it and see if it makes sense to you because it doesn't to me but it could be my lack of understanding -

https://supportforums.cisco.com/thread/2262246?tstart=0

Many thanks.

Jon

david-swope · ‎01-20-2014

Sorry, should have put that data in the first post

CDP shows

R1#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

S - Switch, H - Host, I - IGMP, r - Repeater

Device ID Local Intrfce Holdtme Capability Platform Port ID

Sw1

Gig 0/2 168 S I WS-C2960G Gig 0/15

R2#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

S - Switch, H - Host, I - IGMP, r - Repeater

Device ID Local Intrfce Holdtme Capability Platform Port ID

Sw2

Gig 0/2 150 S I WS-C2960G Gig 0/15

R1#traceroute x.x.x.3

Type escape sequence to abort.

Tracing the route to x.x.x.3

1 * *

x.x.x.3 0 msec

R1#

R1#ping x.x.x.3

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.x.x.3, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

R1#

R2#traceroute x.x.x.2

Type escape sequence to abort.

Tracing the route to x.x.x.2

1 x.x.x.2 0 msec * 0 msec

R2#

R2#ping x.x.x.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.x.x.2, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

There is obviously an issue with pinging the vIP as shown here

R1#ping x.x.x.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.x.x.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

R1#

R2#ping x.x.x.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.x.x.1, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

R2#

Rolf Fischer · ‎01-20-2014

David,

could you do one more ping on R2:

ping 224.0.0.2 source gi0/2

just to see if R1 responds?

Regards

Rolf

david-swope · ‎01-20-2014

R1#ping 224.0.0.2 source g0/2

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 224.0.0.2, timeout is 2 seconds:

Packet sent with a source address of x.x.x.2

Reply to request 0 from x.x.x.3, 1 ms

R1#

R2#ping 224.0.0.2 source g0/2

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 224.0.0.2, timeout is 2 seconds:

Packet sent with a source address of x.x.x.3

Richard Burts · ‎01-20-2014

That certainly is interesting and suggests a one way issue with multi cast. Your earlier test shows that we have good two way communication for unicast. It might be interesting to see the output of show ip interface g0/2 from both routers.

I also wonder if there might be something in the configuration of the switches that might cause this.

HTH

Rick

HTH

Rick

david-swope · ‎01-20-2014

Yeah, an interesting issue to say the least, it's going to be something i've over looked. I can feel it. ha

R1#

sh ip interface g0/2

GigabitEthernet0/2 is up, line protocol is up

Internet address is x.x.x.2/x

Broadcast address is 255.255.255.255

Address determined by non-volatile memory

MTU is 1500 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.2

Outgoing access list is not set

Inbound access list is 101

Proxy ARP is enabled

Local Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is disabled

IP CEF switching is enabled

IP CEF switching turbo vector

IP CEF turbo switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, CEF

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Policy routing is disabled

Network address translation is disabled

BGP Policy Mapping is disabled

Input features: Ingress-NetFlow, Access List, MCI Check

Output features: Post-Ingress-NetFlow

WCCP Redirect outbound is disabled

WCCP Redirect inbound is disabled

WCCP Redirect exclude is disabled

R2#sh ip int g0/2

GigabitEthernet0/2 is up, line protocol is up

Internet address is x.x.x.3/x

Broadcast address is 255.255.255.255

Address determined by non-volatile memory

MTU is 1500 bytes

Helper address is not set

Directed broadcast forwarding is disabled

Multicast reserved groups joined: 224.0.0.2

Outgoing access list is not set

Inbound access list is 101

Proxy ARP is enabled

Local Proxy ARP is disabled

Security level is default

Split horizon is enabled

ICMP redirects are always sent

ICMP unreachables are always sent

ICMP mask replies are never sent

IP fast switching is enabled

IP fast switching on the same interface is disabled

IP Flow switching is disabled

IP CEF switching is enabled

IP CEF switching turbo vector

IP CEF turbo switching turbo vector

IP multicast fast switching is enabled

IP multicast distributed fast switching is disabled

IP route-cache flags are Fast, CEF

Router Discovery is disabled

IP output packet accounting is disabled

IP access violation accounting is disabled

TCP/IP header compression is disabled

RTP/IP header compression is disabled

Policy routing is disabled

Network address translation is disabled

BGP Policy Mapping is disabled

Input features: Access List, MCI Check

WCCP Redirect outbound is disabled

WCCP Redirect inbound is disabled

WCCP Redirect exclude is disabled

Richard Burts · ‎01-20-2014

David

Thanks for this output. But what I asked for was show ip interface and not just show interface.

HTH

Rick

HTH

Rick

david-swope · ‎01-21-2014

Oops, there you go..edited post above

Richard Burts · ‎01-21-2014

David

Thanks for the updated output. I had hoped that it would have some insight into the issue. But other than demonstrating that both have "Multicast reserved groups joined: 224.0.0.2" it does not have much clue (at least that I can detect). One more request: would you post the access-list 101 from both routers?

HTH

Rick

HTH

Rick

david-swope · ‎01-22-2014

I agree, I see nothing at this point and almost at a loss..here are the ACL's

R1#sh access-lists

Standard IP access list 1

10 permit x.x.x.x, wildcard bits 0.0.0.255

Standard IP access list 2

10 permit x.x.x.x, wildcard bits 0.0.0.255

Extended IP access list 101

10 permit tcp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq 22 (7498 matches)

20 permit udp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq ntp

30 permit udp host x.x.x.x x. x.x.x 0.0.1.255 eq ntp (2095 matches)

40 permit udp x.x.x.x 0.0.1.255 host x.x.x.x eq ntp (2265 matches)

50 permit udp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq snmp (475234 matches)

70 deny tcp any host x.x.x.x eq 22 (43 matches)

80 deny udp any host x.x.x.x eq snmp (6 matches)

100 deny tcp any host x.x.x.x eq 22 (92 matches)

110 deny udp any host x.x.x.x eq ntp

120 deny udp any host x.x.x.1 eq snmp (3 matches)

130 deny tcp any host x.x.x.x.2 eq 22 (204 matches)

140 deny udp any host x.x.x.x.2 eq ntp (1 match)

150 deny udp any host x.x.x.2 eq snmp (3 matches)

160 permit ip any any (732299596 matches)

R2#sh access-lists

Standard IP access list 1

10 permit x.x.x.x, wildcard bits 0.0.0.255 (1 match)

Standard IP access list 2

10 permit x.x.x.x, wildcard bits 0.0.0.255

Extended IP access list 101

10 permit tcp x.x.x.x 0.0.1.255 x.x.x.x0.0.1.255 eq 22 (10586 matches)

20 permit udp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq ntp

30 permit udp host x.x.x.x x.x.x.x 0.0.1.255 eq ntp (2 matches)

40 permit udp x.x.x.x 0.0.1.255 host x.x.x.x eq ntp (31 matches)

50 permit udp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq snmp (494446 matches)

60 permit udp host x.x.x.2 host 224.0.0.2 eq 1985 (97910 matches)

70 deny tcp any host x.x.x.x eq 22

80 deny udp any host x.x.x.x eq ntp

90 deny udp any host x.x.x.x eq snmp

100 deny tcp any host x.x.x.1 eq 22

110 deny udp any host x.x.x.1 eq ntp

120 deny udp any host x.x.x.1 eq snmp

130 deny tcp any host x.x.x.3 eq 22 (88 matches)

140 deny udp any host x.x.x.3 eq ntp

150 deny udp any host x.x.x.3 eq snmp (3 matches)

160 permit ip any any (90446403 matches)

Highlighted the ACL I put in place to get a visual on the hit count for the multicast traffic for HSRP, added and removed from R1 with no hits of course

Jon Marshall · ‎01-22-2014

David

Have you checked your switch configurations eg. specifically do you have any port acls applied to any of the interfaces that are part of the path between the routers.

Also worth checking if the switches are using VACLs which could be blocking multicast one way.

Jon

david-swope · ‎01-22-2014

Yeah, I have checked the switches..there are several ACL's on the 2nd switch to which R2 is connected but nothing affecting this issue

Sw2#sh ip int g0/9

GigabitEthernet0/9 is up, line protocol is up

Inbound access list is not set

access-list 103 deny tcp host x.x.x.x eq 1723 any

access-list 103 permit ip any any

access-list 178 deny udp any eq ntp host x.x.x.x

access-list 178 permit ip any any

Connection to R1

interface GigabitEthernet0/9

description R1

switchport access vlan x

switchport mode access

spanning-tree portfast

spanning-tree bpduguard enable

Connection to R2

interface GigabitEthernet0/9

description R2

switchport access vlan x

switchport mode access

spanning-tree portfast

spanning-tree bpduguard enable

I also want to mention that there have been some ARP issues with these routers recently, wondering if this IOS is buggy? (C7200P-ADVSECURITYK9-M) Version 15.0(1)M3

Richard Burts · ‎01-22-2014

David

Thanks for the additional information. I am wondering about the possibility that something on some switch is causing the issue. Perhaps some CGMP/IGMP config? I am wondering if we can try some other multicast traffic and see if it is impacted. Perhaps something like trying to run EIGRP or OSPF on these two router interfaces? We do not need to advertise anything, but it would be interesting to see if R1 receives the multicast hello from R2.

HTH

Rick

HTH

Rick