HSRP Not Working When Routers Connected to Switch

[jonathanw84]

I have kind of an odd one that TAC hasn't been able to really figure out yet and I am hoping someone can point me in the right direction. 

We deployed a new Catalyst 9300 switch stack and two new Catalyst 8200 routers in one of our offices. This is the same architecture and same configuration that we have in other offices and it is working just fine there.

The 8200 routers both have connections to the 9300 switch stack via a trunk port, both on the same VLAN. The routers can ping each other, so layer 2 adjacency is fine. But they cannot establish HSRP. I have debugged HSRP on both routers and we see the routers sending the hellos but getting no response. If we remove the switch and connect the interfaces on the router directly, HSRP establishes as expected. The only difference between this office and others that are working are the software versions, which perhaps could indicate a bug on the version that is not working. But TAC seems to be focused on the VRF configuration on the routers which shouldn't matter since that is only locally significant and as mentioned, they have layer 2 adjacency. Configuration is below and I've also attached a diagram for reference.

Any ideas or suggestions would be helpful.

Thanks!

-----

BOS-IR01-C8200 - Version 17.6.3a

interface GigabitEthernet0/1/0.901
description BOS-AS01-C9300-I-B1F5R1 1/0/1
encapsulation dot1Q 901
ip vrf forwarding EXT1
ip address 10.1.1.252 255.255.255.0
ip nat inside
standby 40 ip 10.1.1.254
standby 40 priority 200
standby 40 preempt
end

-----

BOS-IR02-C8200 - Version 17.6.1a

interface GigabitEthernet0/1/0.901
description BOS-AS01-C9300-I-B1F5R1 3/0/1
encapsulation dot1Q 901
ip vrf forwarding EXT2
ip address 10.1.1.253 255.255.255.0
ip nat inside
standby 40 ip 10.1.1.254
standby 40 priority 150
standby 40 preempt
end

-----

BOS-AS01-C9300 - Version 17.6.5

interface GigabitEthernet1/0/1
description BOS-IR01-C8200-I-B1F5R1
switchport trunk allowed vlan 901
switchport mode trunk
!
interface GigabitEthernet3/0/1
description BOS-IR02-C8200-I-B1F5R1
switchport trunk allowed vlan 901
switchport mode trunk

[balaji.bandi]

i do not see any issue on the simple config as you mentioned.

couple of things i need to ask :

1. do you have vlan 901 created ? (post show vlan brief)

2. i suggest to use Lower VLAN number for testing ?

3. what config you have gig0/1/0 (main interface ) do you have any other sub-interfaces ?

4. both routers show as active in HSRP ?

5. also try remove IP NAT command and test it

last it may be bug

[jonathanw84]

Hello BB,

1. Yep, VLAN 901 is created on the switch.
2. I could try that but the same VLAN # is working elsewhere.
3. The config on gig0/1/0 on both routers is as follows:

interface GigabitEthernet0/1/0
description EXTERNAL
no ip address
negotiation auto
end

There are no other sub-interfaces.

4. Both routers show as active in HSRP and standby is unknown.
5. I can try that but it's also working just fine in other offices with the nat command in place. 

Thanks! 

[balaji.bandi]

2. I could try that but the same VLAN # is working elsewhere.

else where same config same routers ? (cat 8200 ?)

as per debug it sending hello not receiving that mean that L2 may have issue, or router may have bug

plug PCs on both porsts and test ping back to back ? is this works ?

[Giuseppe Larosa]

Hello @jonathanw84 ,

>> The routers can ping each other, so layer 2 adjacency is fine. But they cannot establish HSRP

This makes me think of issues with propagation of multicast frames on the Catalyst on Vlan 901.

the multicast address used by HSRP is a link local 224.0.0.2 all routers on link and it should not be processed limited by IGMP snooping.

A suggestion is to to verify if you have IGMP snooping enabled globally on the switch stack and to check if it is enabled for Vlan 901 and to disable it.

Another possible action is to put both ports on the same member switch to see it it changes something

Hope to help

Giuseppe

[David Ruess]

Hello,

Based on the information given and the fact it works when the routers are connected to eachother the problem is likely with the switch. I didn't see anything wrong with the configuration either.

2 things:

1. Is the switch stack actually stacked...as in can the switches see eachother as all part of the stack. ( I would assume yes but still verification is good)

2. Do you see the HSRP MAC address being learned on both switchports connecting to each router. They should be advertising the same MAC address and if both are claiming to be active then they will both advertise the Virtual MAC address.

0000.0c07.acXX

where XX is the group number. You should be able to verify with the show mac address command on the switch.

You can also try a multicast ping. From one router ping the  224.0.0.2 (the all routers address) and see which IPs respond. If the other router interface IP is in that list then you know the HSRP messages are making it across the switch. If they do then it is a possible bug.

Last thing you can try is have the bare HSRP configs put in place and take off NAT and VRF to see if just that works by itself.

-David

[Joseph W. Doherty]

For laughs, two things you might also try.  1) allow all VLANs on trunk ports (for those router ports).  2) try HSRPv2.

[MHM Cisco World]

Try first connect two router to same SW (nit different SW of stack).

If hsrp start work then upgrade ios xe.

[jonathanw84]

Hey Everyone...

Thanks for all of the suggetsions. I started toying around with it, and upon remove the "ip nat inside" command from BOS-IR02, shutting the interface and bringing it back up, HSRP is now established as expected. I then add back the "ip nat inside" statement and it continues to work as expected. There is definitely some buggy behavior with the router(s), so I'm going to upgrade both of them to 17.6.5 this evening.

The other office that is working as expected with the same configuration is running 17.6.3 on both routers - I think the engineer who shipped this out forgot to commit when upgrading them to 17.6.3 to it fell back to 17.6.1. Either way, definitely something odd going on so upgrading the software seems like the next logical step.

I appreciate all of the suggestions from you all. Take care! 

[Joseph W. Doherty]

BTW, seeing NAT on the interface had caught my interest although not specifically with HSRP not seeing the other routers but in context wondering whether your NAT config was a stateful NAT.  (NB: I'm looking at some old documentation that notes SNAT feature had gone end-of-life some time way back.)

[balaji.bandi]

Sure that what i was suspected NAT command, but may be bug also later its working as expected, try 17.9.3 if you can see if that fix the issue.

other intresting thing suggest @paul driver is an option if you can try, i am sure will try see if that fixes.

[jonathanw84]

Hi All,

Update on this. I upgraded both routers to 17.06.05 and this seems to have fixed the issue. HSRP adjacency establishes instantly on both routers as expected. I made no other changes to the configuration, so it looks to be something odd with version 17.6.3a or 17.6.1a. Either way, I am good now.

Thanks again for all of the help! 

[balaji.bandi]

Glad to hear - that what i suspect :

last it may be bug