3750x HSRP issue

[7layer]

Dear all,

I'm having a vired issue with my HSRP config, because the nodes cannot see each other properly.

When I noticed the issue first I thought the trouble lies at layer2 with the SG300 switch where the two 3750x connected to.

But we have moved the second switch uplink to the first switch and same trouble, they still cannot communicate via the same VLAN for some reason. In this test the trunk had all VLAN enabled and still no luck.

I have this setup at a different location and it does work fine.

One thing is different though which is here there is an ACL applied for the VLAN that you cannot see in this config but other VLAN's have few ACL's.

Those VLAN's where ACL applied the policy has been added for HSRP to work: (permit ip any host 224.0.0.2) 

But as you can see in this VLAN 100 there is no ACL and still the HSRP cannot see each other.

The switch one has both IP and switch two has first standy and second active.

Also other interesting issue is that if I use the second switch as gateway on pingplotter I can see that the the first hop is flipping. So 8.8.8.8 trace the first gw is 172.20.0.11 and in the next 3 sec it changes to 172.20.0.21 and vica versa flipping.

Obviosly the HSRP would be the HA solution here and it is not working as it should.

I fully run out of ideas so I wold much appreaciate if anyone would advise what to do here.

Only different between the two switches is the IOS.

First switch: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(58r)SE1, RELEASE SOFTWARE (fc1)

Second switch: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(53r)SE2, RELEASE SOFTWARE (fc1)

But I would be really surprised if this would be the issue, because I have managed to make work together an 12+ years old 3550 with a brand new 3750x on HSRP V1 when I was upgrading core switches and they worked well with 10+ years different HW and different SW. Here there is an almost similar 3750x only few release betwheen them.

Here are the two switch configs:

SW1:

interface Vlan100
ip address 172.28.0.11 255.255.255.0
no ip mroute-cache
standby 1 ip 172.28.0.10
standby 1 priority 110
standby 1 preempt
standby 1 track 1 decrement 10
standby 2 ip 172.28.0.20
standby 2 priority 90
standby 2 preempt
standby 2 track 1 decrement 10
end

SW1#sh standby brief
P indicates configured to preempt.
Vl100 1 110 P Active local unknown 172.28.0.10
Vl100 2 90 P Active local unknown 172.28.0.20

SW1#sh standby vlan 100
Vlan100 - Group 1
State is Active
2 state changes, last state change 46w0d
Virtual IP address is 172.28.0.10
Active virtual MAC address is 0000.0c07.ac01
Local virtual MAC address is 0000.0c07.ac01 (v1 default)
Hello time 3 sec, hold time 10 sec
Next hello sent in 1.104 secs
Preemption enabled
Active router is local
Standby router is unknown
Priority 110 (configured 110)
Track object 1 state Up decrement 10
Group name is "hsrp-Vl100-1" (default)
Vlan100 - Group 2
State is Active
2 state changes, last state change 46w0d
Virtual IP address is 172.28.0.20
Active virtual MAC address is 0000.0c07.ac02
Local virtual MAC address is 0000.0c07.ac02 (v1 default)
Hello time 3 sec, hold time 10 sec
Next hello sent in 2.464 secs
Preemption enabled
Active router is local
Standby router is unknown
Priority 90 (configured 90)
Track object 1 state Up decrement 10
Group name is "hsrp-Vl100-2" (default)
SW1#

########
SW2:
interface Vlan100
ip address 172.28.0.21 255.255.255.0
standby 1 ip 172.28.0.10
standby 1 priority 90
standby 1 preempt
standby 1 track 1 decrement 10
standby 2 ip 172.28.0.20
standby 2 priority 110
standby 2 preempt
standby 2 track 1 decrement 10
end

Vl100 1 90 P Standby 172.28.0.11 local 172.28.0.10
Vl100 2 110 P Active local unknown 172.28.0.20

SW2#sh standby vlan 100
Vlan100 - Group 1
State is Standby
9 state changes, last state change 00:54:02
Virtual IP address is 172.28.0.10
Active virtual MAC address is 0000.0c07.ac01
Local virtual MAC address is 0000.0c07.ac01 (v1 default)
Hello time 3 sec, hold time 10 sec
Next hello sent in 1.328 secs
Preemption enabled
Active router is 172.28.0.11, priority 110 (expires in 7.840 sec)
Standby router is local
Priority 90 (configured 90)
Track object 1 state Up decrement 10
Group name is "hsrp-Vl100-1" (default)
Vlan100 - Group 2
State is Active
9 state changes, last state change 00:54:24
Virtual IP address is 172.28.0.20
Active virtual MAC address is 0000.0c07.ac02
Local virtual MAC address is 0000.0c07.ac02 (v1 default)
Hello time 3 sec, hold time 10 sec
Next hello sent in 2.656 secs
Preemption enabled
Active router is local
Standby router is unknown
Priority 110 (configured 110)
Track object 1 state Up decrement 10
Group name is "hsrp-Vl100-2" (default)
SW2#

[paul driver]

Hello
TBH there isnt much that can negate this communication, as long as the HSRP versions are the same, you are not denying udp/mc and you have reachability to each node it should work.

I haven't read all the posts on here but i assume you have you already confirmed you can ping each HSRP node to confirm reachability?

Can you also verify what direction is the acl applied to the svi's, Post the acl or even remove it for testing, Also how are the switch's are physically connected,

[7layer]

##

I haven't read all the posts on here but i assume you have you already confirmed you can ping each HSRP node to confirm reachability?

##

Yes they can reach each other.

My first guess was at the uplink switch which is SG300.
I have enabled multicast on those and still were not luck. Then we have moved the second switch uplink to the first one and the issue was the same.

##

as long as the HSRP versions are the same, you are not denying udp/mc and you have reachability to each node it should work

##

I mentioned this in my first post, when I was replacing the other site core switches there were old 3550's pair with 10+ years old IOS and HSRP config and the 3750 worked pretty well active was 3550 and standby was 3750.

Right now I'm still deleting the secondary groups and changing also the first group to be different than the old config.

With this the virtual mac address will be relaced to according to the VLAN.

I think here is also the main issue is the different image version.

However now I could disconnect the first one and then upgrade the image and then do the second later.

So with this they will have the same image at least. But this will be a long process, cause the boxes are already in production.

Cheers,

Laz

[paul driver]

Hello

---

@7layer wrote:

I think here is also the main issue is the different image version.

---

Not so sure...., Obviously I cannot state this as a fact that different software images will be a factor, However if a rtr and switch can peer FHRP protocols(HSRP for one) then i would say two switch's of the same make/model but with differing SW versions should be able to.

What about the Access-lists, Have you tested HSRP without them being applied, do you have them pointing in the correct direction?

SVI ACL logic
IN = traffic originating from WITHIN svi going out of vlan
OUT = traffic originating from OUTSIDE svi going into vlan

[7layer]

Thank you for your help!

##

What about the Access-lists, Have you tested HSRP without them being applied, do you have them pointing in the correct direction?

##

On VLAN100 there is no ACL at all and it is not communicating.
Also where there is ACL only IN direction is applied.

I put the setup on bare minimum and I got 8 HSRP IP-s in 7 VLAN-s right now.

No matching group ID-s, every each group has different number and still the same results.

Only one thing changed that is the incoming packets, I can see now IN packets at the "debug standby packet" before it was empty too only outgoing were showing from the active node. Probably it has changed because I have added the ACL that you mentioned earlier to each VLAN. (permit udp any any eq 1985)

What is very strange is that it is actually working fine when the active side removed and it fails over the the other asap, but the cli shows that they can't see each other properly.

So could be ACL though but I can't remove it right now, WiFi is also getting out from this subnet and cannot be tested right now.

Cheers,

Laz

[paul driver]

Hello

Post the acl configuration please ...

FYI - if you wasn’t aware an ACL can be edited (standard or extended) without removing it from its assigned interface

[7layer]

Hello Paul,

I know about the ACL could be edited on the fly of course.

So as I remember HSRP worked when we implemented this two pair into the network and then the ACL was placed later on. (2 years passed by, so long run)

Also few extra info that I got now, I tested the HSRP group and the first block works as expected.

So the standby IP takes over on the second node when I remove the IP from the first node and this is true on each VLANs HSRP.

Still they cannot see each other on the cli first node active and can't see the second as sh standby says second unknown.

Also only one multi group HSRP setup is in the first VLAN, which does not work as it expected neither at the cli either at the end devices.

##

SW1#sh standby brief

Interface Grp Pri P State Active Standby Virtual IP
Vl1 1 110 P Active local unknown 172.20.0.10
Vl1 12 80 P Active local unknown 172.20.0.20
Vl2 21 110 P Active local unknown 192.168.102.10
Vl3 31 110 P Active local unknown 192.168.103.10
Vl4 41 110 P Active local unknown 192.168.104.10
Vl5 51 110 P Active local unknown 192.168.105.10
Vl6 61 110 P Active local unknown 192.168.106.10
Vl7 71 110 P Active local unknown 192.168.204.10

##

Second node:

SW2#sh standby brief
P indicates configured to preempt.
|
Interface Grp Pri P State Active Standby Virtual IP
Vl1 1 90 P Standby 172.20.0.11 local 172.20.0.10
Vl1 12 120 P Active local unknown 172.20.0.20
Vl2 21 90 P Standby 192.168.102.11 local 192.168.102.10
Vl3 31 90 P Standby 192.168.103.11 local 192.168.103.10
Vl4 41 90 P Standby 192.168.104.11 local 192.168.104.10
Vl5 51 90 P Standby 192.168.105.11 local 192.168.105.10
Vl6 61 90 P Standby 192.168.106.11 local 192.168.106.10
Vl7 71 90 P Standby 192.168.204.11 local 192.168.204.10
SW2

##

So if I'd say okay this is "fine" because it does what it should be, only issue lies on the cli.

And the second problem is the secondary HSRP IP on the second switch.

As you can see the 172.20.0.20 IP is active on both nodes.

Mac address is the same on both nodes for 172.20.0.20, it is not changing.

This makes very wired thing at the client site when you do trace route, first jump goes via 11 and then interrupt the trace and start asap then the second goes via the 21 switch.

This could be seen clearly on pingplotter and it swaps every 3 sec the default gw.
And it really is changing because packet loss is 0.2% on the 21 switch for 3 sec and then 0% on the 11 switch. (pingplotter shows this very well )

Also here is the ACL, and as I said it is messy not summarised on purpose just to see what goes where and if it is used or not.

It is a very long process to swap over an old firewall where IPSEC and 3 ISP sits with OSPF, GRE, you name it, this is the process to get rid of this... (Old Debian based layer 3 switch, firewall, purposely not named) 

###SW1#sh ip access-lists LAN
Extended IP access list LAN
10 permit udp any any eq 1985 (233082 matches)
20 permit ip 172.20.0.0 0.0.255.255 172.20.0.0 0.0.255.255 (29640733 matches)
30 permit ip host 172.20.100.21 any (56895 matches)
40 permit udp any host 255.255.255.255 eq bootps (574208 matches)
50 permit udp any host 255.255.255.255 eq bootpc (516731 matches)
60 permit ip host 172.20.100.12 any (4772327 matches)
70 permit ip 172.20.0.0 0.0.255.255 10.0.1.0 0.0.0.255 (227 matches)
80 permit ip 172.20.0.0 0.0.255.255 10.0.2.0 0.0.0.255 (252 matches)
90 permit ip 172.20.0.0 0.0.255.255 10.0.3.0 0.0.0.255 (23 matches)
100 permit ip 172.20.0.0 0.0.255.255 10.0.4.0 0.0.0.255 (9 matches)
110 permit ip 172.20.0.0 0.0.255.255 192.168.102.0 0.0.0.255 (196637 matches)
120 permit ip 172.20.0.0 0.0.255.255 192.168.104.0 0.0.0.255 (1427255 matches)
130 permit ip 172.20.0.0 0.0.255.255 192.168.204.0 0.0.0.255 (618537 matches)
140 permit ip 172.20.0.0 0.0.255.255 192.168.108.0 0.0.0.255 (295592 matches)
150 permit ip 172.20.0.0 0.0.255.255 192.168.214.0 0.0.0.255 (80439 matches)
160 permit udp host 172.20.100.20 eq bootps bootpc ntp 389 192.168.105.0 0.0.0.255 (7369 matches)
170 permit ip host 172.20.100.21 192.168.105.0 0.0.0.255 (4 matches)
180 permit ip host 172.20.100.20 192.168.105.0 0.0.0.255 (4401803 matches)
190 permit ip host 172.20.100.12 192.168.105.0 0.0.0.255 (12 matches)
200 permit ip host 172.20.100.20 192.168.103.0 0.0.0.255 (480920 matches)
210 permit ip host 172.20.100.12 192.168.103.0 0.0.0.255 (5 matches)
220 permit ip host 172.20.100.20 192.168.106.0 0.0.0.255 (520607 matches)
230 permit ip host 172.20.100.12 192.168.106.0 0.0.0.255 (10 matches)
240 deny ip 172.20.0.0 0.0.255.255 192.168.103.0 0.0.0.255 (1 match)
250 deny ip 172.20.0.0 0.0.255.255 192.168.105.0 0.0.0.255 (116170 matches)
260 deny ip 172.20.0.0 0.0.255.255 192.168.106.0 0.0.0.255 (19072 matches)
270 permit ip any any (147002494 matches)
SW1#

Thanks again for your advises!

[paul driver]

Hello
Where exactly is this acl being applied?  And tbh in its present form its doesn't make sense, For as it stands only what is required are 11 aces, the rest isn't necessary.

Your obviously denying ip between certain networks but i dont understand what they are however the following would acl would perform the same function

no ip access-list extended LAN
ip access-list extended LAN
permit ip host 172.20.100.21 192.168.105.0 0.0.0.255
permit ip host 172.20.100.20 192.168.105.0 0.0.0.255 
permit ip host 172.20.100.12 192.168.105.0 0.0.0.255 
permit ip host 172.20.100.20 192.168.103.0 0.0.0.255
permit ip host 172.20.100.12 192.168.103.0 0.0.0.255 
permit ip host 172.20.100.20 192.168.106.0 0.0.0.255 
permit ip host 172.20.100.12 192.168.106.0 0.0.0.255 
deny ip 172.20.0.0 0.0.255.255 192.168.103.0 0.0.0.255 
deny ip 172.20.0.0 0.0.255.255 192.168.105.0 0.0.0.255 
deny ip 172.20.0.0 0.0.255.255 192.168.106.0 0.0.0.255 
permit ip any any

[7layer]

Hi Paul,

It's in the 172.20 network (VLAN 1) IN where the first HSRP sits.
Not really safe I know cause VLAN 1 should be disabled but this is an inherited network and we are trying to get rid of the SPOF first which is one of the main issue here.

Anyhow I will clean up this ACL and make the SW version the same on both nodes. 12.2 first node and 15.0(1) second node.
Some arp commands are even missing from the 12.2 version.

Kind Regards,
Laz

[MHM Cisco World]

"A switch running HSRPv1 cannot identify the physical router that sent a hello packet because the source MAC address of the router is the virtual MAC address.

HSRPv2 has a different packet format than HSRPv1. A HSRPv2 packet uses the type-length-value (TLV) format and has a 6-byte identifier field with the MAC address of the physical router that sent the packet.

If an interface running HSRPv1 gets an HSRPv2 packet, the type field is ignored."
this from cisco doc. so change from HSRP v1 to v2 may be solve the issue of virtual-mac address of each group under the SVI.