Issues with HSRP + Routing

[thefirespray]

Hello everyone,

I have been pulling my hair out the past couple of days trying to resolve a routing issue in a network. I have checked and verified everything I can think of but it seems to no avail. Issue is as follows:

The ZW_L3_2 and ZW_L3_4 are configured as an HSRP group. This is working correctly, I am able to ping the 10.1.20.3 or 10.1.10.3 address from every host. If I turn of the primary router it will failover over without issue. However, I seem to have some sort of routing issue between the HSRP Switches and the Firewalls (ZW_FW_1 and ZW_FW_2). The Firewalls are PFsense's configured in a failover cluster (CARP IP: 192.168.0.1). I can ping this address from both the ZW_L3_2 and ZW_L3_4 without issue. (Also the other way around).

However, If I ping this address from the hosts it will timeout. A traceroute tells me that the connections stops at either ZW_L3_L4 or ZW_L3L2 (depending on the active router). I don't understand this, as both of these are able to ping the hosts and the firewall, but will apparently not route traffic between them. To summarize:

Firewall connections to HSRP switches ZW_L3_2 and ZW_L3_4: OK

Hosts connections to HSRP switches ZW_L3_2 and ZW_L3_4: OK

Connections between Hosts and Firewall: Fails.

Connections from Hosts to the 192.168.0.6 Address: OK (Which makes this even stranger to me)

I can also ping the 10.1.XX addresses on the switches, so the firewall has a correct route to the subnet.

Does anybody have any clue what is going on here? I would greatly appreciate any help! The topology and Configs are included in the post.

[Giuseppe Larosa]

Hello thefirespray,

your HSRP configurations on devices ZW_L3_4 and ZW_L3_2 are not correct to perform failover.

All HSRP members of an HSRP group need the standby XX preempt not only the device with the higher priority in the group.

The devices named R31 and R32 look like to act as L2 only switches as their configurations have no L3 SVI interfaces.

They are useless and create only redundancy at Layer2 that should be managed carefully by STP.

From network diagram in topology we see that ZW_L3_4 and ZW_L3_2 already have a port-channel between them that provide all the necessary L2 redundancy between them for all Vlans.

You have added R31 and R32 probably to act as media converter between fiber based links on switches and RJ 45 ports on firewalls.

If this is the case only Vlan 100 is needed on them. There is no advantage on carrying all Vlans to them and then

sending only Vlan 100 to the firewalls.

Just to understand possible issues in Vlan 100

Which switch is the root bridge for Vlan 100 ?

It should be the same device that is HSRP master for VIP 192.168.0.6.

Hope to help

Giuseppe

[thefirespray]

Hello Giuseppe,

Thank you very much for your response, it has certainly helped me further. Couple of changes I made after your response:

-HSRP configuration has been changed to include preempt on both sides.

-The L2 Switches were indeed a misguided attempt at more L2 redundancy, and have thus been removed.

-Set the STP Priority to primary for VLAN 10,20,100 on the primary HSRP switch. The secondary HSRP switch has the root secondary statement.

-Virtual IP's have been removed from the Firwalls and the Firwewall Facing Side of the HSRP side. They were unneeded for my needs.

Now everything works without problems (Host to Firewall communication OK) if I have both switches in the HSRP cluster turned on. However, If I turn one of (example attached as topology) it fails over properly and I can still ping the 10.1.10.3 adress, even the 192.168.0.4 outside address. I can however not ping the firewall (192.168.0.2 Left or 192.168.0.3 right) from the host. I can do this from the ZW_L3_3 switch or the ZW_L3_4. A traceroute tells me that the connections stops at the HSRP switch. Both the firewall and the host reach the switch but are not routed further. Even though the switch contains routes for both networks. I cannot seem to find any misconfiguration in my configs.

I have attached the new configs and topology. It would be of great help if you could take a look, as I feel I am very close. (L2  Switches configs have stayed the same).

Kind regards

[Giuseppe Larosa]

Hello thefirespray,

I'm happy that the scenario has improved and that when both switches are up and running all the required connectivity between clients hosts and firewalls is successful.

You have removed R31 and R32 from the topology and this simplifies STP job.

You have configured device ZW_L3_4 as the root bridge with command

>> spanning-tree vlan 10,20,100 priority 24576

And device ZW_L3_3 is the secondary root with command

>> spanning-tree vlan 10,20,100 priority 28672

You have implemented preemption on all HSRP groups both members.

However, I don't understand your new configuration in Vlan 100 the Vlan to the firewalls.

I would expect to have an HSRP group here as before to provide an always on VIP next-hop to the firewalls.

I would recommend to re-introduce an HSRP group on the switches ZW_L3_4 master for VIP 192.168.0.6 and ZW_L3_3 standby for HSRP group 100.

It is also important to know how the two firewalls work if they are working in failover (Active/Standby) or they are a cluster (Active/Active ).

Depending on the firewall redundancy strategy the firewalls can react to one switch power off in different ways.

First of all if you switch off the current master ZW_L3_4 the firewall directly attached to it will sense its interface going down.

This fact depending on the firewall redundancy strategy can cause changes:

For example in failover the standby firewall can become the Active firewall and it can take over also the IP addresses used by the previously Active firewall.

As a first step in improving Vlan 100 routing I will do the following:

a) re-introduction of HSRP group with VIP 192.168.0.6 to be used as next-hop for static routes for internal networks on both the firewalls.

b)  Each switch ZW_L3_4 and ZW_L3_3 must have two default static routes and not only one.

on ZW_L3_3

ip route 0.0.0.0 0.0.0.0 192.168.0.2

We can add a static route to other firewall

ip route 0.0.0.0 0.0.0.0 192.168.0.3

the same on ZW_L3_4

c) As an alternative to point b) I would also consider to re-introduce Virtual IP on firewalls themselves.

It really depends on how they behave after link failure

If standby takes over and becomes new 192.168.0.3 for its active state the single static route on ZW_L3_3 is pointing to old address 192.168.0.2 with current setup and it can become useless.

This can be part of the problem you are facing when powering off ZW_L3_4.

Hope to help

Giuseppe