network unreachable via standby router, misconfigured standby groups?

[o.breiler]

This is our topology:

The two Nexus 7010 are our core switches. The two Catalyst 6509 "65-01" and "65-02" are our core routing devices. They provide the OSPF processes and standby groups for all our VLAN interfaces.

VLAN 2 is our internet vlan. The ASA failover cluster is connected via two Catalyst 3550 switches to VLAN 2. "ASA-2" is the primary and active device.

"65-02" is always the active member for all standby groups:

65-2#sho stand brief

                     P indicates configured to preempt.

                     |

Interface   Grp Prio P State    Active addr     Standby addr    Group addr

Vl1         1   110  P Active   local           192.168.114.51  192.168.114.50

Vl2         2   110  P Active   local           10.10.10.51       10.10.10.50

65-2#sho run int vlan 2

!

interface Vlan2

ip address 10.10.10.52 255.255.255.0

no ip redirects

standby 2 ip 10.10.10.50

standby 2 priority 110

standby 2 preempt

end

65-2#sho run int vlan 1

!

interface Vlan1

ip address 192.168.114.52 255.255.254.0

no ip redirects

standby 1 ip 192.168.114.50

standby 1 priority 110

standby 1 preempt

end

On both "65-1" and "65-2" a default route is set and pointing to ASA failover cluster inside address 10.10.10.200:

ip route 0.0.0.0 0.0.0.0 10.10.10.200

So, now the problem !!!

yesterday interface 1/1 on "65-02" went down due to a malfunctioning line card. The primary active "ASA-2" stayed active and didn't make a failover to secondary ASA, because all local interface were still connected. OK

The two standby groups 1 and 2 also probably stayed active on "65-02" and didn't switch over to "65-01". OK

www was unreachable for all clients in VLAN 1, but also for all other clients. NOT OK !!

After powering active ASA-2 off and ASA-1 becoming active, www was reachable again. ???

What could be the problem for it?

After fixing the line card issue and switching back to ASA-2 as active device we tested a little. So, usually the MAC address of standby ASA-1 ip 10.10.10.254 is learned on "65-02" via Po3. OK

After shutting down interface 1/1 on "65-01" manually the MAC was learned on "65-02" via interface 1/1, because the cross link between two "35-2" provides the mac to "65-02". OK

So that part works.

Could the way back (answer packets) from ASA to client in VLAN 1 be the problem? After a failure of int 1/1 on "65-02" all packets from ASA-2 to client arrive on "65-01", which is only standby member for VLAN interfaces?

We have no ideas and are not allowed to test it again the hard way and shut interface 1/1 on "65-02" again. :-(

Any ideas?

[acampbell]

Oliver,

You saw an issue with int 1/1 on 65-02.
Did the coresponding int on 35-02 show that
it was down ?

If this in was still up it could be passing data to
65-02 int 1/1 as if nothing was wrong.

You could also try changing your HSRP for VLAN2 on 65-02

!

interface Vlan2

ip address 10.10.10.52 255.255.255.0
no ip redirects
standby 2 ip 10.10.10.50
standby 2 priority 110
standby 2 preempt
standby 2 track (int 1/1) 20
end

This will mean that the HSRP will hand off to 65-01 in the event
of 65-02 int 1/1 failing.

[o.breiler]

Hi,

thank you for your response.

yes, corresponding interface on 35-2 was down.

Yes, it's a good advise for a workaround to track the interface. We also thought about it. But we think, it is just a workaround, not a solution. In addition we have 10 standby groups and 65-02 is the active member for all of these groups. So we need to track the interface 1/1 within all standby groups. Does this makes sense?

In our topology there must be a way to use the usual standby group rerouting and reswitching mechanisms, must it not?

What we don't know is, what happens with packets (containing vlan1 and vlan2 addresses in source and destination) arriving on 65-01, if 65-01 is only the standby member for standby groups 1 and 2? What is its default behavior?

[Abzal]

Hi Oliver,

yes, you need to track interface withtin all standby groups on active HSRP. Active HSRP should be also STP root bridge in this way standby HSRP trunk ports will be on ALT state unless root port goes down. So packets will flow toward active HSRP.

Hope it will help.

[pille1234]

Hi,

you have an interesting problem there. Tracking the interfaces is a good practise because it improves your traffic flow, however your problem seems to go deeper Based on this

After powering active ASA-2 off and ASA-1 becoming active, www was reachable again. ???

I would check the interlink between 35-1 and 35-2. VLAN 2 should be allowed on this trunk (?) port on both sides. This way you can always reach the current active ASA-IP.

To your question

What we don't know is, what happens with packets (containing vlan1 and  vlan2 addresses in source and destination) arriving on 65-01, if 65-01  is only the standby member for standby groups 1 and 2? What is its  default behavior?

The packets from the Internet destined for vlan 1 will be sent to the next hop address from the active ASA. In your case, the next hop is most probably the HSRP IP of vlan 2. Therefore the ASA arps for the HSRP virtual MAC address and (hopefully) gets an answer from 65-02. The packet is then sent out on the inside interface, goes through 35-1 to 65-01 and will be switched in a pure layer 2 style out to portchannel 2 until the 65-02 is reached where the routing to vlan 1 takes place.

Hope it helps.

Regards Pille

[Peter Koltl]

I would add the ASA's to the OSPF domain then you could forget HSRP and all tracking in VLAN2. Another important condition: there must be a VLAN apart from VLAN2 where 65-01 and 65-02 speak OSPF with each other (on a different physical interface).