08-20-2008 06:04 AM - edited 03-06-2019 12:55 AM
Hi,
I have a serious problem with HSRP during a SUP720 SSO switchover.
Let's consider to 6500 switches, SW01 (10.56.6.2) and SW02 (10.56.6.3), lined together with a trunk. We have a standby group 12 on VLAN 300. Preemtion is configured on SW01.
During a supervisor SSO switchover on SW01, HSRP moves on SW02. But clients connected to SW01 are not able to reach the HSRP address, until the secondary supervisor on SW01 takes-over the HSRP active state (remember preemtion is configured).
However, clients on SW02 can see the HSRP virtual IP address correctly, just after HSRP moves to SW02.
I traced HSRP, ARP and ICMP to troubleshoot.
The symptom is that the SW01 statically keeps ownership of the HSRP MAC address even in the standby state. It seams that if a packet arrives on SW01 with the destination MAC 00.00.0C.07.AC.0C(HSRP MAC Address), the switch thinks that he is still the owner of this MAC address but the virtual IP address 10.56.6.1 (HSRP address) is actually active on the new active router on SW02 !
During the supervisor switchover, each time a packet is received on SW01 and destined to the HSRP MAC address, we can see that the switch SW01 sends an ICMP TTL expiration message back to the client :
10.56.6.100 10.56.6.1 ICMP Echo (ping) request
10.56.6.2 10.56.6.100 ICMP Time-to-live exceeded (Time to live exceeded in transit)
10.56.6.100 10.56.6.1 ICMP Echo (ping) request
10.56.6.2 10.56.6.100 ICMP Time-to-live exceeded (Time to live exceeded in transit)
10.56.6.2 224.0.0.2 HSRP Hello (state Standby)
10.56.6.3 224.0.0.2 HSRP Hello (state Active)
10.56.6.100 10.56.6.1 ICMP Echo (ping) request
10.56.6.2 10.56.6.100 ICMP Time-to-live exceeded (Time to live exceeded in transit)
10.56.6.100 10.56.6.1 ICMP Echo (ping) request
10.56.6.2 10.56.6.100 ICMP Time-to-live exceeded (Time to live exceeded in transit)
10.56.6.2 224.0.0.2 HSRP Hello (state Standby)
10.56.6.100 10.56.6.1 ICMP Echo (ping) request
We run IOS version : 12.2(18)SXF14
I know that SSO does not keep state information for HSRP, but I expect that all clients continue to reach the HSRP virtual address on the standby switch.
Thank you for any hints
Yves Haemmerli
08-20-2008 06:16 AM
Yves,
I have had this problem before and it took me quite a while to figure it out. It turned out to be a mis configuration of the standby preempt under the interface config.
When the pings to the virtual IP went TTL on a supervisor failover this was the HSRP config:
standby preempt
standby 20 ip 10.0.20.1
standby 20 priority 130
After adding standby 20 preempt instead of standby preempt fixed the problem right away.
HTH,
Mark
08-20-2008 06:50 AM
Mark,
Thank you for your suggestion. Actually, the configuration is correct in my switches. here is the example of VLAN301 on SW01 and SW02 :
on SW01 :
---------
interface Vlan301
ip vrf forwarding CH01RT01
ip address 10.56.6.2 255.255.255.0
ip helper-address 10.56.6.222
ip helper-address 10.240.4.147
no ip redirects
standby 12 ip 10.56.6.1
standby 12 timers 1 4
standby 12 priority 200
standby 12 preempt
standby 12 authentication blablabla
On SW02 :
---------
interface Vlan301
description *** Interface to Infrastructure Servers VLAN ***
ip vrf forwarding CH01RT02
ip address 10.56.6.3 255.255.255.0
no ip redirects
standby 12 ip 10.56.6.1
standby 12 timers 1 4
standby 12 priority 150
standby 12 authentication blablabla
As you can see, the interface on which HSRP is configured is in a VRF... This shout however work anyway.
Yves
08-20-2008 06:56 AM
Yves,
On SW02 under Vlan 301 there is no preempt command configured. Is this a typo??
Mark
08-20-2008 07:33 AM
Mark,
Oh yes, it is a typo... here is the SW02 configuration cut&past from the switch :
SW01 :
------
SW02 :
------
interface Vlan301
description *** Interface to Infrastructure Servers VLAN ***
ip vrf forwarding CH01RT02
ip address 10.56.6.3 255.255.255.0
ip helper-address 10.56.6.222
ip helper-address 10.240.4.147
no ip redirects
ip ospf authentication message-digest
ip ospf authentication-key 7 046C03071B04405D0C
standby 12 ip 10.56.6.1
standby 12 timers 1 4
standby 12 priority 150
standby 12 preempt
standby 12 authentication blablabla
One another interesting thing : Under normal conditions, when SW01 is the HSRP master, the HASR MAC Address on SW02 is dynamic and the same MAC address is static on SW01, the active router. BUT, when SW01 is standby and SW02 active, SW01 does not change the MAC address as dynamic. SW01 keeps a static entry !! here is the problem I think...
You can see hereafter the mac address table before switchover :
CH01SW02#show mac-address-table vlan 301
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
.
.
* 301 0000.0c07.ac0c dynamic Yes 0 Po1
CH01SW01#show mac-address-table vlan 301
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
.
.
* 301 0000.0c07.ac0c static No - Router
During the Supervisor switchover in SW01, the entry is still static on SW01 :
CH01SW01#sh mac-address-table vlan 301
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
------+----------------+--------+-----+----------+--------------------------
.
.
* 301 0000.0c07.ac0c static No - Router
08-20-2008 08:56 AM
That is strange. What is the output of "show standby" on both switches? I would be curious to see how different the output would be on both switches. I'm not very familiar with VRF and couldn't say if that is playing a role with the problem.
Mark
08-21-2008 12:10 AM
Mark,
Thank you for following this posting, I really appreciate. In the mean time, Cisco confirmed to me that this is a bug in the IOS version 12.2(18)SXF14 ! In fact, the observatkion I did was correct : the MAC address table entry for the HSRP address (you know, the famous 00.00.0C.07.AC.xx) is not refreshed during SSO switchover. Therefore, the entry remains "static" intead of changing to "dynamic" (static should be only on the HSRP active router). Therefore, the switch doing the SSo switchover continues to think it "owns" the HASR MAC address, but the real HSRP state is standby !
The bug won't be fixed by Cisco. Cisco recommends to upgrade to IOS 12.2(33)SXH, that support HSRP switchover with SSO...This means that, during a supervisor switchover, the HSRP state stays active on the switch. I will do the test today and let you knoe on the result.
To complete the explanation, I attach my personal problem analysis file in this post.
Yves
08-21-2008 05:31 AM
Yves,
Thanks for letting us know that it was a bug, and analysis on your issue. I am sure that this post will also help others in the future. Rating post as deserved.
Mark
08-23-2008 12:10 AM
Hi Yves,
We are in a process of migrating our configuration from one 4507 to two 6513 switches where we will be using HSRP between the two 6513 switches. Can you let me know which particular IOS version of 12.2(33)SXH should we go for?
Thanks in advance.
08-23-2008 02:48 AM
Hi,
I installed version 12.2(33)SXH2a, the latest release in the CCo. It works fine now, HSRP does not jump on the secondary switch in case of a supervisor switchover on the primary switch. I observe about 3 ping loss during switchover, which is acceptable in our environment.
By the way, we have Catalyst 6509 not 6513, but it shouldn't caus any difference.
Yves
08-24-2008 09:19 AM
Hi Yves,
Thanks for the reply. But I would like to confirm thing whether I can use this image to configure HSRP on SUP32? Also can I use this image to configuere AutoQoS and VOIP?
Thanks in advance.
08-26-2008 02:03 AM
Hi,
Yes you can use AutoCos and VoIP of course. I installed this version ona SUP 720, not on a SUP32, but I don't see any reason for any restriction.
Yves
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide