strange Etherchannel/HSRP/WS-X6724-SFP behaviour

[Arjan Lugtenberg]

Last year we experienced on our C6509 a card failure in slot 2.

In that slot following card is present:

6500-MER-A#sh module 2
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  2   24  CEF720 24 port 1000mb SFP              WS-X6724-SFP       SAL1024R4P1

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  2  0018.191e.7570 to 0018.191e.7587   2.3   12.2(14r)S5  12.2(18)SXF7 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  2  Centralized Forwarding Card WS-F6700-CFC       SAL1021PAWH  2.0    Ok

Mod  Online Diag Status
---- -------------------
  2  Pass

the card itself was rebooted after the failure and did come back properly, so that was not a problem.

During the reboot of the card a strange behaviour was detected.

Let me first explain our core network:

In our network we have 2 6509.

One in each MER.

The 2 6509's are connected with 2 gigabit fiber connections.

These 2 connections are in a etherchannel. (port-channel3)

I will attach the relevant configs to this thread.

What we saw is that both 6509's reported their SVI's as HSRP active!!

So actually the 2 6509's did not see eachother anymore via the Etherchannel!

And that does not make sense to us because the second interface of the Etherchannel (Gi3/24) was not impacted by this reboot/failure.

We have tested it out in the lab with our network distributor but than we could not reproduce the problem we experienced.

Does anybody know if it has somehow to do with the hardware revision of the card?

Or is it some other configurational problem which relates back to this problem?

We have looked at the spanning-tree, hsrp and etherchannel configs and that looks fine to us (including our network distributor)

[Giuseppe Larosa]

Hello Arjan,

>> We have tested it out in the lab with our network distributor but than we could not reproduce the problem we experienced.

>> What we saw is that both 6509's reported their SVI's as HSRP active!!

Have you verified if a bridging loop and broadcast storm had formed on your network during the time that card in slot 2 was rebooting?

How much traffic was on the healthy link g3/24 during this issue?

I ask this because in most cases of bridging loops /broadcast storms one of the symptoms is an active/active on HSRP also on C6500 platforms.

Hope to help

Giuseppe

[Arjan Lugtenberg]

> Have you verified if a bridging loop and broadcast storm had formed on your network during the time that card in slot 2 was rebooting?

There was no sign of a bridging loop or broadcast storm in the logging (syslog and local logging) of the 6509 or in one of the access layer switches which are connected to the card in slot 2. Is there some other way to see this?

> How much traffic was on the healthy link g3/24 during this issue?

The actual issue was a couple of months ago so I only have graphs with a 2 hour average value in it. In there I can not determine how much traffic was on the heatlhy interface. In normal circumstances average traffic rate is around 16 Mbit/second on Gi3/24 and on gi2/24 average is 22 Mbit/sec.

On both interface we see a max. of 100Mbit/s of traffic.

So that should not be our problem.