Problems with Cisco 9200 Stack When the Active Switch Fails

[agh1978]

Good morning, I have detected an issue with a Cisco 9200L switch stack running software version 17.06.03. We observe that when the switch with the active role fails, hosts connected to the backup switch lose communication for about 5 or 6 seconds (we have noticed this with both RTP traffic and ICMP). Is this behavior normal? I understood that the failure of a stack member should be transparent to the network.

The failure should not trigger topology changes, as both switches share port channels for external connectivity, meaning those ports do not go down (though they may degrade). Additionally, hosts connected to the same switch also lose communication with each other.

Does the role switchover from Standby to Active on Cisco 9200 switches cause a restart of processes that results in these outages?

Additionally, I see that when the active switch fails, the device that was standby and becomes active shows messages on the console about the ports of the other switch going down, which is normal, but I also see as if its own ports are being brought up again. Attached are the messages:

ec 11 12:51:47.927: %PLATFORM-6-HASTATUS: RP switchover, received chassis event to become active
Dec 11 12:51:48.298: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 2: EMP_RELAY: Channel UP!
Dec 11 12:51:47.620: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 1 on Switch 2 is down
Dec 11 12:51:48.752: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_NOT_PRESENT)
Dec 11 12:51:48.752: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_DOWN)
Dec 11 12:51:48.752: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_REDUNDANCY_STATE_CHANGE)
Dec 11 12:51:49.267: %PLATFORM-6-HASTATUS: RP switchover, sent message became active. IOS is ready to switch to primary after chassis confirmation
Dec 11 12:51:47.623: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 2 on Switch 2 is down
Dec 11 12:51:47.624: %STACKMGR-4-SWITCH_REMOVED: Switch 2 R0/0: stack_mgr: Switch 1 has been removed from the stack.
Dec 11 12:51:49.432: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
Dec 11 12:51:49.605: %PLATFORM-6-HASTATUS: RP switchover, received chassis event became active
Dec 11 12:51:49.622: %PLATFORM_FEP-1-FRU_PS_SIGNAL_OK: Switch 2: signal on power supply A is restored
Dec 11 12:51:49.761: %PLATFORM-6-HASTATUS_DETAIL: RP switchover, received chassis event became active. Switch to primary (count 1)
Dec 11 12:51:49.762: %HA-6-SWITCHOVER: Route Processor switched from standby to being active
Dec 11 12:51:49.924: Unable to set IPV4 table id for BT interface

Dec 11 12:51:49.930: Unable to set IPV6 table id for BT interface

Dec 11 12:51:51.723: %HMANRP-5-CHASSIS_DOWN_EVENT: Chassis 1 gone DOWN!
Dec 11 12:51:52.133: %LINK-3-UPDOWN: Interface Lsmpi11/3, changed state to up
Dec 11 12:51:52.148: %LINK-3-UPDOWN: Interface EOBC11/1, changed state to up
Dec 11 12:51:52.149: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN11/2, changed state to up
Dec 11 12:51:52.150: %LINK-3-UPDOWN: Interface LIIN11/2, changed state to up
Dec 11 12:51:53.028: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
Dec 11 12:51:53.316: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi11/3, changed state to up
Dec 11 12:51:53.317: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC11/1, changed state to up
Dec 11 12:51:53.317: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Dec 11 12:51:53.317: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to down
Dec 11 12:51:53.317: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/4, changed state to down
Dec 11 12:51:53.317: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to down
Dec 11 12:51:53.318: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/6, changed state to down
Dec 11 12:51:53.318: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to down
Dec 11 12:51:53.318: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/8, changed state to down
Dec 11 12:51:53.318: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/9, changed state to down
Dec 11 12:51:53.318: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/10, changed state to down
Dec 11 12:51:53.476: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/11, changed state to down
Dec 11 12:51:53.476: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/12, changed state to down
Dec 11 12:51:53.476: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to down
Dec 11 12:51:53.477: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/23, changed state to up
Dec 11 12:51:53.477: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to down
Dec 11 12:51:53.477: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to down
Dec 11 12:51:53.477: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down
Dec 11 12:51:53.477: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/17, changed state to down
Dec 11 12:51:53.528: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/18, changed state to down
Dec 11 12:51:53.528: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/19, changed state to down
Dec 11 12:51:53.528: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/20, changed state to down
Dec 11 12:51:53.529: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/21, changed state to down
Dec 11 12:51:53.529: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/22, changed state to down
Dec 11 12:51:53.576: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to down
Dec 11 12:51:53.577: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/24, changed state to down
Dec 11 12:51:53.970: %CALL_HOME-6-CALL_HOME_ENABLED: Call-home is enabled by Smart Agent for Licensing.
Dec 11 12:51:56.162: %LINK-3-UPDOWN: Interface Null0, changed state to up
Dec 11 12:51:56.163: %LINK-3-UPDOWN: Interface Vlan3, changed state to up
Dec 11 12:51:56.165: %LINK-3-UPDOWN: Interface Vlan4, changed state to up
Dec 11 12:51:56.167: %LINK-3-UPDOWN: Interface Vlan99, changed state to up
Dec 11 12:51:56.168: %LINK-3-UPDOWN: Interface Vlan322, changed state to up
Dec 11 12:51:56.170: %LINK-3-UPDOWN: Interface Vlan323, changed state to up
Dec 11 12:51:56.173: %LINK-3-UPDOWN: Interface Vlan324, changed state to up
Dec 11 12:51:56.176: %LINK-3-UPDOWN: Interface Vlan325, changed state to up
Dec 11 12:51:56.178: %LINK-3-UPDOWN: Interface Vlan326, changed state to up
Dec 11 12:51:56.180: %LINK-3-UPDOWN: Interface Vlan327, changed state to up
Dec 11 12:51:56.181: %LINK-3-UPDOWN: Interface Vlan328, changed state to up
Dec 11 12:51:56.183: %LINK-3-UPDOWN: Interface Vlan329, changed state to up
Dec 11 12:51:56.185: %LINK-3-UPDOWN: Interface Vlan330, changed state to up
Dec 11 12:51:56.187: %LINK-3-UPDOWN: Interface Vlan331, changed state to up
Dec 11 12:51:56.189: %LINK-3-UPDOWN: Interface Vlan332, changed state to up
Dec 11 12:51:56.190: %LINK-3-UPDOWN: Interface Vlan333, changed state to up
Dec 11 12:51:56.192: %LINK-3-UPDOWN: Interface Vlan1322, changed state to up
Dec 11 12:51:56.194: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to up
Dec 11 12:51:56.195: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to up
Dec 11 12:51:56.196: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to up
Dec 11 12:51:56.197: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to up
Dec 11 12:51:56.199: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/5, changed state to up
Dec 11 12:51:56.200: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to up
Dec 11 12:51:56.201: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/13, changed state to up
Dec 11 12:51:56.202: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/14, changed state to up
Dec 11 12:51:56.204: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/17, changed state to up
Dec 11 12:51:56.205: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/18, changed state to up
Dec 11 12:51:56.206: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/21, changed state to up
Dec 11 12:51:56.207: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/22, changed state to up
Dec 11 12:51:56.208: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/23, changed state to up
Dec 11 12:51:56.208: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/24, changed state to up
Dec 11 12:51:56.251: %LINK-3-UPDOWN: Interface GigabitEthernet2/1/1, changed state to up
Dec 11 12:51:56.253: %LINK-3-UPDOWN: Interface GigabitEthernet2/1/2, changed state to up
Dec 11 12:51:56.254: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
Dec 11 12:51:56.281: %LINK-3-UPDOWN: Interface Port-channel2, changed state to up
Dec 11 12:51:57.160: %LINEPROTO-5-UPDOWN: Line protocol on Interface Null0, changed state to up
Dec 11 12:51:57.163: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan4, changed state to up
Dec 11 12:51:57.180: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan322, changed state to up
Dec 11 12:51:57.182: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan323, changed state to up
Dec 11 12:51:57.184: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan324, changed state to up
Dec 11 12:51:57.186: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan325, changed state to up
Dec 11 12:51:57.187: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan326, changed state to up
Dec 11 12:51:57.189: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan327, changed state to up
Dec 11 12:51:57.190: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan328, changed state to up
Dec 11 12:51:57.192: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan329, changed state to up
Dec 11 12:51:57.194: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan330, changed state to up
Dec 11 12:51:57.195: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan331, changed state to up
Dec 11 12:51:57.197: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan332, changed state to up
Dec 11 12:51:57.198: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan333, changed state to up
Dec 11 12:51:57.200: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1322, changed state to up
Dec 11 12:51:57.202: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/1, changed state to up
Dec 11 12:51:57.203: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/2, changed state to up
Dec 11 12:51:57.203: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/3, changed state to up
Dec 11 12:51:57.203: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/4, changed state to up
Dec 11 12:51:57.204: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/5, changed state to up
Dec 11 12:51:57.204: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Dec 11 12:51:57.205: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/13, changed state to up
Dec 11 12:51:57.205: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to up
Dec 11 12:51:57.205: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/17, changed state to up
Dec 11 12:51:57.205: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/18, changed state to up
Dec 11 12:51:57.206: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/21, changed state to up
Dec 11 12:51:57.206: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/22, changed state to up
Dec 11 12:51:57.210: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/24, changed state to up
Dec 11 12:51:57.248: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1/1, changed state to up
Dec 11 12:51:57.253: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1/2, changed state to up
Dec 11 12:51:57.253: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
Dec 11 12:51:57.280: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel2, changed state to up
Dec 11 12:51:57.595: %SMART_LIC-6-REPORTING_REQUIRED: A Usage report acknowledgement will be required in 0 days.
Dec 11 12:51:59.955: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
Dec 11 12:52:00.206: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
Dec 11 12:52:00.208: %LINK-5-CHANGED: Interface Vlan999, changed state to administratively down
Dec 11 12:52:00.210: %LINK-5-CHANGED: Interface GigabitEthernet1/1/1, changed state to administratively down
Dec 11 12:52:00.211: %LINK-5-CHANGED: Interface GigabitEthernet1/1/2, changed state to administratively down
Dec 11 12:52:00.211: %LINK-5-CHANGED: Interface GigabitEthernet1/1/3, changed state to administratively down
Dec 11 12:52:00.211: %LINK-5-CHANGED: Interface GigabitEthernet1/1/4, changed state to administratively down
Dec 11 12:52:00.211: %LINK-5-CHANGED: Interface GigabitEthernet2/0/8, changed state to administratively down
Dec 11 12:52:00.211: %LINK-5-CHANGED: Interface GigabitEthernet2/0/16, changed state to administratively down
Dec 11 12:52:00.211: %LINK-5-CHANGED: Interface GigabitEthernet2/0/19, changed state to administratively down
Dec 11 12:52:00.212: %LINK-5-CHANGED: Interface GigabitEthernet2/0/20, changed state to administratively down
Dec 11 12:52:00.212: %LINK-5-CHANGED: Interface GigabitEthernet2/1/3, changed state to administratively down
Dec 11 12:52:00.212: %LINK-5-CHANGED: Interface GigabitEthernet2/1/4, changed state to administratively down

[Flavio Miranda]

@agh1978 

   My suspicious is related to spanning-tree. The primary switch probably is the root for the spanning-tree and the port-channel connected to it must be in forwarding state. When the primary switch goes down and the second switch takes over, there will be a spanning-tree recalculation and the second port-channel will change from blocking to forwarding.

 I dont know your topology but this could be one explanation.

[agh1978]

Hello Flavio, thank you very much for your response.

I was also concerned about STP, but I don't see any topology changes since the port channels share one port on each switch. That is, Port Channel 1 is made up of ports GB 1/1/1 and GB 2/1/1, so at the logical level, the port does not go down, and STP should not recalculate.

Let me adjust my response slightly. I have noticed that the problem seems to be with the port channel. Let me explain: the port channel is formed by two ports, as mentioned earlier, GB 1/1/1 and GB 2/1/1. When we power down the active switch, let’s call it SW1, the GB 1/1/1 port obviously goes down, but the port channel remains active. Despite this, we still see a loss of about 5 or 6 seconds. However, if before powering down switch 1 we first shut down port GB 1/1/1, leaving the port channel active but only with GB 2/1/1, and then power down the active switch (SW1), we don’t observe that traffic loss.

We don’t see changes in protocols like Spanning Tree, but I don’t understand why this behavior occurs with a port channel. Shouldn’t the load balancing mechanism make the failure of one member almost transparent? Is it normal for it to take 5 or 6 seconds to start sending traffic through the second link in the port channel?