Hi,

mmason221 · ‎08-24-2016

Cisco C6880-X running 151-2.SY3

We we installed OIR a C6880-X-16P10G hot-swappable card and CPU went to 100%, also appears routed SVI's dropped on card install and subsequent card cycle post upgrade. Furthermore, seems L3 routed interfaces stayed online, so I'm surmising switching path was stalled.

term exec prompt timestamp

During card install:

show module
Load for five secs: 99%/1%;

Aug 24 13:35:51.838 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.102.243.4 on Vlan100 from FULL to DOWN, Neighbor Down: Interface down or detached
Aug 24 13:35:51.842 CDT: %LDP-5-NBRCHG: LDP Neighbor 10.102.243.4:0 (1) is DOWN (Interface not operational)
Aug 24 13:35:51.846 CDT: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5582: Neighbor 10.102.240.162 (Vlan1438) is down: interface down
Aug 24 13:35:55.554 CDT: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5582: Neighbor 10.102.240.162 (Vlan1438) is up: new adjacency
Aug 24 13:35:56.630 CDT: %LDP-5-NBRCHG: LDP Neighbor 10.102.243.4:0 (1) is UP
Aug 24 13:35:58.394 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.102.243.4 on Vlan100 from LOADING to FULL, Loading Done

Aug 24 13:36:58.778 CDT: %C6KPWR-4-DISABLED: power to module in slot 1 set off (Reset)
Aug 24 13:37:26.451 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.102.243.4 on Vlan100 from FULL to DOWN, Neighbor Down: Interface down or detached
Aug 24 13:37:26.455 CDT: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5582: Neighbor 10.102.240.162 (Vlan1438) is down: interface down
Aug 24 13:37:26.467 CDT: %LDP-5-NBRCHG: LDP Neighbor 10.102.243.4:0 (1) is DOWN (Interface not operational)
Aug 24 13:37:28.460 CDT: %OSPF-5-ADJCHG: Process 1, Nbr 10.102.243.4 on Vlan100 from LOADING to FULL, Loading Done
Aug 24 13:37:32.048 CDT: %LDP-5-NBRCHG: LDP Neighbor 10.102.243.4:0 (8) is UP
Aug 24 13:37:32.884 CDT: %DUAL-5-NBRCHANGE: EIGRP-IPv4 5582: Neighbor 10.102.240.162 (Vlan1438) is up: new adjacency

Jose Solano · ‎08-24-2016

Hi,

This is expected, please check this out:

Caution: When a module is inserted or removed, the switching bus can sometimes stall for about 3 seconds. This can disrupt the adjacencies in protocols such as Open Shortest Path First (OSPF), Border Gateway Protocol (BGP), or Multiprotocol Label Switching (MPLS) Label Distribution Protocol (LDP) if their timers have been configured for fast convergence.

Note: Do not remove or install more than one module at a time. The switch can bring only an identical replacement module online. If the replacement module is different from the removed module, you must configure it before the switch can bring it online.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/97065-oir-mod-catsw.html

HTH,

Leo

mmason221 · ‎08-24-2016

That write up is from 2007 and involved 6500's, makes no references to 6800 architectures. I am 100% positive the article correct because we have seen those events in 6500 arch's.

Cisco Live Wargo presentation I didn't notice anything being stated about OIR, or modules stalling bus.

http://d2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKARC-3465.pdf

6880-X C6880-X-16P10G OIR Gone Wrong!