Cat9500-32C SPA removed from chassis

[rwesel]

Hello,

we have 2 Cat9500-32C in Stackwise Virtual mode. I don't know exactly what it triggers, but whlie removing and inserting transceivers, the switch disables all ports on one chassis.

Both switches are still "visible" and in redundancy mode sso but all the ports on one chassis are removed from config. Hence the ports can not enabled again. A reboot helps but this could not be the solution.

these are the logging messages:

Sep 25 14:38:06 switch1 549: Sep 25 14:38:04: %TRANSCEIVER-6-REMOVED: Chassis 2 R0/0: iomd: Transceiver module removed from HundredGigE2/0/29
Sep 25 14:38:18 switch1 550: Sep 25 14:38:17: %TRANSCEIVER-6-INSERTED: Chassis 2 R0/0: iomd: transceiver module inserted in HundredGigE2/0/29
Sep 25 14:43:27 switch1 551: Sep 25 14:43:26: %SFF8472-3-READ_ERROR: Hu2/0/29: Error reading DOM data from transceiver
Sep 25 14:44:39 switch1 552: Sep 25 14:44:37: %TRANSCEIVER-6-REMOVED: Chassis 2 R0/0: iomd: Transceiver module removed from HundredGigE2/0/29
Sep 25 14:45:15 switch1 553: Sep 25 14:45:14: %TRANSCEIVER-6-INSERTED: Chassis 2 R0/0: iomd: transceiver module inserted in HundredGigE2/0/29
Sep 25 14:45:52 switch1 554: Sep 25 14:45:51: %TRANSCEIVER-6-INSERTED: Chassis 2 R0/0: iomd: transceiver module inserted in HundredGigE2/0/25
Sep 25 14:45:59 switch1 555: Sep 25 14:45:58: %TRANSCEIVER-6-REMOVED: Chassis 2 R0/0: iomd: Transceiver module removed from HundredGigE2/0/25
Sep 25 14:45:59 switch1 556: Sep 25 14:45:59: %IOSXE_OIR-6-SPARELOAD: SPA reloaded on subslot 5/0
Sep 25 14:45:59 switch1 557: Sep 25 14:45:59: %IOSXE_OIR-6-REMSPA: SPA removed from chassis 2 subslot 1/0, interfaces disabled
Sep 25 14:45:59 switch1 558: Sep 25 14:45:59: %SPA_OIR-6-OFFLINECARD: SPA (C9500-32C) offline in chassis 2 subslot 1/0
Sep 25 14:46:00 switch1 559: Sep 25 14:45:59: %PMAN-3-PROCHOLDDOWN: Chassis 2 R0/0: root: The process iomd has been helddown (rc 139)
Sep 25 14:46:01 switch1 560: Sep 25 14:46:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel102, changed state to down
Sep 25 14:46:01 switch1 561: Sep 25 14:46:01: %LINK-3-UPDOWN: Interface HundredGigE2/0/1, changed state to down
Sep 25 14:46:01 switch1 562: Sep 25 14:46:01: %LINK-3-UPDOWN: Interface HundredGigE2/0/2, changed state to down
Sep 25 14:46:01 switch1 563: Sep 25 14:46:01: %LINK-3-UPDOWN: Interface HundredGigE2/0/3, changed state to down
Sep 25 14:46:01 switch1 564: Sep 25 14:46:01: %LINK-3-UPDOWN: Interface HundredGigE2/0/4, changed state to down
Sep 25 14:46:01 switch1 565: Sep 25 14:46:01: %LINK-3-UPDOWN: Interface HundredGigE2/0/5, changed state to down
....

the virtual stack seems to be ok

switch1#show switch 
Switch/Stack Mac Address : xxxx.xxxx.xxx2 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State 
-------------------------------------------------------------------------------------
*1 Active xxxx.xxxx.xxx1 1 V02 Ready 
2 Standby xxxx.xxxx.xxx2 1 V02 Ready 

Cisco says: "No action is required." -  Hmmm

Any ideas to avoid this situation or to activate the disabled chassis again without reboot ?

Thanks

[balaji.bandi]

Since these are new gears this problems you have observed may not be as usal as expected in the Live envrironment.

Cisco TAC should able to give you more information, why this was caused the issue - i have done many remove and insertion today after you reported on my SVL lab switch, i can not replicate  your issue.

i am running 16.12.X for your reference.

[akelenyi]

I'm experiencing a similar issue...do you guys have any update on this ...by any chance?

[DanielP211]

Hello!

Had exacly the sam issue. Cisco TAC did a RMA of my switch.

Their answer was:

We can see that the CPU hog was called due to some issue involving an IOMD operation, more specifically, reading optics data.

Some of this reading, on the interface, was giving unexpected/error values, that could cause the CPU hog to be called.

For example, SFP being removed/flapped, causing power readings to be incorrect/unexpected.

This points to hardware failure on the switch, so we have to replace the effected switch

BR

[roland.stumpner@fh-ooe.at]

Hello , 

Had a similar issue with version Cisco IOS XE Software, Version 17.03.05  . The Resolution was remove all sfps and reconnect carefully one by another to find a faulty one and replace it.

hope this was a usefull hint