We have a stack of 7 c3650-48TD.
There are an increasing number of ports that when they have been inactive for a while, refuse to come back up when a client is connected.
On some suspected faulty ports, the indicator light on the front of the switch goes green, but there is no traffic flowing.
On others the light will go amber suggesting they are blocked by STP. However, looking at the state of that specific port, it doesn’t not show it to be in an err-disabled state.
I noticed these error messages appear on the console while unplugging, then plugging in, a cable to a suspect port:
*Apr 20 14:52:28.543: %SPI_FC-3-HIGH_WMARK_REACHED: Number of messages in the queue for channel 1 has reached maximum.
-Traceback= 1#58bc7daf4fe087af80f5f05af9e15bb7 :545E8000+1FB56C4 :545E8000+12B3A38 :545E8000+12B3C2C :545E8000+B17AD8 :545E8000+B198BC :545E8000+B1BE38 :545E8000+CC8C64 :545E8000+CD9230 :545E8000+9CF3C8 :545E8000+B0F390 :545E8000+B18C58
*Apr 20 14:52:28.554: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_PROTECTED, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:52:28.554: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_UK_MCAST_BLK, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:52:28.555: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_UK_UCAST_BLK, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:52:28.556: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:VP_STP_STATE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:52:29.553: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/10, changed state to down
*Apr 20 14:52:30.553: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/10, changed state to down
*Apr 20 14:52:30.554: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_ARP_SNOOP, Targets: 0x4, spi-fc ec: 3
stack#
stack#
stack#
*Apr 20 14:53:08.315: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_MODE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.316: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_DTP_MODE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.316: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_MODE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.316: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_DTP_MODE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.316: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_MODE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.316: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_PAGP_MODE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.317: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_PAGP_MODE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.317: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_PROTECTED, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.317: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_UK_MCAST_BLK, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.317: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_UK_UCAST_BLK, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.317: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_MODE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.318: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:VP_STP_STATE, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:08.320: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:E_PORT_SET_ALL_VLANS, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:10.320: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/10, changed state to up
*Apr 20 14:53:10.321: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:PORT_ARP_SNOOP, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:11.324: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/10, changed state to up
*Apr 20 14:53:12.320: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:E_PORT_SET_ALL_VLANS, Targets: 0x4, spi-fc ec: 3
*Apr 20 14:53:16.311: %PLATFORM_PM-0-SPI_FC_FAIL: SPI-FC Failure while sending PM SPI msg. If:Gi2/0/10, Msg Type:E_PORT_SET_ALL_VLANS, Targets: 0x4, spi-fc ec: 3
stack#
....
...
This stack is still running on software version 3.03.04SE, and from my quick research, this may be related to a bug in this version?
But am hoping there is another solution that we have missed which will avoid us having to reboot the stack
