interface reset on C2960X stack cause the switch to reload.

[jgustafzon]

I've recently implemented 802.1x on a stacked C2960X and ever since, as soon as I reset an interface (default interface GigabitEthernet 1/2/3) in order to remove the 802.1x config, the switch drops vty connection for a short while, everything on the switch drops connection and the switch seems to reboot itself and lose the master role to the stacked switch. I'm not sure why it would do that.

Interface config:

interface GigabitEthernet2/0/46
switchport access vlan 21
switchport mode access
authentication event fail action authorize vlan 61
authentication event no-response action authorize vlan 61
authentication port-control auto
authentication periodic
dot1x pae authenticator
dot1x timeout tx-period 2
dot1x max-reauth-req 1
spanning-tree portfast

This was seen in the log after an interface reset and connection was lost for a short while:

Sep 3 13:21:10.808: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY
Sep 3 13:21:10.902: %PLATFORM-6-FLEXSTACK_INSERTED: FlexStack module inserted in Switch 2.
Sep 3 13:21:11.123: %STACKMGR-5-SWITCH_READY: Switch 2 is READY
Sep 3 13:21:12.570: %USB_CONSOLE-6-MEDIA_RJ45: Console media-type is RJ45.
Sep 3 13:21:12.577: %PHY-5-TRANSCEIVERINSERTED: Slot=2 Port=49: Transceiver has been inserted
Sep 3 13:21:16.121: %CFGMGR-4-SLAVE_WRITING_STARTUP_CFG: only master can do that
Sep 3 13:21:16.121: %CFGMGR-4-SLAVE_WRITING_STARTUP_CFG: only master can do that
Sep 14 05:54:26.151: %STACKMGR-4-SWITCH_REMOVED: Switch 1 has been REMOVED from the stack
Sep 14 05:54:26.158: license_init_config_queue: initializing the license_switch_boot_config_q
Sep 14 05:54:26.165: %STACKMGR-4-MASTER_ELECTED: Switch 2 has been elected as MASTER of the stack
Sep 14 05:54:26.459: %CFGMGR-6-APPLYING_RUNNING_CFG: as new master
Sep 14 05:54:26.553: %SSH-5-ENABLED: SSH 1.99 has been enabled

Today this was seen:

Sep 19 11:10:56.811: %PM-3-INTERNALERROR: Port Manager Internal Software Error (!pm_vlan_test_portlist(vlan, pd->globalNumber): ../switch/pm/pm_vlan.c: 1743: pm_vlan_add_port) (Sw-K015-1)
-Traceback= 5ABD50z 30D0A1Cz 3149828z 313C224z 24823F0z 312E830z 2482388z 3125EC0z 313268Cz 24823F0z 312C568z 2482388z 3125FC8z 311DAC8z 313315Cz 24823F0z (Sw-K015-1)
Sep 19 11:10:56.811: %PM-3-INTERNALERROR: Port Manager Internal Software Error (!pm_vtpvlan_bitlist_test(&pd->operInfo.trunkVlans, vlan): ../switch/pm/pm_vlan.c: 1749: pm_vlan_add_port) (Sw-K015-1)
-Traceback= 5ABD50z 30D0A1Cz 314987Cz 313C224z 24823F0z 312E830z 2482388z 3125EC0z 313268Cz 24823F0z 312C568z 2482388z 3125FC8z 3

I will go ahead and update to recommended software since this might be related to a bug. Current version 15.2(3)E

[Daniele Giordano · Accepted solution]

Hi, check the logs in flash (show flash). If you have crash info files the switch reloaded due to a system crash.

In this case you can check cisco bug toolkit to find a releted bug or open a case to TAC.

If you want you can test one of the latest IOS 15.2.4, 15.2.5 or 15.2.6.

Regards.

[jgustafzon]

I did find a crash log with

more flash1:crashinfo_ext/crashinfo_ext_1

I will try upgrading firmware. 

CMD: 'show mac address-table ' 07:53:48 CET Fri Sep 14 2018
CMD: 'conf t' 07:54:08 CET Fri Sep 14 2018
CMD: 'default interface gi2/0/2' 07:54:23 CET Fri Sep 14 2018
CMD: 'interface GigabitEthernet2/0/2' 07:54:23 CET Fri Sep 14 2018
CMD: 'default spanning-tree portfast' 07:54:23 CET Fri Sep 14 2018
CMD: 'default dot1x max-reauth-req 1' 07:54:23 CET Fri Sep 14 2018
CMD: 'default dot1x timeout tx-period 2' 07:54:23 CET Fri Sep 14 2018
CMD: 'default dot1x pae authenticator' 07:54:23 CET Fri Sep 14 2018
CMD: 'default authentication periodic' 07:54:23 CET Fri Sep 14 2018
CMD: 'default authentication port-control auto' 07:54:23 CET Fri Sep 14 2018
CMD: 'default authentication event no-response action authorize vlan 61' 07:54:23 CET Fri Sep 14 2018

=== Start of Crashinfo Collection (07:54:23 CET Fri Sep 14 2018) ===

For image:
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.2(3)E, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.

[Leo Laohoo]

That's nice, however, the firmware of the switch needs to be upgraded.

[jgustafzon]

Upgrading to a latest stable version solved the issue.

[Leo Laohoo]

Current IOS version is a plain "E" (aka "E0"). I'd upgrade the firmware first.

[ysotocob]

I found an internal defect that might be the root cause of the crash event.

The documentation is still not ready to release the notes to the public, however in case that this bug is the root cause you might need to perform an IOS upgrade to the latest version of 15.2(5)E or 15.2(6)E trains.

In case that the crash persist in the new version you might need to open a ticket with Cisco TAC including the "show tech" and the latest crashfiles.