Catalyst 9300 stack strange issue with 10gig interfaces

PACC · ‎09-17-2023

Hi all,

After successfully moving two C9300 stacked switches without powering them off or affecting the network... we had a power event which forced them off

Anyway, they powered back up ok and everything is running except a strange issue with the 10gig interfaces on the second switch.

The first switch (active) has the 1Gig interfaces as gi1/0/1, gi1/0/2 etc, and the 10gig as Te1/1/1, Te1/1/2 etc.

The second switch had that as well, but now the 10gig interfaces are Te2/0/29, Te2/0/30 etc. Only one of the 10gig was being used as a port channel, and the second switch has lost that configuration. When attempting to configure those ports I get:

core-switch(config)#int Te2/0/29
%ERROR: Standby doesn't support this command ^
% Invalid input detected at '^' marker.

Output of show switch detail:

Switch/Stack Mac Address : d4eb.683f.3900 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch#  Role     Mac Address     Priority Version State
-------------------------------------------------------------------------------------
*1      Active  d4eb.683f.3900     15        V05    Ready
2       Standby d4eb.6828.1100      1        V05    Ready

Stack Port Status Neighbors
Switch# Port 1 Port 2 Port 1 Port 2
--------------------------------------------------------
1        OK     OK     2       2
2        OK     OK     1       1

Can anyone advise what the issue is and how to resolve?

thanks

Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.3.4, RELEASE SOFTWARE (fc3)

Leo Laohoo · ‎09-17-2023

@PACC wrote:
int Te2/0/29

Unless switch 2 happens to be a 9300-48S, that port number is incorrect.

Post the complete output to the command "sh etherchan <PO #> summary".

PACC · ‎09-17-2023

Only shows the 10gig interface from the first switch:

Number of channel-groups in use: 8
Number of aggregators: 8

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
7 Po7(SU) LACP Te1/1/5(P)

Switch 2 definitely had the 10gig interfaces as Te2/1/X before the power cycle.

EDIT: both switches are C9300-24T

Interesting, when running sh inv it has the following for the sfp module:

NAME: "Te2/1/5", DESCR: "SFP-10GBase-SR"
PID: SFP-10G-SR , VID: V03 , SN: CLDT202112230067

Leo Laohoo · ‎09-17-2023

Post the complete output to the command "sh interface status m 2 | i Te2".

PACC · ‎09-17-2023

Thanks for the replies @Leo Laohoo

#sh interface status m 2 | i Te2
Te2/0/29 err-disabled 1 full 1000 1000BaseSX SFP
Te2/0/30 err-disabled 1 full 1000 1000BaseSX SFP
Te2/0/31 err-disabled 1 full 1000 1000BaseSX SFP
Te2/0/32 notconnect 1 full 1000 1000BaseLX SFP
Te2/0/33 connected 1 full 10G SFP-10GBase-SR
Te2/0/34 notconnect 1 full 1000 1000BaseLX SFP
Te2/0/35 notconnect 1 full 1000 1000BaseLX SFP
Te2/0/36 notconnect 1 full 1000 1000BaseLX SFP

Te2/0/33 is the only one that is connected. It was port channeled with Te1/1/5.

Leo Laohoo · ‎09-18-2023

Post the complete output to the command "sh interface status err".

PACC · ‎09-18-2023

#sh interface status err
Port Name Status Reason Err-disabled Vlans

Te2/0/29 err-disabled gbic-invalid
Te2/0/30 err-disabled gbic-invalid
Te2/0/31 err-disabled gbic-invalid

I've seen this before with non-Cisco sfp modules. Usually doing a shut/no shut on the interface activates them but I can't configure the interfaces at all.

PACC · ‎09-18-2023

The sfp modules in those Te2 ports are the same model as in Te1. After the power cycle I had noticed that they were err-disabled and I was able to re-enable the Te1 ports and that was when I noticed the strange thing with the Te2 ports.

PACC · ‎09-20-2023

I'm thinking that switch 1 has something wrong with its config. It tries to send the command int Te2/0/29 to switch 2 and it has no idea what that means as it has Te2/1/1, hence the error Standby doesn't support this command. Rebuilding the stack seems the thing to do I think.

I've never done this in production before, in fact yesterday was the first time doing anything stack related. We luckily have a couple of old C3750's with power and data stacking cables so I have been testing with these and have a plan. I ideally want switch 1 to remain functional at all times (but will do it during a maintenance window just in case, and I have a copy of the config). Switch 1 (current active/master) has stack priority 15 while switch 2 (standby) has priority 1.

Switch 2:

Unplug stack-power cables and power cable (switch 2 offline)
Unplug stack-data cables

Switch 1:

run command - no switch 2 provision (hopefully to remove dodgy config)
run command switch 2 provision c9300-24t
save config

Switch 2:

connect stack-data cables
connect power cable and stack-power cables
win?

PACC · ‎09-25-2023

No win. Still shows the switch 2 Te interfaces as Te2/0/XX.

I saw this as the stack was rebuilding:

ISSU-3-PEER_IMAGE_COMPATIBLE: Peer image (CAT9K_IOSXE), version (17.3.4) on peer uid (2) is incompatible

A show version has this:

Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
*    1 41    C9300-24T          17.03.04          CAT9K_IOSXE           INSTALL
     2 41    C9300-24T          17.03.04          CAT9K_IOSXE           INSTALL

It's been running for almost 2 years without issue (this is the first time it has restarted as far as I am aware).

EDIT: should I have saved the config on switch 1 after de-provisioning switch 2?

PACC · ‎10-02-2023

I powered off switch 2. On switch 1 I ran no switch 2 provision and saved the config. A show run and show conf only listed the switch 1 interfaces. I ran switch 2 provision c9300-24t and when I viewed the config, all the old switch 2 interface config came back, interface descriptions and all. Seems this does not delete the config for the switch 2 interfaces, it stays there in the background somewhere.

I powered on switch 2 (while disconnected from switch 1 and the network in general). It had errors about interface TenGigabitEthernet2/0/29 is an invalid command. However a show interface status showed the correct interface names.

Seems the config is corrupted. Has anyone heard of this and can advise on how to recover (hopefully in the least disruptive way possible)?

I had thought of renumbering and re-prioritising switch 2 while it is disconnected so it would become the active in a stack, and see if it sorts its interface names out in the config. If that works, reconfigure some ports and do a cable swap from old switch 1 to new switch 1. Then renumber and re-prioritise old switch 1 so it would become the standby. How does that sound, or is there perhaps an easier method?

MihaiN · ‎10-03-2023

Hello, I had noticed similar strange behaviors on stacked switches that have different firmware versions on them, it's worth checking

PACC · ‎10-03-2023

Hey @MihaiN

If I do a show version I get:

Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 41    C9300-24T          17.03.04          CAT9K_IOSXE           INSTALL
     2 41    C9300-24T          17.03.04          CAT9K_IOSXE           INSTALL

They appear to have the same firmware.

It also has:

BOOTLDR: System Bootstrap, Version 17.6.1r[FC2], RELEASE SOFTWARE (P)

But only one entry. I assume this is only for the active switch. If the standby switch had a different BOOTLDR image would that cause this issue?

PACC · ‎10-04-2023

Looking at switch 2 (standby) while it is disconnected from everything, I can see that its startup config has the dodgy interface names, however its running config has the correct interface names! Writing its running config to startup should solve the issue, but would require it becoming the active.

So how does my plan sound:

power off switch 2 and disconnect all stacking cables
renumber and re-prioritise 2 so it will be elected active (save config and reload)
network cable swap from switch 1 to 2 (brief network outage)
ensure everything works
renumber and re-prioritise 1 so it will be elected standby
power off switch 1 (now 2) and connect stacking cables
power on switch 1 (now 2) and hope for the best...