Re: Catalyst c1000 - 15.2.7E9 broke my network

chrismbrook · ‎10-16-2023

What a complete nightmare! I got an email notification that a new software package is available for my Catalyst C1000 8P-2G-L switches.

I have two identical switches stacked using fibre. After the upgrade the first switch continued to work as normal but the second switch has gone offline.

I've tried absolutely everything I can with switch 2 but no joy. I can console into it, everything boots normally but zero communication. I think the problem is something to do with the stack config......

Even if I delete the config.text and vlan.dat and reboot, the next reboot takes forever as it is waiting for the stack. Eventually, it times out and boots but no communication.

I've tried doing the zero-day procedure, but nothing works. I'm not sure what the 15.2.7E9 software is but its completely destroyed my very stable network.

I've downgraded my switch 1 to the previous 15.2.7E8 build which has been rock stable since it came out. I am currently using XMODEM to transfer (very slowly) 15.2.7E8 to switch 2 in the hope that it comes back to life.

Maybe the software upgrade went bad for switch 2? Not sure. Its never done this before when upgrading the software. I think I've done 3 or 4 software upgrades in stack previously and all have worked without a hitch. Not sure why this one is different.

Think I'll leave the software alone in future. I've spent 5 hours trying to sort switch 2 out now - and finally gave up, resorting to XMODEM.

Surely, if I delete the config.text from switch 2 and reboot it should boot with no config - but why does it keep looking for 'master election' and take forever to boot?

Leo Laohoo · ‎10-16-2023

Post the bootup of Switch 2.

How was the IOS upgraded? Did someone "copied" the BIN file from one switch to another (instead of using the TAR file)?

chrismbrook · ‎10-16-2023

The IOS was upgraded by using;

archive download-sw /overwrite /reload tftp://192.168.5.1/c1000-universalk9-tar.152-7.E9.tar

I can't post bootup of switch 2 as I'm still waiting for XMODEM to finish.

balaji.bandi · ‎10-16-2023

Ca1K have USB Port, wny not use that and upgrade or downgrade ?

Surely, if I delete the config.text from switch 2 and reboot it should boot with no config - but why does it keep looking for 'master election' and take forever to boot?

i am in guess if the switch number is 2 it always look for master switch, if this was not connected to master switch in stack that boot normally.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

chrismbrook · ‎10-16-2023

I've never been able to get the USB ports to work on these C1K switches. I've always had to rely on TFTP server running from my pfSense router.

I've come to realise 90% of the Cisco documentation does not work on the C1K series. These things are great when working correctly, but any problems you will be frustrated by the incorrect documentation supplied by Cisco.

From what I've read - the C1K is a stripped down version with regards to software. I'd of chosen very differently if I'd of known that.

Leo Laohoo · ‎10-16-2023

@chrismbrook wrote:
the C1K is a stripped down version with regards to software.

I have received any info about this fact. As far as I am aware, IOS for the Catalyst 1000 is the same "lineage" as 2960S, 2960X and IE4000. The base IOS is the same and nothing is "stripped down" in regards to functionality.

@chrismbrook wrote:
I've never been able to get the USB ports to work on these C1K switches.

Try a different brand of USB flash drive. It is imperative to have, at least one, brand that is able to work for situation like this.

chrismbrook · ‎10-16-2023

OK - everything is now sorted.

I am running both switches on the newer E9 software and both switches are behaving correctly.

I decided to remove the H-Stack functionality from my environment. Both switches are running independently with their own IPs.

H-Stack is a great idea in principle, but I have a funny feeling Cisco have put very little into their implementation on these C1K units.

For the first time since owning these switches I can actually see the correct colour coding of each port in the Web GUI. Something that is also very broken using H-Stack.

It feels somewhat like I am a alpha tester for Cisco. You have to find out the problems, and find your own solution for them

Thank the world for XMODEM and coffee. It's literally taken me about 5 hours to get the switch2 back online and running correctly.

StevieB · ‎01-29-2024

I have an open ticket with Cisco support and they've confirmed this issue exists on E9 but only on the 8-port model. They weren't able to reproduce using larger c1000 switches.

The issue appears to be that the SFP won't initialize during the boot sequence before the switches try and discover each other using the fiber link in E9.

Booting in E8 you can see the connected SFP's assigned to the H-Stack light up and start blinking during discovery, and the switches immediately see each other. A stack is then formed. This unfortunately won't happen in E9 with the C1000-8T-2G-L as no SFPs inserted in this model using this version will initialize during the boot phase, so there is no possibility of the 8-port switch communicating with other members of the stack during discovery. I've tried GLC-SX-MM, GLC-SX-MMD, and GLC-LH-SMD SFPs and they all behave the same.

From my experience, upgrading a stacked 8-port c1000 to E9 from any prior version will result in that stacked-member losing connectivity with the rest of the stack, and it will become it's own master while incapable of controlling the others. You will see the H-Stacked SFPs light up after the switch(es) boot, but the lights remain solid as no data is being sent over the fiber. Normal behavior of H-Stacked SFPs are a fast and constant blinking while information is exchanged between stacked members, but this will not happen in E9 on 8-port switches.

At the time of writing this E9 is still the MD and I don't see a bug in the tracker. Hopefully that changes soon.

Leo Laohoo · ‎02-03-2024

If this bug/feature exist in E9, how about E8 or E7?

StevieB · ‎02-05-2024

E8 and E7 have no issues stacking the 8-port c1000. The issue first appeared in E9.

chrismbrook · ‎03-16-2024

I see E10 has just been released on 15th March (my birthday). Not able to see any release notes so no idea what E10 brings.

Does anyone know of a more reliable place to track release notes?

meetriks · ‎03-18-2024

Same here, i can't find the E10 release notes. Strang that the stacking problem of 8 ports with E9 is not on the bug list.

Cisco IOS Software, C1000 Software (C1000-UNIVERSALK9-M), Version 15.2(7)E10, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2024 by Cisco Systems, Inc.
Compiled Tue 12-Mar-24 08:26 by mcpre

ROM: Bootstrap program is c1000 boot loader
BOOTLDR: C1000 Boot Loader (C1000-HBOOT-M) Version 15.2(7r)E3, RELEASE SOFTWARE (fc4)

StevieB · ‎03-18-2024

The bug referenced by TAC is here:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwi79812

The linked page is not showing any affected or fixed releases for some reason.

I was told by TAC that E10 would fix the issue. Hopefully that will be the case.

StevieB · ‎03-19-2024

I just installed E10 (from E8) and it doesn't break my C1000-8T-2G-L stack like E9 did.

I have an open question with TAC as to why the bug appears categorized incorrectly (under Cisco Catalyst 9300 Series Switches).

It does appear the bug is now correctly associated with the Known Affected Releases and Known Fixed Releases.

meetriks · ‎03-19-2024

Many C1000 bugs are wrongly classified.

i google on the following:
c1000 site:bst.cisco.com

to find c1000 bugs. And a lot can be found…

still no release notes online which is anoying