cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
197
Views
3
Helpful
16
Replies

C9350 Stack Ring Closure Caused Stack Reload – Expected Behavior?

athan1234
Level 8
Level 8

Hi everyone,

I experienced an issue yesterday with a Cisco C9350-48T stack running IOS XE version 17.18.03a.

The stack consists of 7 members. During the initial installation, I was unable to fully close the stack ring because the available stacking cable was not long enough. As a result, the stack has been running in production with an open ring topology.

Yesterday, I attempted to close the ring by connecting a new 1-meter stack cable. However, as soon as I connected the cable to the first switch, the entire stack reloaded unexpectedly.

Is this normal behavior when closing a stack ring on a live production stack?

I am aware that Cisco recommends performing stack cabling and ring formation while the stack is powered off, but I would like to understand whether a stack reload is expected in this scenario or if this could indicate another issue.

Has anyone experienced something similar?

Thanks in advance.

16 Replies 16

Mark Elsen
Hall of Fame
Hall of Fame

 

  - @athan1234               (Added 😞 for future similar operations you can for instance attach a console to the
                                                      stack master and check the console messages when there is a crash.
                                                      PuTTY (e.g.) allows you to connect to the console and save all output
                                                      to  a file (e.g.)
                                        Configure a syslog server on the stack master, and examines logs send to it
                                        after you attempt to connect the cable, look at 'last breath' logs 
                                       
 before the stack reloaded to possibly get additional information

  M.



-- Let everything happen to you  
       Beauty and terror
      Just keep going    
       No feeling is final
Reiner Maria Rilke (1899)

balaji.bandi
Hall of Fame
Hall of Fame

It's not, but if you have made a small mistake, it happens (you will not be able to see any logs, since the switch rebooted and until you have captured any console logs or a generated crash log).

 

BB

=====️ Preenayamo Vasudevam ️=====

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Is it possible that this version has a bug related to a 7-member stack? Or is it mandatory to power off the entire stack before closing the ring?

I only connected the stacking cable to member 1, and as soon as I plugged it in, the whole stack reloaded.

Joseph W. Doherty
Hall of Fame
Hall of Fame

@athan1234 wrote:

I am aware that Cisco recommends performing stack cabling and ring formation while the stack is powered off, but I would like to understand whether a stack reload is expected in this scenario or if this could indicate another issue.


I wouldn't say that a stack reload is expected but the reason for the power off recommendation is to avoid such unpleasant surprises.

Cannot say for sure regarding 9k stacks, but I recall (?) the classical recommendation was to never stack cable interconnect two powered on stack members.  No need to power off the all the stack members, just power off one of the two members to be interconnected.  (If one of those is the stack master, power off the other stack member.)

BTW, you may not have needed the longer cable.  Often you can dual interconnect all stack members without using the "traditional" daisy chain and a top/bottom connection.  See https://community.cisco.com/t5/networking-blogs/connecting-stack-cables-a-better-way/ba-p/4109155

We have 9300s and we have broke the ring and added switches without issues with the system running. The only difference is ours was a full ring on boot, is there a bug or issue with it booting without a connection not liking a connection being made? Even so it should detect and prompt an error or not use until a reboot, not a stack reboot. Not sure if there is a difference with the 9350s, but if you have spares I would experiment to see if you can recreate.

pieterh
VIP
VIP

>>> As a result, the stack has been running in production with an open ring topology <<<
this is not correct!
if a stack port has no cable connected, the ring will be closed internally in the switch.
so you still have a stack ring,
the switch that has single stack connection will operate at half the bandwith 

source: https://www.ciscolive.com/c/dam/r/ciscolive/global-event/docs/2025/pdf/BRKARC-2668.pdf

pieterh_0-1784038023045.png

 

 

@pieterh that's a very, very interesting diagram from the PDF you've referenced (on page 62).

I wonder, perhaps, if some junior Cisco engineer, saw that two stack ports vs. one stack port, doubles bandwidth?

Un-huh, so only switches 1 and 3 bandwidth is halved, eh?  Hmm, walk the path between any two switches, and see what's the bandwidth actually available between them!

Between switches 2 and 4, there's only one link between them!  I.e. logically there is now a daisy chain, no longer a ring.  Switch 2's other stack port only leads to switch 1, while switch 4's other stack port only leads to switch 3.  There's no second path!  Indeed, it's now an open ring, again, it's now a daisy chain.

I would really like to see a better explanation how bandwidth between any two switch members is not halved when there's a down stack connection between stack members, anywhere in the stack.

Yes, it's true, for example, stack members 2 and 4 can exchange data with two other stack members while stack members 1 and 3 can only exchange data with one other stack member, but stack members 2 and 4 do NOT have two stack ports bandwidth to each other!

I believe earlier stacks operated the same way.

Also, interestingly, on the prior page of the referenced PDF, shows how stack member can interconnected.  It's sort of implied this is something new with the C9350, but it's not.

As a side note, when Cisco introduced StackWise, with the 3750 series, it was advertised as 32G.  Well, as best I could tell, each stack port was actually 8G, duplex.  Cisco didn't actually provide much detail on how the stack ring operated.

However, with the introduction of the 3750E series, and its StackWise+, they noted it was now 64G (I believe each stack port was now 16G, duplex).  They also noted a couple of "improvements", besides bandwidth, with StackWise+.  Firstly, they now described, at least for unicast traffic, it was only placed on the ring, when it needed to be placed on the ring.  (I.e. this implying the original StackWise, flooded all traffic on the stack ring, even if between two switch ports on the same physical switch member.)  Secondly, the destination switch removed unicast traffic, from the ring, "spatial reuse".  (I.e. implying the original StackWise source switch removed traffic, that it placed on the ring, after it fully transited all the other switches!)

Until encountering the StackWise+ enhancements, it was unclear how yucky original StackWise was, at least in stack ring bandwidth management.  It's certainly possible, there may be improvements in StackWise, for 9Ks, possibly specifically for C9350s, but it's unclear how page 62 is actually an improvement.  (Although, also on page 62, mention is made of packets being fragmented and sent across both ring paths, when available.  Yet the C9350 isn't impacted the same way with the ring being broken!?)

BTW, I only mention the 3750 and 3750E, because, Cisco, like other sellers, often touts "features" that aren't much of a feature, at least when it comes to performance.  Back in the 3750 day, it only had FE edge ports (the later 3750G had gig), and Gig was still rather novel, so a 32G ring, was wow!  (But was it really?  Laugh, another StackWise sales "feature" was when a max stack member stack, sum of all its stack ports' bandwidth, was touted.)

Caveat emptor.

Hi @Joseph W. Doherty 

as I understand the architecture EACH port has multiple eastbound and westbound datapaths
as I said before when  a stack port is  "not connected" , the ring is closed internally in the switch (between easbound and westbound datapath), this results in what you call a daisy-chain is actually a ring.

about bandwith

pieterh_0-1784057922499.png

 

pieterh_1-1784058029238.png

 

 

 

Ah, using multiple physical stack ring links isn't new, believe it was introduced with the 3650/3850 series.

It does provide additional bandwidth, and very possibly, if just one of the internal ring paths is broken, remaining bandwidth likely used and ring not broken.  (Possibly think of this as a Etherchannel bundle.)

However, if physically the stack is totally without one stack port interconnection between a stack pair, walk me through how a ring is provided.

If removing one stack connection does NOT break the ring, and if you don't have a daisy chain, what other stack member stack connection can we remove and not partition the stack?

I came across another Cisco PDF (https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2026/pdf/BRKENS-2504.pdf) which touches upon the new stacking features of the C9350.  It too describes how it's better then older StackWise, when a stack port to port path fails.  This newer stack management does appear to offer features not available (like SPF [BTW available on brand J stacks almost a decade ago]) and in the case of a totally failed stack port to port failure, the negative impact might be managed better, but I still see the overall stack bandwidth should be decreased, i.e. physical ring is broken.

Again, @pieterh , if you can explain or document this is not the case, I'm open to learning.

Oh, looking at Cisco C9350 stack cable installation videos, a single stack cable appears to comprise 3 distinct cables that connect to the physical stack plug.

aleabrahao
Meraki Community All-Star
Meraki Community All-Star

@athan1234 

A full stack reload is not considered normal or expected behavior simply because you close an open StackWise ring on a running Catalyst 9300 stack.

The reload could have been caused by various factors. I suggest you take a look at this document:
https://www.cisco.com/en/us/support/docs/switches/catalyst-9300-series-switches/217684-verify-and-troubleshoot-stackwise-on-cat.html

I am not a Cisco employee. My suggestions are based on documentation of Meraki best practices and day-to-day experience.

Please, if this post was useful, leave your kudos and mark it as solved.

 


@aleabrahao wrote:

The reload could have been caused by various factors. I suggest you take a look at this document:
https://www.cisco.com/en/us/support/docs/switches/catalyst-9300-series-switches/217684-verify-and-troubleshoot-stackwise-on-cat.html


When I try your link, I get:

JosephWDoherty_0-1784041404279.png

 

aleabrahao
Meraki Community All-Star
Meraki Community All-Star

That's strange, try it again please.

Verify and Troubleshoot Stackwise on Catalyst 9200/9300 - Cisco

I am not a Cisco employee. My suggestions are based on documentation of Meraki best practices and day-to-day experience.

Please, if this post was useful, leave your kudos and mark it as solved.

That one works.

PS:

From that reference:

Add or Replace a Member

Several factors must be considered when you implement a new stack or add a member to an established stack.  Importantly, never connect a powered-on switch into a powered-on stack. Connect new member(s) while powered-down to avoid a stack-merge.  These are other points to consider: