Strange SwitchStack Problem

coreyhelms · ‎06-29-2011

I have a switchstack that is flapping between 16G and 32G.

I did a "show switch detail" and the "stack port status" section points to a problem between switch #1 and #3. I replaced the stack cable and the problem persists.

I'm a bit confused by what the "neighbors" section is telling me though. If switch #1 & #3 are flapping, then why is #2 telling me it sees no neighbors?

Here are four outputs from "show switch detail". These four were taken within a 30-second period.

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok 2 3

2 Ok Ok None None

3 Down Ok None None

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Down None None

2 Ok Ok None None

3 Down Ok None None

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok None None

2 Ok Ok 3 1

3 Down Ok None 2

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok 2 3

2 Ok Ok 3 1

3 Ok Ok 1 2

Reza Sharifi · ‎06-29-2011

Do you see any error messages in the logs?

what is the output of "sh swi stack-ring speed"?

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/switch_stacks.html#wp58952

coreyhelms · ‎06-29-2011

The logs seem to indicate that Port #2 of Switch #1 and Port #1 of Switch #3 have a problem.

The ring-speed is flapping from 16G to 32G.

I have replaced the cable.

I guess my real question is this: "Why does "show switch detail" show that switch #2 doesn't see switch #1 or #3 sometimes?"

Could there be a brief period between the 16/32 flap that is causing switch #2 to freak out?

Reza Sharifi · ‎06-29-2011

The only time I have seen flapping between 32 and 16 is when the stacking cable is loose or some thing is wrong with the pins. I know you have changed the cable, but maybe it is a good idea to replace all the stacking cables and see if there is a change. Are these E or X series switches?

coreyhelms · ‎06-29-2011

WS-C3750-48P

I replaced all the cables.

I mean, there isn't a whole lot to troubleshooting the stack. It's either the cable or the switch. I've replaced the cables, so now it's just figuring which switch (#1 or #3) to replace.

I guess I could add another switch to the stack betwen #1 and #3 and then see if the problem stays on the #1 or #3; that'll isolate the bad backplane.

But, like I said, it isn't so much *what* I should replace; that I can figure out. It's more of a, "hey, this is weird" kinda thing.

I would think that a flapping port between #1 and #3 would not cause #2 to have a problem. And #2 may *not* actually have a problem. But what the console is telling me is that the flap is causing problems across the entire stack.

Reza Sharifi · ‎06-29-2011

The other option to look at is the IOS version. I would load a different version before start adding and replacing switches.

Who know, it could be bug.

Leo Laohoo · ‎06-29-2011

The only times I see outputs similar to what you posted is when you have a faulty stack cable or stack cables that are not screwed in securely or a faulty stack backplane.

However, that being said, to see alot of them happening to all your switches would suggest that the stack cable are not screwed in properly. One millimeter short can cause issues like this.

coreyhelms · ‎06-29-2011

As I said before, I have replaced all three cables.

I think what I'd really like to know is why does Sw#2 seem to lose contact with its neighbors?

I'll probably open up a TAC tomorrow and send the full details to Cisco.

If other people with a bad cable or interface have seen the same behavior, then maybe this is normal. I just don't want to start replacing switches until I know *exactly* what I'm seeing and why.

I guess I should create a *known* bad switchstack cable and do some testing in my float rack. I can't really touch the production network unless I know more about why the #2 switch is reporting a loss of contact with its neighbors...

Leo Laohoo · ‎06-29-2011

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok 2 3

2 Ok Ok None None

3 Down Ok None None

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Down None None

2 Ok Ok None None

3 Down Ok None None

Switch #2 (or the middle switch) is not the issue, I hope. Make sure the stack cables are properly screwed in. I usually use a #2 s_crew driver to drive the s_crews.

coreyhelms · ‎06-30-2011

I went out there and replaced the cable again. The port is still flapping.

So, no one is bothered by Sw#2 *sometimes* reporting that it has no neighbors?

Reza Sharifi · ‎06-30-2011

Try a different IOS image.

j.t.faust · ‎10-10-2011

I am seeing what could be the same problem on a six switch 3750 stack running 12.2(52)SE IP Base. Replaced a stack cable with a brand new one, but still stuck in 16G mode and stack neighbors are up/down, changing every few seconds. Opening a TAC case on it now. Did you correct the problem?

Ted

glen.grant · ‎10-10-2011

I have found that these cables can be very flakey , while they may seem tight by hand sometimes you have to use a screwdriver on them just dont heave on them too hard and strip the screws . Check the cable setup , the version we use is always start at the top from the top switch port 1 go to switch 2 port 2. From switch 2 port 1 go to switch 3 port 2 and so on down the stack and the bottom switch cable will go back up to the top switch por that is open.

lirinpatel0829 · ‎05-06-2014

Has anyone found a resolution to this.. We are having the same issue with a 8 switch stack... almost all stack ports are bouncing.. and Ring Speed going from 16-32..

Lirin