Blocked Stack Ports on 2960X-48FPD-L Stack (Unstable Switch Stack!) Spanning Tree?

[NPT_2]

I am having an issue where 2 2960X-48FPD-L Switches in a redundant flexstack (stack port 1 SW1 to port  2 SW2 and port 2 SW1 to port 1 SW2) ring. 

At first running the 15.0(2).EX5 (and earlier EX3, and EX4) version IOS yielded all the ports on the stack master switch refusing to run spanning tree and would only link in amber and not pass any traffic other than CDP information (the slave switch linked in fine). 

I upgraded to 15.2(3)E and this solved the problem of the ports not linking in green and participating in spanning tree. 

Now, however, about every week or two I lose connectivity to the switch stack and I was able to go to the switch stack locally and found that for some reason the switch stack is blocking and unblocking VLANs on StackPort1 frequently (see below).  When I was at the site, I sometimes had connectivity, sometimes not.  A stack hard reboot brought everything back up, but this is the second time this has occurred and I would expect the same problem in the next week or so. 

Has anyone else run into these issues, and have you found a solution?

I'm guessing that if I either get rid of the redundancy on the switch stack or stack using Ethernet cables between switches the problem will go away, but then what is the point of using stackable switches in a non redundant low speed stack.  It seems to me that Spanning tree thinks that I have a spanning tree loop going on with the stack ports which I didn't even think was possible.   

What do you think?

Jim

_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:02:59: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking StackPort1 on VLAN0307. Port consistency restored.
Mar 11 09:03:16: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:03:27: %SPANTREE-2-BLOCK_PVID_PEER: Blocking StackPort1 on VLAN0307. Inconsistent peer vlan.
Mar 11 09:03:42: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking StackPort1 on VLAN0307. Port consistency restored.
Mar 11 09:03:46: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:03:47: %SPANTREE-2-BLOCK_PVID_PEER: Blocking StackPort1 on VLAN0307. Inconsistent peer vlan.
Mar 11 09:04:12: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking StackPort1 on VLAN0307. Port consistency restored.
Mar 11 09:04:22: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:04:56: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:05:13: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 207 on StackPort1 VLAN307.
Mar 11 09:05:13: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking StackPort1 on VLAN0307. Inconsistent local vlan.
Mar 11 09:05:30: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:06:00: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:06:04: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking StackPort1 on VLAN0307. Port consistency restored.
Mar 11 09:06:32: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:07:02: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:07:03: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 207 on StackPort1 VLAN307.
Mar 11 09:07:03: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking StackPort1 on VLAN0307. Inconsistent local vlan.
Mar 11 09:07:34: %SPANTREE-2-RECV_BAD_TLV: Received SSTP BPDU with bad TLV on StackPort1 VLAN1.
Mar 11 09:07:45: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking StackPort1 on VLAN0307. Port consistency restored.

[Reza Sharifi]

Jim,

We have also the same problem with our 2960-X switches (access) connecting to a pair of 4500x (VSS) except our issue is with Portchannel with 2 physical links connecting the 2960xs to the 4500.

If we disconnect one of the physical links from the portchannel everything works fine, but when we connect the same physical link back all users lose connectivity and the physical link starts flapping. Here are some of the messages we see in the logs when both physical links are in the portchannel:

Mar 10 18:00:43 EST: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 1 on Port-channel5 VLAN90.
Mar 10 18:00:43 EST: %SPANTREE-2-BLOCK_PVID_PEER: Blocking Port-channel5 on VLAN0001. Inconsistent peer vlan.
Mar 10 18:00:43 EST: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking Port-channel5 on VLAN0090. Inconsistent local vlan.
Mar 10 18:00:58 EST: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel5 on VLAN0001. Port consistency restored.
Mar 10 18:00:58 EST: %SPANTREE-2-UNBLOCK_CONSIST_PORT: Unblocking Port-channel5 on VLAN0090. Port consistency restored.
Mar 10 18:01:29 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/1, changed state to down
Mar 10 18:01:37 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/1, changed state to up
Mar 10 18:01:48 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/1, changed state to down
Mar 10 18:01:51 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/1, changed state to up

We have upgraded to 15.0(2a).EX5 and still have the same issue.

We have a ticket open with Cisco and have sent them all the logs and debugs and waiting to hear back from IOS developers.

HTH

[Rejohn Cuares]

Does the port-channel running dynamic aggregation protocol like PAGP or LACP?
 

[Reza Sharifi]

The Port-channels runs LACP, but we have also tried mode on and have the same issue.

[Rejohn Cuares]

I encountered this issue before but I was using mode ON. It was fixed after enabling dynamic aggregation protocol.

[NPT_2]

Keep this thread updated if you wouldn't mind and I'll do the same if I find a solution.  I was actually considering trying 15.0(2a).EX5 but it sounds like your similar problem wasn't solved with that so I'm thinking that won't help me either. 

Jim

[Reza Sharifi]

Jim,

I will update the thread if there is a change. 

So, there are a couple of bug fixes in 15.0(2a) regarding the fiber uplinks and GLC-T on the 2960X (see link)

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst2960x/software/15-0_2_EX/release_notes/OL30810.html#pgfId-255042

I think you should upgrade to 15.0(2a) so you don't have to worry about these other bugs mentioned in the link.

HTH

[NPT_2]

After getting replacement hardware and loading my configuration back into 15.0(2)EX5 to see if the link in problem went away, it did not.  Still all ports on my master switch linked in orange and would not pass any traffic.  Rather than upgrade again to 15.2(3)E I decided to see if I could find why half my ports (the ones on the master) wouldn't link in.

After wiping my entire configuration other than the stacking configuration all my ports linked in. 

To make a long story short, it appears that their is a bug (not sure if it is documented by Cisco) in 15.0(2)EX5 that prevents ports on the master switch in a 2960X switch from linking in if I set a voice vlan on an access port.  As soon as I turned off the voice vlan and only used an access port vlan the port without the voice vlan worked fine.  Rather than upgrading to 15.2(3)E that I don't think is stable I am sticking with 15.0(2)EX5 and just running a single VLAN for voice and data without a "switchport voice vlan" configuration on each port. 

[NPT_2]

In our situation, our supplier is replacing the hardware and hopefully that will take care of the issue and its not a bug or configuration issue.  I should know soon. 

[yetisekrem]

Hi Reza ,

Did you solve this problem ? i issue with the same problem and cant find any solution about this

[NPT_2]

Did you try removing the voice vlans from each interface configuration.

Go under an interface and do a "no switchport voice vlan 100" or whatever vlan # you are using.  This appears to be a bug and using a single access vlan on each port seems to fix the issue.  As far as I know this issue has not been fixed in newer versions.

Hopefully you don't need a separate voice vlan on your switch ports, if you don't disable the voice vlan and just use one access vlan and your ports should link right in. 

Jim

[andreabat72]

Hello,

did you solve the problem?

We same the same issue with 15.2(2)E3 and also 15.2(3)E1.

I found only a workaround:

- reload the device

- shut the interface

Did someone opened a TAC case?

Thanks a lot