Spanning Tree Changes

[MP13]

Hi,

I am currently reviewing our STP setup as we are getting a number of regular storm control alerts even though it's set to 10% of a 10Gbit port for multicast and broadcast traffic and considering port utilisation for all traffic is typically only 1Gbit I think we have some issues with some ports and we're seeing STP transition from listen to block every few minutes on some VLAN's.

We've also found that if we add or remove a VLAN from trunk ports then over the course of about 15 minutes we get a growth in multicast and broadcast traffic until the Nexus loop detection kicks in and stops MAC learning for 180 seconds which causes around a 3 minute outage of the network and then it returns to normal.

I'm planning during a maintenance window to try and get our STP topology uniform and configured better and also conduct a packet capture on a mirror port to work out exactly what traffic is causing the storm too. It may not be STP but it seems to have started since we added some new switches to the network.

I have attached a quick mock-up diagram of our network which shows where we have Rapid PVST and where we have just PVST. My plan is to change all of the switches to Rapid PVST which will hopefully also help with our convergence time. Current convergence time is about 30-60 seconds which from reading around the community and some other sites although it's ok to mix modes the convergence time that Rapid PVST should bring to a few seconds will not occur because of the devices just running PVST. 

My plan on all of the switches running PVST is to just issue "spanning-tree mode rapid-pvst". I just want to check if there is anything in particular that I should watch out for when doing this or whether there are any additional steps recommended? 

We also have a couple of customer networks that link off of ours which I have shown on the diagram. To my knowledge at the moment our STP merges with theirs for the VLAN's that we provide them. Should we be making some changes to block STP at our edge to them so that their STP is then completely separate to ours?

Last of all is root bridges. At the moment we don't specify what the root bridges should be using any priorities. Would it make sense to change this and have the Nexus 5K's just under the ASR's as the primary and secondary root bridge?

I've tried to read up on things a fair bit but really just looking to clarify some additional queries here in relation to our own network using the attached topology diagram.

[MHM Cisco World]

[MP13]

The NSK's are all separate devices. We just have standard trunk ports between them tagged with the required VLAN's to go across them.

As an example this would be our config on the ports on both sides of the link going between 2 of the NSK's.

interface Ethernet1/7
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 1,107,110,116,120,125,137
storm-control broadcast level 1.00
storm-control multicast level 1.00

And then output as an example for STP on VLAN 107 which shows it's only got one of the ports in forwarding state at the moment:

VLAN0107
Spanning tree enabled protocol rstp
Root ID Priority 20587
Address 0017.95bd.5cc0
Cost 8
Port 134 (Ethernet1/6)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32875 (priority 32768 sys-id-ext 107)
Address 547f.eef7.dbbc
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/5 Altn BLK 2 128.133 P2p Peer(STP)
Eth1/6 Root FWD 2 128.134 P2p Peer(STP)
Eth1/7 Altn BLK 2 128.135 P2p
Eth1/20 Altn BLK 4 128.148 P2p Peer(STP)
Eth1/21 Altn BLK 4 128.149 P2p Peer(STP)

[MHM Cisco World]

that issue of the uplink SW have only one FWD and all other is BLK. 
sure there is something to do with STP.
 0017.95bd.5cc0 <<- this root bridge mac where is this SW ??

[MP13]

My understanding was always that if you had multiple paths that STP should have just 1 forwarding for that VLAN and the others blocked. Is that not correct?

[MHM Cisco World]

Sure Yes, 
but you not BLK the uplink for Core and make link interconnect (with low BW) FWD!!
that wrong.
we try always to make uplink between Access SW to Agg/Core SW FWD and make this link high BW to carry traffic from all Access SW

[MP13]

Thank you for the clarification.

Ports 1 to 8 on the top 2 Nexus are all 10Gbit ports. Port mapping is below for the Nexus on the top left if it helps and the one on the top right is largely the same. I think the STP output earlier may have been from the top right one so I've also included the STP output for the top left one below.

1 - Feed to ASR with WAN VLAN's
2 - Feed from ASR with LAN VLAN's
3 - Part of PO to Top Right Nexus
4 - Part of PO to Top Right Nexus
5 - WAN Feed In
6 - Feed to 2960S
7 - Feed to 2960S
8 - Feed to 6503

VLAN0107
Spanning tree enabled protocol rstp
Root ID Priority 20587
Address 0017.95bd.5cc0
Cost 4
Port 145 (Ethernet1/17)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32875 (priority 32768 sys-id-ext 107)
Address 547f.eed3.5981
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/6 Desg FWD 2 128.134 P2p Peer(STP)
Eth1/7 Desg FWD 2 128.135 P2p Peer(STP)
Eth1/8 Desg FWD 2 128.136 P2p Peer(STP)
Eth1/17 Root FWD 4 128.145 P2p
Eth1/18 Desg FWD 4 128.146 P2p Peer(STP)
Eth1/20 Desg FWD 4 128.148 P2p Peer(STP)

Ports 16 and above are 1Gbit ports primarily just used for directly attached devices instead of switches but there are a couple of those ports which are facing external networks to which there was a query around if we should and how we would stop their STP traffic becoming part of ours.

The main difference on the top right one is that Port 8 which is the feed to the 6503 is to the bottomleft Nexus instead. The 6503 we do have a Nexus to put in it's place instead but have left it was the 6503 whilst we work this out.

[MHM Cisco World]

are I am right 
4 - Part of PO to Top Right Nexus <<-Eth1/17 Root FWD 4 128.145 P2p ???

[MP13]

Ports 3 and 4 on both the top left and top right are in a PO and that PO goes between the 2 top switches.

Eth1/17 is just a downstream switch that will be fed by both of the top 2 switches.

Looking back at some old configuration backups we historically had the top left set with:

spanning-tree vlan 1-4094 priority 4096

and top right then set with:

spanning-tree vlan 1-4094 priority 8192.

This is not currently in place though.

Should I basically be working down the topology so that these are set again at the top level and then the 2nd level setting bottom left 14096 and bottom right 18192 then the access switches at the bottom setting 24096 for example?

[MHM Cisco World]

I dont get which port interconnect two NSK, 
but if you use PO then it must appear in STP 
and there is no info. about PO in STP 
so do show port-channel summary , check if the PO is SU and both port member is (P)

[MP13]

Eth1/3 and Eth1/4 on both of the top Nexus switches interconnect them.

These ports are in PO1 it's just that VLAN 107 isn't on PO1 which is why it's not in STP. If I do VLAN 125 though you will see this then in the STP information:

VLAN0125
Spanning tree enabled protocol rstp
Root ID Priority 32768
Address 10f9.2052.3002
Cost 4
Port 136 (Ethernet1/8)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32893 (priority 32768 sys-id-ext 125)
Address 547f.eed3.5981
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 1 128.4096 P2p
Eth1/2 Desg FWD 2 128.130 P2p
Eth1/6 Desg FWD 2 128.134 P2p Peer(STP)
Eth1/7 Desg FWD 2 128.135 P2p Peer(STP)
Eth1/8 Root FWD 2 128.136 P2p Peer(STP)
Eth1/18 Desg FWD 4 128.146 P2p Peer(STP)
Eth1/20 Desg FWD 4 128.148 P2p Peer(STP)

Output of requested command for PO:

--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth NONE Eth1/3(P) Eth1/4(P)

[MHM Cisco World]

one Q, are you allow VLAN1 in trunk ??

[MP13]

I have just double checked and we allow VLAN 1 on all trunk ports.

[MHM Cisco World]

Eth1/2 Desg FWD 2 128.130 P2p <<-
Eth1/6 Desg FWD 2 128.134 P2p Peer(STP) <<-

you have two STP here 

one legacy and other is RSTP 
I prefer make the SW that run RSTP root for any VLAN that use by SW run legacy.

this prevent any compatible issue between two STP mode

[MP13]

Thank you. This is part of the planned work to change everything to Rapid PVST. Just to recap from the original question there was the below query in relation to this change.

"My plan on all of the switches running PVST is to just issue "spanning-tree mode rapid-pvst". I just want to check if there is anything in particular that I should watch out for when doing this or whether there are any additional steps recommended?"

Out of curiosity from that output before how did you determine one is RSTP and one is STP. Is it based on the Peer(STP)? Does that indicate RSTP and the one without is just STP? Just trying to learn whilst going through these queries as well.