vPC Spanning Tree issue

[Craddockc]

Community,

I am noticing a weird Spanning Tree forwarding situation within my Data center topology. Ive attached a PDF of the topology for reference but I will also lay it out for you here:

At the spine/core we are running a pair of 9396PQ's in NX-OS mode. These two switches are connected in a vPC peer set up in vPC domain 1, with 9396-1 being the peer primary. The 9396-1 is the spanning tree root for all vlans. The vPC peer link is Po1 between these two switches.

At the Access/Dist I just got through connecting up two 9332PQ's to the 9396's in the following fashion:

9332PQ-1 (POD-1A) and 9332PQ-2 (POD-1B) are connected to eachother in a vPC peer setup in vPC domain 10, with POD1 being the vPC primary for vPC domain 10. The vPC peer link is Po1 between these two switches.

POD-1A is uplinked to both 9396's via PC12 and vPC12 (running in vPC domain 1 on the 9396's).

POD-1B is uplinked to both 9396's via PC13 and vPC13 (running in vPC domain 1 on the 9396's).

Here is the situation:

on POD-1A Po12 is root forwarding in the spanning tree, which is what I expect considering thats the uplink to the Root Bridge (the 9396).

However, on POD-1B Po13 is Blocking, and its using Po1 (the vPC peer link) as its spanning tree root port. I dont understand why this is as Po13 is the uplink to the spanning tree root. is this normal? is there something messed up in my config?

QTS-POD-1A# show spanning-tree vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 4097
Address f4cf.e292.f39f
Cost 200
Port 4107 (port-channel12)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 12289 (priority 12288 sys-id-ext 1)
Address f8c2.8890.b085
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 250 128.4096 (vPC peer-link) Network P2p
Po12 Root FWD 200 128.4107 (vPC) P2p
Po13 Altn BLK 200 128.4108 (vPC) P2p

QTS-POD-1B# show spanning-tree vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 4097
Address f4cf.e292.f39f
Cost 450
Port 4096 (port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 16385 (priority 16384 sys-id-ext 1)
Address 88f0.3187.0c7f
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 250 128.4096 (vPC peer-link) Network P2p
Po13 Altn BLK 200 128.4108 (vPC) P2p (This is the uplink to the core, why is it blocking?)

Is Po13 actually forwarding data but just blocking for spanning tree? That actually doesnt make any sense but Im not as well versed in how vPC and STP coincide. 

Any help you can provide would be greatly appreciated.

Thanks.

[Francesco Molino]

Hi

Can you share your configs (please attach them as text files).
Usually we activate the peer-switch feature on the vpc to allow both Nexus switches to share the virtual bridge id and both act as root stp.

I'll wait for your configs but it makes sense of the pod1-a has the higher stp priority.

[Craddockc]

Francesco,

Ive attached the configs of both the Core 9396's and the Aggregation 9332's. Just to recap. 9332-1 is uplinked to both 9396s via PC12 and vPC 12 and PC12 is Root Forwarding as shown:

QTS-POD-1A# show spanning-tree vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 4097
Address f4cf.e292.f39f
Cost 200
Port 4107 (port-channel12)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 12289 (priority 12288 sys-id-ext 1)
Address f8c2.8890.b085
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 250 128.4096 (vPC peer-link) Network P2p
Po12 Root FWD 200 128.4107 (vPC) P2p
Po13 Altn BLK 200 128.4108 (vPC) P2p

9332-2 is uplinked to both core 9396's via PC13 and vPC13 but PC13 is not Root forwarding as I would expect it to, its Blocking. 9332-2 is using its vPC peer link as the Root port. Very strange:

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 4097
Address f4cf.e292.f39f
Cost 450
Port 4096 (port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 16385 (priority 16384 sys-id-ext 1)
Address 88f0.3187.0c7f
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 250 128.4096 (vPC peer-link) Network P2p
Po13 Altn BLK 200 128.4108 (vPC) P2p

[Francesco Molino · Accepted solution]

Ok Based on your config 9396-1 is the root STP.

Now there's rule on VPC that VPC will never be blocked and that's normal because important traffic is passing through it. In your design, normal STP would have to choose between Po13 and your Po1. If now you take into consideration that VPC peer-link will never be blocked then that's normal that Po13 will be the ALTN port.

Now, if I can, I would highly recommend to use peer-switch feature to share the virtual bridge id and this on both pair of Nexus. The other question, is why not aggregating all ports between both pair of Nexus is a single Po/vPC?

[Craddockc]

Francesco,

Thanks so much for your helpful feedback. It makes sense now that the switch would block Po13 if Po1 must be kept forwarding at all times since its the vPC peer link. I think I see what youre getting at here. Without the peer-switch feature enabled, the POD-1B switch believes the root can be reached via Po1 (the vPC peer link) instead of Po13 because 9396-1 alone is the root bridge for all vlans. Would Po13 go forwarding if the peer-switch option were enabled? 

regarding putting the uplinks from both 9332's and 9396's into a single Port Channel and vPC. I was not aware this was an option. Im still learning about Nexus and vPC and am not aware of all the design best practices. Ive attached an updated diagram to verify this is your suggestion.

If I put them all in the same vPC and port channel i would think that this would eliminate my STP forwarding issue. Im still trying to wrap my head around how two downstream devices (both 9332's) can be in the same port channel as each other as it pertains to the Core 9396's. I do understand that the two Core 9396's appear as a single device to the downstream 9332's but if all 4 links on the 9396's are bundled into a single port channel, how does the 9396 know that traffic originating from one 9332 needs to go back to the same 9332? wouldnt this create an asynchronous switching situation where traffic from one 9332 going to the 9396 could possible be sent back down the links toward the other 9332 in the bundle? 

Thanks.

[Craddockc]

Francesco,

I did as you instructed. I enabled the peer-switch feature on my 9396 pair as shown below:

PHX-AGG-1A# show running-config vpc

!Command: show running-config vpc
!Time: Wed Jan 24 13:03:37 2018

version 7.0(3)I4(6)
feature vpc

vpc domain 1
peer-switch
role priority 1
system-priority 1000
peer-keepalive destination 172.16.0.2 source 172.16.0.1 vrf VPC-KEEPALIVE
delay restore 120
peer-gateway
auto-recovery
ip arp synchronize

PHX-AGG-1B# show run vpc

!Command: show running-config vpc
!Time: Wed Jan 24 13:04:18 2018

version 7.0(3)I4(6)
feature vpc

vpc domain 1
peer-switch
role priority 2
system-priority 1000
peer-keepalive destination 172.16.0.1 source 172.16.0.2 vrf VPC-KEEPALIVE
delay restore 120
peer-gateway
auto-recovery
ip arp synchronize

This resulted in both switches claiming themselves as the RB for all vlans using the virtual BID of 0023.04ee.be01

PHX-AGG-1A# show spanning-tree vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 4097
Address 0023.04ee.be01
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

PHX-AGG-1B# show span vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 4097
Address 0023.04ee.be01
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

I then went ahead and bundled all 4 uplinks (2 from each switch on each end) into Po12 and vPC12 and the spanning tree on the 9332 in question now looks like this:

PHX-POD-1B# show spanning-tree vlan 1

VLAN0001
Spanning tree enabled protocol rstp
Root ID Priority 4097
Address 0023.04ee.be01
Cost 450
Port 4096 (port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 16385 (priority 16384 sys-id-ext 1)
Address f8c2.8890.4f75
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 250 128.4096 (vPC peer-link) Network P2p
Po12 Root FWD 200 128.4107 (vPC) P2p

Is this normal? I come from a heavy IOS background so seeing two Root ports for the same vlan is very abnormal to me. Will the switch make its forwarding decisions based on cost here? Im assuming that no data traffic is actually going to traverse the peer link (because that would be a loop) and the traffic will use Po12?

Thanks.

[Francesco Molino]

You bundle all ports in 1 PO (vPC).

With peer-switch feature, Nexus switches are using BID. That's why you see root port on both nexus going same PO 12. From downstream Nexus, upstream Nexus switches are 1 big logical switch. At downstream layer, it won't see that these are 2 equipments.

Traffic will pass through peer-link as well and this isn't going to be a loop. The goal of VPC is to bundle the 2 nexus switches in 1 logical like VSS in Catalyst word. 

Is that more comprehensible?

[Craddockc]

Francesco,

Thank you for the reply. So the way ive configured it is now correct? With peer-switch feature as well as bundling all 4 devices into one logical Port Channel?

Thanks.

[Francesco Molino]

Yeah for the interconnection of your 4 Nexus switches that's correct.
You're welcome

[Craddockc]

Thank you Francesco! 

[Francesco Molino]

You're welcome