doublesided vPC set STP blocking portchannel and uses peerlink instead

[moelka]

Hi all,

I have the following doublesided vPC topology

 wfw

Say, vlan 10 traffic passes through NX9K3, instead of using the port channel, it goes through the vPC peerlink, and through NX9K4. STP is blocking the port channel, and designating the peerlink, which is an indication of a loop detection. Is it normal behavior, or did I just make a bad design or wrong configuration?

EDIT: Just to make it clear, NX9k1&2 are spines, and 3&4 are leaf switches, each pair have their own vpc domains. So, the spines provide vpc to the leafs, and leafs provide vpc to the clients. I'm thinking that if a client connected on NX9K3 with vlan 10, the traffick should go NX9K3 -> Spines, and not through NX9K3 -> NX9K4 -> Spines.  

Heres is the config for the 4 switches:

NX9K1# sh run vpc && sh run int po1 membership
feature vpc
vpc domain 10
role priority 20
peer-keepalive destination 10.0.0.2 source 10.0.0.1

interface port-channel1
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  vpc 1

interface Ethernet1/3
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  channel-group 1 mode active
  no shutdown

interface port-channel2
  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  vpc 2

interface Ethernet1/4

  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  channel-group 2 mode active
  no shutdown

NX9K2# sh run vpc && sh run int po1 membership
feature vpc
vpc domain 10
role priority 30
peer-keepalive destination 10.0.0.1 source 10.0.0.2

interface port-channel1
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  vpc 1

interface Ethernet1/3
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  channel-group 1 mode active
  no shutdown

interface port-channel2
  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  vpc 2

interface Ethernet1/4

  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  channel-group 2 mode active
  no shutdown

NX9K3# sh run vpc && sh run int po1 membership
feature vpc
vpc domain 20
role priority 20
peer-keepalive destination 10.0.1.2 source 10.0.1.1

interface port-channel53
  description Link - Spines
  switchport mode trunk
  mtu 9216

interface Ethernet1/53
  description Link - NX9K1
  switchport mode trunk
  mtu 9216
  channel-group 53 mode active
  no shutdown

interface Ethernet1/54
  description Link - NX9K2
  switchport mode trunk
  mtu 9216
  channel-group 53 mode active
  no shutdown

NX9K4# sh run vpc && sh run int po1 membership
feature vpc
vpc domain 20
role priority 300
peer-keepalive destination 10.0.1.1 source 10.0.1.2

interface port-channel53
  description Link - Spines
  switchport mode trunk
  mtu 9216

interface Ethernet1/53
  description Link - NX9K1
  switchport mode trunk
  mtu 9216
  channel-group 53 mode active
  no shutdown

interface Ethernet1/54
  description Link - NX9K2
  switchport mode trunk
  mtu 9216
  channel-group 53 mode active
  no shutdown

[MHM Cisco World]

are You run double sided vPC ? If Yes then check link below, your vPC config is wrong. 
https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/218332-configure-and-verify-double-sided-virtua.html

[balaji.bandi]

Look at enhanced vPC :

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/218333-understand-and-configure-nexus-9000-vpc.html

[khorram1998]

It appears that your vPC topology and configurations have a few issues that may be causing the loop detection and STP blocking the port channel.

First, vPC domain 10 is configured on NX9K1 and NX9K2, but vPC domain 20 is configured on NX9K3 and NX9K4. This mismatch in vPC domains can cause issues with the vPC peerlink and the port channels. The vPC domain should be the same across all vPC peers.

Second, the vPC peer-keepalive link is configured with different IP addresses on each switch. The peer-keepalive link should have the same IP addresses configured on both vPC peers.

Third, the interfaces on NX9K3 and NX9K4 that connect to the spine switches are not configured as vPC interfaces. These interfaces should be configured as vPC member ports on the same vPC domain as the other interfaces.

Finally, it seems that the port channel is not configured properly on NX9K3 and NX9K4, the port channel is not associated with any vpc domain.

Fixing these issues should resolve the loop detection and STP blocking the port channel. You should reconfigure the vPC domains, the peer-keepalive link and the port channel interfaces with the same settings across all switches.

It's also a good idea to check the logs and monitor the network while making these changes to ensure that the traffic is flowing as expected.


here are the commands to configure a proper vPC topology on the four switches:
NX9K1:

feature vpc
vpc domain 10
role priority 20
peer-keepalive destination 10.0.0.2 source 10.0.0.1

interface port-channel1
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  vpc 1

interface Ethernet1/3
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  channel-group 1 mode active
  no shutdown

interface port-channel2
  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  vpc 2

interface Ethernet1/4
  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  channel-group 2 mode active
  no shutdown

NX9K2:

feature vpc
vpc domain 10
role priority 30
peer-keepalive destination 10.0.0.1 source 10.0.0.2

interface port-channel1
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  vpc 1

interface Ethernet1/3
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  channel-group 1 mode active
  no shutdown

interface port-channel2
  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  vpc 2

interface Ethernet1/4
  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  channel-group 2 mode active
  no shutdown

NX9K3:

feature vpc
vpc domain 10
role priority 20
peer-keepalive destination 10.0.0.2 source 10.0.0.1

interface port-channel53
  description Link - Spines
  switchport mode trunk
  mtu 9216
  vpc 3

interface Ethernet1/53
  description Link - NX9K1
  switchport mode trunk
  mtu 9216
  channel-group 53 mode active
  no shutdown

interface Ethernet1/54
  description Link - NX9K2
  switchport mode trunk
  mtu 9216
  channel-group 53 mode active
  no shutdown

NX9K4:

feature vpc
vpc domain 10
role priority 30
peer-keepalive destination 10.0.0.1 source 10.0.0.2

interface port-channel53
  description Link - Spines
  switchport mode trunk
  mtu 9216
  vpc 3

interface Ethernet1/53
  description Link - NX9K1
  switchport mode trunk
  mtu 9216
  channel-group 53 mode active
  no shutdown

interface Ethernet1/54
  description Link - NX9K2
  switchport mode trunk
  mtu 9216
  channel-group 53 mode
  no shutdown

This will ensure that vlan 10 traffic passes through the vPC peer-link, instead of using the port channel. This will also enable STP to block the port channel and designate the peer-link as the active path, which is necessary for loop detection.

Please note that these commands are just examples and may need to be adjusted to match your specific network setup and requirements.

Please rate this and mark as solution/answer, if this resolved your issue
All the best,
AK

[moelka]

Thank you for your reply. I think I have chosen a wrong title. It is intentional that The spines (NX9K1&2) have their own vpc domain. In this example, i have simplified the diagram, where in reality, i have 2 spines, and 10 leaf switches, each 2 form a vpc domain. In other words, the spines provide vPC to leaf switches, and leaf switches provides vpc to clients. Therefore, the configuration on 1&2 have "vpc 1" configuration, and 3&4 have their ports configured as normal portchannels

[khorram1998]

Based on your updated information, it sounds like you have a more complex network topology, with the spines acting as the vPC domain providers for the leaf switches, and then the leaf switches providing vPC connections to the clients.

In this case, it would make sense that the spines (NX9K1 and NX9K2) have their own vPC domain, and that the leaf switches (NX9K3 and NX9K4) have their ports configured as normal port channels.

To ensure that the traffic flows correctly, you should also ensure that the vlans that are used for the client connections are configured as vPC VLANs on the spines and as normal VLANs on the leaf switches.

You might also want to verify that the STP settings are configured correctly, to prevent loops and ensure that the correct ports are blocked on the spines and the leaf switches.

You also need to make sure that the port-channel interfaces and vlans have the correct vpc number.

Here is an example of the configuration commands you would use to set up the vPC domain on the spines (NX9K1 and NX9K2) and the normal port channels on the leaf switches (NX9K3 and NX9K4)

Spines (NX9K1 and NX9K2):

feature vpc
vpc domain 10
role priority 20
peer-keepalive destination 10.0.0.2 source 10.0.0.1

interface Ethernet1/3
  description vPC link to leaf switch 1
  switchport mode trunk
  mtu 9216
  channel-group 1 mode active
  no shutdown

interface Ethernet1/4
  description vPC link to leaf switch 2
  switchport mode trunk
  mtu 9216
  channel-group 2 mode active
  no shutdown

interface vlan 10
  vpc 10

Leaf switches (NX9K3 and NX9K4):

interface Ethernet1/53
  description Link to spine 1
  switchport mode trunk
  mtu 9216
  channel-group 53 mode active
  no shutdown

interface Ethernet1/54
  description Link to spine 2
  switchport mode trunk
  mtu 9216
  channel-group 54 mode active
  no shutdown

This example uses vPC domain 10 and vlan 10, you should adjust the vpc number, vlan number and IP addresses according to your network setup.

Please note that these commands are just examples and may need to be adjusted to match your specific network setup and requirements.

Also, please make sure that you are configuring the same vpc domain number on both spines and leaf switches.

Please rate this and mark as solution/answer, if this resolved your issue
All the best,
AK

[Christopher Hart · Accepted solution]

Hello!

Typically in a double-sided vPC, both vPC domains have a single vPC running between them. In your scenario, you have two separate vPCs on each vPC peer, which is forcing Spanning Tree Protocol to block one of them.

An example might work best here. In your topology, NX9K1 has Ethernet1/3 in vPC Po1 towards NX9K3 and Ethernet1/4 in vPC Po2 towards NX9K4. NX9K2 has identical configuration. Typically, we could expect both Ethernet1/3 and Ethernet1/4 of both NX9K1 and NX9K2 to be placed in the same vPC (let's call it Po10, just to differentiate it from your existing vPC Po1 and vPC Po2). Similarly, Ethernet1/53 and Ethernet1/54 of NX9K3 and NX9K4 would be placed in the same vPC as well (we'll call this Po20, just to differentiate it from the new vPC Po10).

A rough, untested example of this configuration is below.

NX9K1# show running-config
<snip>
feature vpc

vpc domain 10
  role priority 20
  peer-keepalive destination 10.0.0.2 source 10.0.0.1

interface port-channel10
  description vPC link - NX9K3 and NX9K4
  switchport mode trunk
  mtu 9216
  vpc 10

interface Ethernet1/3
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  channel-group 10 mode active
  no shutdown

interface Ethernet1/4
  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  channel-group 10 mode active
  no shutdown


NX9K2# show running-config
<snip>
feature vpc

vpc domain 10
  role priority 30
  peer-keepalive destination 10.0.0.1 source 10.0.0.2

interface port-channel10
  description vPC link - NX9K3 and NX9K4
  switchport mode trunk
  mtu 9216
  vpc 10

interface Ethernet1/3
  description vPC link - NX9K3
  switchport mode trunk
  mtu 9216
  channel-group 10 mode active
  no shutdown

interface Ethernet1/4
  description vPC link - NX9K4
  switchport mode trunk
  mtu 9216
  channel-group 10 mode active
  no shutdown


NX9K3# show running-config
<snip>
feature vpc

vpc domain 20
  role priority 20
  peer-keepalive destination 10.0.1.2 source 10.0.1.1

interface port-channel20
  description Link - Spines
  switchport mode trunk
  mtu 9216
  vpc 20

interface Ethernet1/53
  description Link - NX9K1
  switchport mode trunk
  mtu 9216
  channel-group 20 mode active
  no shutdown

interface Ethernet1/54
  description Link - NX9K2
  switchport mode trunk
  mtu 9216
  channel-group 20 mode active
  no shutdown


NX9K4# show running-config
<snip>
feature vpc

vpc domain 20
  role priority 20
  peer-keepalive destination 10.0.1.1 source 10.0.1.2

interface port-channel20
  description Link - Spines
  switchport mode trunk
  mtu 9216
  vpc 20

interface Ethernet1/53
  description Link - NX9K1
  switchport mode trunk
  mtu 9216
  channel-group 20 mode active
  no shutdown

interface Ethernet1/54
  description Link - NX9K2
  switchport mode trunk
  mtu 9216
  channel-group 20 mode active
  no shutdown

 I hope this makes sense and is helpful - thank you!

-Christopher

[MHM Cisco World]

exactly. 

[moelka]

Hello Christopher,

I will try to test it out. I'll be back after the results

[moelka]

Hello again,

Did you intentionally leave NX9K3&4 without a vpc configuration on the port channels? Because right now, you have configured 1 VPC going to two different switches, and then on those two switches, you have configured a normal protchannel. This resulted in taking only one of the switches (whichever came online first). In other words, With this setup, The spines will only connect to 1 of the switches (NX9K3 or 4) and the other link will be suspended.

[Christopher Hart]

Hello!

That was not intentional, that was a mistake on my part - my apologies! I edited the configuration in my previous post to include vpc 20 configuration on both NX9K3 and NX9K4's Po20 interfaces.

Thank you!

-Christopher

[moelka]

Thank you so much Christopher. I've tested it with various vPC-failure scenarios and it works like a charm. Peer-Link is "designated fwd" and the vPC toward the spines is in "Root fwd" state. Just one last thing if you don't mind, should the command "Peer-switch" only be used on the spines, or both sides?   

[MHM Cisco World]

good Q, if I can I will lab and see the effect of peer-SW in 

case1 In one vPC domain 

case2 In both vPC domain 

[Christopher Hart]

Hello!

I'm glad to hear that configuration worked as expected!

Generally speaking, the peer-switch vPC domain enhancement should be configured on both vPC peers of the vPC domain that should be the Root Bridge for all VLANs. I am assuming that the spines will be performing inter-VLAN routing (similar to a classic collapsed core network design), so I would generally recommend configuring peer-switch on the spines. You would not need to configure peer-switch on the leafs (you can, but it will not change any behavior since the leaf switches are not the Root Bridge for the VLANs in question).

I hope this helps - thank you!

-Christopher

[MHM Cisco World]

Hello @Christopher Hart 
as I know the peer-Switch will make both vPC pair send same Bridge-ID to Connect SW (none vPC)
here as I see (and I will try lab it) the leaf is connect to other SW not the Spine, so it sense to config it to Leaf not Spine. 
but what effect of config Peer-Switch in both vPC pair in Double-Sided vPC design need double check. 
thanks