Question about setting up different VLAN between Nexus VPC peer

[kay.kang]

Hi,

We're operating VPC between two N3K devices.

And try to add a different VLAN to each VPC member switch(Nexus3000-A-1 and Nexus3000-A-2) as attached diagram.

That additional VLAN is connecting through each ISP  SW(ISP SW_1 & ISP SW_2).

We're not planning to run VPC between Nexus switches and ISP switches.

Is it possibly to run different VLAN on existing VPC peer switches without any issue?

[shoppman]

Hi Kay Kang,

Yes this is possible so long as the VLAN is not a VPC VLAN. 

A VPC VLAN is defined as any VLAN that is carried on the VPC peer-link. So we would need to ensure that this VLAN is not allowed on the VPC peer-link trunk configurations.  This is because a VPC VLAN is expected to exist on both VPC peers, and therefore the VPC peer-link is configured for Bridge Assurance (spanning-tree port-type network) by default. 

When Bridge Assurance is configured on a port, the switch sends Spanning Tree Protocol Bridge Protocol Units ( STP BPDUs) for each VLAN that exists locally and is allowed on the port, every 2 seconds. It also expects to receive STP BPDUs for each VLAN every 2 seconds.  If the expected STP BPDU is not received then the VLAN is suspended locally.  Removing the VLAN from the peer-link prevents this scenario. 

Of course, if a VLAN is not carried on the VPC peer-link, it also cannot be carried on any VPC member ports either, such as VPC 10, VPC 20, VPC 100, etc.  As long as the VLAN in question only needs to be carried on non-VPC ports (orphan ports), then configuring it on the local peer only should be fine as long as it is dis-allowed from the peer-link trunk.  In your case, since VLAN 30 needs to exist locally on A-1 and only needs to be carried on the orphan port where the ISP is connected, then it does not need to be a VPC VLAN and can exist only on this peer.

I hope this helps!

Scott Hoppmann

Cisco HTTS - Data Center Route and Switch

RTP, NC

[kay.kang]

Thanks, Scott for the reply.

I am thinking of a best practice to make this scenario work.

Without any process of removing the VLAN from VPC, Configure VLAN 30,40 on both Nexus 3000-A-1 and A-2 as VPC VLANs.

And, each Nexus switch has an orphan port connecting to the relevant ISP link.

 I am not sure if it can be feasible in this way.

[M02@rt37]

Hello @kay.kang 

Configure Trunk interfaces facing ISP as "orphan ports".

[shoppman]

Hi Kay Kang,

Removing the VLAN from the VPC peer-link would not break any best practices that I'm aware of - this is a perfectly acceptable way to configure a VLAN that does not need to be carried on any VPC links and only needs to cross an orphan port.

Adding the VLAN to both VPC peers can also work but has added caveats as well.  For instance, if you create an SVI for VLAN 30 on one VPC peer, you must also create an SVI for VLAN on the other VPC peer.  Is is not supported or best practice to have an SVI on only one VPC peer when the VLAN is a VPC VLAN.

I hope this helps!

Scott Hoppmann

Cisco HTTS - Data Center Route and Switch

RTP, NC

[kay.kang]

Thanks for the response.

From my lab test, N3K-A-1 showed VLAN 30 on Locally suspended VLANs and N3K-A-2 showed VLAN 40 on Locally suspended VLANs in the scenario. I just configured relevant SVI on each Nexus switch and tested ping to next hop IP over the each ISP SW-1 and It seems okay. The status of SVI on each Nexus switch looked okay. I am not sure what issue can be caused from locally suspended VLANs.

I didn't need to create the same SVI on both Nexus switches.

The solution that clearing locally suspended VLANs was adding both VLANs on both Nexus switches. 

[shoppman]

Hi Kay Kang,

Thanks for the update. This was almost certainly the bridge assurance that was causing the VLAN to be suspended.  See my earlier post on this. 

If you have VLAN 30 only created on A-1 and the VLAN is allowed on the peer-link, it will suspend due to the Bridge Assurance that is on the peer-link by default because the VLAN does not exist on A-2.  Bridge Assurance will suspend any VLAN that is allowed on the VPC peer-link but only exists on one VPC peer. So, creating the same VLAN on A-2 clears the Bridge Assurance fault and removes the VLAN from being suspended.  This is why I originally suggested disallowing the VLAN on the peer-link if it was only created on one VPC peer.  This would also have cleared the VLAN suspension without having to add VLAN 30 to A-2.

The takeaway here is  - if you want the VLAN to be configured only on one of the two VPC peer switches, that VLAN must be disallowed from the VPC peer-link trunk configuration.  In the case of your topology for instance,  this would look similar to the following:

interface port-channel10
switchport
switchport mode trunk
switchport trunk allowed vlan 10,20  <<<<  VLAN 30 is not allowed, since we only want it to exist locally
spanning-tree port type network  <<<< Bridge assurance, configured by default on the peer-link
vpc peer-link

Also, to your point about not needing the SVI on both VPC peers.  Yes, you can create the SVI on just one peer.  But be aware that this is not best practice and can lead to unexpected traffic behavior because it creates a VPC type-2 inconsistency.  You will see this when you run 'show vpc brief' in this state.

I hope this helps!

Scott Hoppmann

Cisco HTTS - Data Center Route and Switch

RTP, NC

[MHM Cisco World]