vPC and peer link does it forward traffic or only state

[romanroma]

I am new to Nexus and vPC peer links concept, and due to the latter I am reading the following:

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/design_guide_c07-625857.html#_Toc271759439

However, what I do not understand if the link (east-west) forwards traffic or just keep alive, and link state. I have read the link numerous times, but if they are answering my question - i am just not getting it.

So, does the East-West peer link forward layer 2 traffic or is it just management traffic? Would traffic ever take this path to forward frames or packets over if routing?

Thank you

RR

[Reza Sharifi]

Hi,

The east-west traffic using the vPC peer link is just layer-2 data traffic (not management) with all vlans included. For management, all Cisco Nexus devices come with an out-of-band management port that can be used to manage the device. It can also be used for vPC peer-keep-alive.

figure-3 in this link:

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/configuration_guide_c07-543563.html

HTH

[romanroma]

@Reza Sharifi,

When you say 'layer 2' does that mean STP will making blocking or change the path that frames can take? I thinking I am confusing myself, because the figures show traffic going up and down the leaf/spine but not the cross-link between the two switches. So is there ever a case that traffic user/production traffic will take the peer link that is connected between the two switches?

Thank you

[Sergiu.Daniluk]

Hi @romanroma 

vPC peer-link will transfer two type of traffics:

• vPC control-plane traffic using Cisco Fabric Services over Ethernet (CFSoE). This includs:
  • Configuration validation and comparison (consistency check)
  • Synchronization of MAC addresses for vPC member ports
  • vPC member port status advertisement
  • Spanning Tree Protocol management
  • Synchronization of HSRP and IGMP snooping
  • and few more
• user traffic. This will include:
  • traffic destined to servers connected on orphan port on the vPC peer
  • BUM traffic - this is mandatory to be sent over vPC peer-link in case there are orphan ports on vPC peer. This traffic will not be sent out of vPC enabled port-channels (this is where loop avoidance mechanism kicks in)

If you want to learn vPC, I strongly recommend you to read the vPC design guide:

https://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf 

It is written for Nexus 7000, but the information and details from the doc, applies on all Nexus platforms.

Cheers,

Sergiu

[romanroma]

@Sergiu.Daniluk 

"

• user traffic. This will include:
  • traffic destined to servers connected on orphan port on the vPC peer
  • BUM traffic - this is mandatory to be sent over vPC peer-link in case there are orphan ports on vPC peer. This traffic will not be sent out of vPC enabled port-channels (this is where loop avoidance mechanism kicks i

Does this link only get used during an orphaned situation? I am still trying to understand if traditional STP methods keeps this link in block state, and only the leaf/spin links are used, and only opens the link when the vPC is in a bad health state.

Reading all I can since got moved to Nexus switches due to low man power and resources.

[Sergiu.Daniluk]

Hi @romanroma 

The unicast traffic destined to servers connected on orphan ports and all BUM (Broadcast, Unknown unicast and Multicast) traffic is being sent over Peer-Link.

The vPC Peer-link interface is always* in STP forwarding state.

*Note: the vpc peer-link is by default configured as STP port type network, meaning if one of the peers is not sending STP BPDUs to the other one, the port goes into BA_inc (bridge assurance inconsistency).

Regards,

Sergiu

[romanroma]

@Sergiu.Daniluk 

Another confusion,

I am reading the following docs:

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/configuration_guide_c07-543563.html

And is says:

"• vPC peer link: The vPC peer link is the link used to synchronize states between the vPC peer devices. The vPC peer link carries control traffic between two vPC switches and also multicast, broadcast data traffic. In some link failure scenarios, it also carries unicast traffic. You should have at least two 10 Gigabit Ethernet interfaces for peer links. "

So how do you know or when are the "some link failures" when the peer link will allow unicast traffic? I am just not getting the concept when the vPC link, which usually carries: keep alive, multicast, broadcast and control traffic.

[Sergiu.Daniluk]

Hi @romanroma 

The failure scenario where unicast traffic is being forwarded over the peer-link, is when one link from the vPC port-channel goes down, and unicast traffic is received on the peer where the link is down. Here is a graphical view of the scenario:

left - working scenario ; right - failure scenario

Cheers,

Sergiu

[romanroma]

@Sergiu.Daniluk
That is very helpful thank you...

One last question - So I am trying to find the path of a remote interface and port, which I have already found by using 'sh fabricpath route switchid' and the 'sh fabricpath switch-id'; however, if I have more then one path in the spine-leaf model - how do I find out what path is being taken? I am using vPC and fabricpath, but does the STP and root/bridge switch concept still apply to the vlans and topology path?

Thank you so much... much appreciated.

[Sergiu.Daniluk]

Hello,

The FP will load-balance the unicast traffic over equal cost routes.

You can try the following to find the egress interface:

show fabricpath load-balance unicast forwarding-path ftag ftag-id switchid switch-id flow-type {l2 {{dst-mac dst-mac | source-mac src-mac} ether-type ether-type}} | {l3 {dst-ip dst-ip | src-ip src-ip | dst-ipv6 dst-ipv6 | srcipv6 src-ipv6}} | {l4 {l4-src-port l4-src-port | l4-dst-port l4-dst-port | dst-ip dst-ip | src-ip src-ip | dst-ipv6 dst-ipv6 | srcipv6 src-ipv6}}} {vlan vlan-id} {module mod-no}

Example:

switch# show fabricpath load-balance unicast forwarding-path ftag 1 switchid 200 src-mac 
00:10:20:30:40:50 dst-mac 00:30:40:50:60:70 vlan 200
Missing params will be substituted by 0's.
crc8_hash: 229
This flow selects interface Po400

Cheers,

Sergiu

[romanroma]

@Sergiu.Daniluk 

Do you recommend using src-mac as one of the local interfaces of the switch or possibly use the mac address of a network device?

[Sergiu.Daniluk]

source and destination of the user traffic

[Joseph W. Doherty]

Basically, vPC (and VSS too) will always prefer a same device port rather than using the cross (East-West) link between peers for egress data traffic. This because often the cross (East-West) link is sort of a "donut spare tire" for data traffic, i.e. it's intended to deal with a same device failure of a link, port, ASIC or line card.

[romanroma]

@Joseph W. Doherty,
I appreciate your help... I believe that is what is taking place, I am not seeing traffic go over my East-West link, so I think there is a mesh switch that is in leaf-spine topology that is being used. I am very new to vPC and Fabricpath.

Much appreciated... I will keep this in mind.