Nexus 9396PX - VPC, port-priority and max-bundle settings

[ss1]

Hi

I hope everybody here is fine. I'm willing to ask about a working scenario for VPC over Nexus 9396PX switch plus lacp port-priority and max-bundle parameters in action.

Our current case is as follows:
Nexus 9396 switch <----20G lacp----> Nexus 3064 switch
There are two routes/ports between the 9396 and the 3064 switch. One of the routes is less reliable so I set on the reliable pair of ports: lacp port-priority 8 and on the unreliable pair of ports: lacp port-priority 16.
Then I set lacp max-bundle 1 on the Port-channel interface. Hence, the reliable pair of ports is always bundled and the other one is Hot standby. That's exactly how I need it to work and it works. 

Now, my idea is to install another one Nexus 9396 switch and bring a VPC up in that scenario. So it would be:
Nexus 9396 switch A/B <-----10G+10G lacp-----> Nexus 3064 switch

The question is, if the VPC concept/algorithm already exchanges the max-bundle and port-priorities throughout the VPC peers. So if I set lacp port-priority 8 on the port on switch A and lacp port-priority 16 on the port on switch B, and then, max-bundle 1 on the LACPs of both switches, are they going to know about each other's configuration and am I going to get a hot standby port on switch B resulting in an orphan link in this case? Or are these parameters only considered locally and independently by each of the VPC nodes?

Thank you,
Stefan

[Christopher Hart · Accepted solution]

Hello!

Unfortunately, the maximum number of members allowed in an LACP port-channel (lacp max-bundle {x}) as well as LACP port priorities (lacp port-priority {x}) are not communicated between vPC peers through a vPC Consistency Check or CFS (Cisco Fabric Services). These parameters are locally significant and are independent to each vPC peer.

As a result, with your proposed topology, one of the vPC port-channel members will not going into a Hot-Standby state. To the best of my knowledge, there is not a way to accomplish this within the confines of vPC. Just to be safe, I tested this in my lab - see below:

+++ N9K-C93180YC-FX running NX-OS 9.3(5).

N9K-1# show module
<snip>
Mod Ports             Module-Type                      Model           Status
--- ----- ------------------------------------- --------------------- ---------
1    54   48x10/25G/32G + 6x40/100G Ethernet/FC N9K-C93180YC-FX       active *  

Mod  Sw                       Hw    Slot
---  ----------------------- ------ ----
1    9.3(5)                   1.1    NA  

+++ Port-channel10 has `lacp max-bundle 1` configured and consists of Ethernet1/3, which has `lacp port-priority 8` configured. 

N9K-1# show running-config interface port-channel10 membership 
<snip>
interface port-channel10
  switchport
  switchport mode trunk
  switchport trunk allowed vlan none
  mtu 9216
  lacp max-bundle 1
  vpc 10

interface Ethernet1/3
  lacp port-priority 8 
  switchport
  switchport mode trunk
  switchport trunk allowed vlan none
  spanning-tree port type edge trunk
  mtu 9216
  channel-group 10 mode active
  no shutdown

+++ Ethernet1/3 is up in Port-channel10.

N9K-1# show port-channel summary interface port-channel10
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        b - BFD Session Wait
        S - Switched    R - Routed
        U - Up (port-channel)
        p - Up in delay-lacp mode (member)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
10    Po10(SU)    Eth      LACP      Eth1/3(P)    

+++ Ethernet1/3 connects to Ethernet1/45 of a downstream Nexus 3064PQ switch.

N9K-1# show cdp neighbors interface Ethernet1/3
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
N3K.cisco.com(ABCD12345)
                    Eth1/3         147    R S I s   N3K-C3064PQ   Eth1/45       

Total entries displayed: 1

+++ Remote vPC peer is also a Nexus 93180YC-FX running NX-OS 9.3(5)

N9K-2# show module 
<snip>
Mod Ports             Module-Type                      Model           Status
--- ----- ------------------------------------- --------------------- ---------
1    54   48x10/25G/32G + 6x40/100G Ethernet/FC N9K-C93180YC-FX       active *  

Mod  Sw                       Hw    Slot
---  ----------------------- ------ ----
1    9.3(5)                   1.1    NA  

+++ Port-channel10 has `lacp max-bundle 1` configured and consists of Ethernet1/3, which has `lacp port-priority 16` configured. 

N9K-2# show running-config interface port-channel10 membership 
<snip>>
interface port-channel10
  switchport
  switchport mode trunk
  switchport trunk allowed vlan none
  mtu 9216
  lacp max-bundle 1
  vpc 10

interface Ethernet1/3
  lacp port-priority 16 
  switchport
  switchport mode trunk
  switchport trunk allowed vlan none
  spanning-tree port type edge trunk
  mtu 9216
  channel-group 10 mode active
  no shutdown

+++ Ethernet1/3 is up in Port-channel10.

N9K-2# show port-channel summary interface port-channel10
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        b - BFD Session Wait
        S - Switched    R - Routed
        U - Up (port-channel)
        p - Up in delay-lacp mode (member)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
10    Po10(SU)    Eth      LACP      Eth1/3(P)    

+++ Ethernet1/3 connects to Ethernet1/46 of the same downstream Nexus 3064PQ switch.

N9K-2# show cdp neighbors interface Ethernet1/3
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
N3K.cisco.com(ABCD12345)
                    Eth1/3         131    R S I s   N3K-C3064PQ   Eth1/46       

Total entries displayed: 1

+++ From the N3K's perspective, both physical interfaces are properly bundled into Port-channel10.

N3K# show module 
<snip>
Mod Ports             Module-Type                       Model          Status
--- ----- ------------------------------------- --------------------- ---------
1    52   48x10GE + 16x10G/4x40G Supervisor     N3K-C3064PQ           active *  

Mod  Sw                       Hw    Slot
---  ----------------------- ------ ----
1    9.3(3)                   0.3    NA  

N3K# show running-config interface port-channel1 membership 
<snip>
interface port-channel1

interface Ethernet1/45
  channel-group 1 mode active

interface Ethernet1/46
  channel-group 1 mode active

N3K# show port-channel summary interface port-channel1
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        b - BFD Session Wait
        S - Switched    R - Routed
        U - Up (port-channel)
        p - Up in delay-lacp mode (member)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth1/45(P)   Eth1/46(P)  

We don't have the details of your configuration, but if you're using lacp rate fast, you have a 3-second failover in the event of a worst case failure scenario on either your reliable or unreliable link (such as a unidirectional WAN link or some other issue that doesn't result in an immediate link-down scenario). If you're not using lacp rate fast, you have a 90-second failover in the event of a worst case failure scenario. 

If you're not using lacp rate fast, it might be in your best interest from a failover scenario to use traditional Spanning Tree Protocol to block the unreliable link instead of using LACP to suspend it. Spanning Tree would likely be able to reconverge the network faster than LACP (although I've not explicitly tested this - your mileage may vary!)

I hope this helps!

-Christopher