How does VPC allocates port identifier?

[Difan_Zhao]

Please look at this output

# show lacp neighbor interface po2
Flags:  S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
port-channel2 neighbors
Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/45     32667,0-23-4-ee-be-64  0x109           2348889     SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x800b                      0x3d

Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/46     32667,0-23-4-ee-be-64  0x4109          16639657    SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x800b                      0x3d

The other side is also the Cisco Nexus switch operating in the VPC. 0x109 is the E1/3 interface. I know that the above switch connects to two different switches but on the same port E1/3. There is no port with the Port ID of 0x4109.

# show lacp interface e1/3
...
Local Port: Eth1/3   MAC Address= d4-c9-3c-23-d3-5f
  System Identifier=0x8000,  Port Identifier=0x8000,0x109

Do you think the VPC would add 0x4000 to the Port ID for the ports on the secondary unit? Is there a document about that? I can't find a command to show the LACP Port ID in the VPC scenario. I think the "show lacp" command only shows the LACP status in the standalone scenario. 

Let me know what you think

Thanks!

Difan

[Sergiu.Daniluk]

Hi @Difan_Zhao 

I have no available devices to test this, however I am curious why do you need this information? To rephrase my question, in which scenario/context would you use this knowledge on how the code on NXOS was implemented to assign the PortID in a vPC?

Note: If you think this will be useful in some troubleshooting sessions I am telling you it's not. Not even if you work in TAC.  Probably you would need the info if you work as a developer for INSBU  (N9K BU). But other than that, I cannot think of any instance where you would need such information.

Again, just out of curiosity. 🙂

Cheers,

Sergiu

[Difan_Zhao]

Hi Sergiu, thanks for the response. One reason is that I am just curious how it works. Another reason which is more important is that I am making a diagram that documents the connections between the switches and the environment doesn't allow CDP or LLDP to be turned on. The only way I can think of to find the neighbor is with the LACP (luckily that the environment uses LACP most of the time)

[Sergiu.Daniluk]

Fair point with the no-CDP/LLDP, but supposing that vPC will do some tweaks with the Partner's PortID, it looks to me that it will be impossible to identify which interface is connected to which vpc peer, when identical Port Number is used for the same vpc (i.e. eth1/10 on both peers configured with vpc 10).

Anyway, I will have a look next week and see if the results are similar with what you have there.

Just out of curiosity, what Nexus platform and NXOS version you've done the testings on?

Cheers,

Sergiu

[Difan_Zhao]

Thanks, Sergiu. I hope the VPC primary will assign E1/10 with port ID 0x110 and the secondary will assign E1/10 with 0x4110. Then it will be certain which ports connect to which. I just can't find a command to verify the allocation. I only see the Port ID on the neighbor but not on the local switches.

[Christopher Hart · Accepted solution]

Hello!

The vPC peer with the higher system MAC address will set the most significant bit of the actor port ID advertised by LACPDUs on a vPC to "1". This typically causes the vPC peer with the higher system MAC address to send LACPDUs with an actor port ID that begins with "0x4", while the vPC peer with the lower system MAC address sends LACPDUs with an unmodified actor port ID.

An example of this from my lab is shown below.

N9K-1# show vpc role

vPC Role status
----------------------------------------------------
vPC role                        : primary, operational secondary
Dual Active Detection Status    : 0
vPC system-mac                  : 00:23:04:ee:be:01             
vPC system-priority             : 32667
vPC local system-mac            : 00:de:fb:fb:50:e7    <<< Local is higher
vPC local role-priority         : 150 
vPC local config role-priority  : 150 
vPC peer system-mac             : 00:de:fb:fa:64:c7    <<< Remote is lower
vPC peer role-priority          : 32667
vPC peer config role-priority   : 32667

N9K-1# show running-config interface port-channel 10 membership 
<snip>
interface port-channel10
  description vPC 10 (Configured by Ansible)
  switchport mode trunk
  switchport access vlan 10
  spanning-tree port type edge trunk
  mtu 9216
  vpc 10

interface Ethernet1/3
  description vPC 10 member (Configured by Ansible)
  switchport mode trunk
  switchport access vlan 10
  spanning-tree port type edge trunk
  mtu 9216
  channel-group 10 mode active

N9K-1# ethanalyzer local interface front-panel ethernet1/3 display-filter slow limit-captured-frames 0 detail 
Capturing on front panel interface
Frame 2 (124 bytes on wire, 124 bytes captured)
    Arrival Time: Aug 22, 2021 12:11:24.566457000
    [Time delta from previous captured frame: 18.910178000 seconds]
    [Time delta from previous displayed frame: 18.910178000 seconds]
    [Time since reference or first frame: 18.910178000 seconds]
    Frame Number: 2
    Frame Length: 124 bytes
    Capture Length: 124 bytes
    [Frame is marked: False]
    [Protocols in frame: eth:slow]
Ethernet II, Src: 00:de:fb:fb:50:ea (00:de:fb:fb:50:ea), Dst: 01:80:c2:00:00:02 (01:80:c2:00:00:02)
    Destination: 01:80:c2:00:00:02 (01:80:c2:00:00:02)
        Address: 01:80:c2:00:00:02 (01:80:c2:00:00:02)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Source: 00:de:fb:fb:50:ea (00:de:fb:fb:50:ea)
        Address: 00:de:fb:fb:50:ea (00:de:fb:fb:50:ea)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
    Type: Slow Protocols (0x8809)
Link Aggregation Control Protocol
    Slow Protocols subtype: LACP (0x01)
    LACP Version Number: 0x01
    Actor Information: 0x01
    Actor Information Length: 0x14
    Actor System Priority: 32667
    Actor System: 00:23:04:ee:be:01 (00:23:04:ee:be:01)
    Actor Key: 32778
    Actor Port Priority: 32768
    Actor Port: 16649    <<< Translates to hex value of 0x4109
    Actor State: 0x3d (Activity, Aggregation, Synchronization, Collecting, Distributing)
        .... ...1 = LACP Activity: Yes
        .... ..0. = LACP Timeout: No
        .... .1.. = Aggregation: Yes
        .... 1... = Synchronization: Yes
        ...1 .... = Collecting: Yes
        ..1. .... = Distributing: Yes
        .0.. .... = Defaulted: No
        0... .... = Expired: No
    Reserved: 000000
    Partner Information: 0x02
    Partner Information Length: 0x14
    Partner System Priority: 32768
    Partner System: 00:05:73:d9:9c:01 (00:05:73:d9:9c:01)
    Partner Key: 1
    Partner Port Priority: 32768
    Partner Port: 259
    Partner State: 0x3d (Activity, Aggregation, Synchronization, Collecting, Distributing)
        .... ...1 = LACP Activity: Yes
        .... ..0. = LACP Timeout: No
        .... .1.. = Aggregation: Yes
        .... 1... = Synchronization: Yes
        ...1 .... = Collecting: Yes
        ..1. .... = Distributing: Yes
        .0.. .... = Defaulted: No
        0... .... = Expired: No
    Reserved: 000000
    Collector Information: 0x03
    Collector Information Length: 0x10
    Collector Max Delay: 0
    Reserved: 000000000000000000000000
    Terminator Information: 0x00
    Terminator Length: 0x00
    Reserved: 000000000000000000000000000000000000000000000000...

N9K-2# ethanalyzer local interface front-panel ethernet1/3 display-filter slow limit-captured-frames 0 detail 
Capturing on 'Eth1-3'
1 Frame 2: 124 bytes on wire (992 bits), 124 bytes captured (992 bits) on interface Eth1-3, id 0
    Interface id: 0 (Eth1-3)
        Interface name: Eth1-3
    Encapsulation type: Ethernet (1)
    Arrival Time: Aug 22, 2021 12:11:24.516825212 UTC
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1629634284.516825212 seconds
    [Time delta from previous captured frame: 18.922381756 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 18.922381756 seconds]
    Frame Number: 2
    Frame Length: 124 bytes (992 bits)
    Capture Length: 124 bytes (992 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:slow:lacp]
Ethernet II, Src: 00:de:fb:fa:64:ca, Dst: 01:80:c2:00:00:02
    Destination: 01:80:c2:00:00:02
        Address: 01:80:c2:00:00:02
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: 00:de:fb:fa:64:ca
        Address: 00:de:fb:fa:64:ca
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: Slow Protocols (0x8809)
Slow Protocols
    Slow Protocols subtype: LACP (0x01)
Link Aggregation Control Protocol
    LACP Version: 0x01
    TLV Type: Actor Information (0x01)
    TLV Length: 0x14
    Actor System Priority: 32667
    Actor System ID: 00:23:04:ee:be:01
    Actor Key: 32778
    Actor Port Priority: 32768
    Actor Port: 265    <<< Translates to hex value of 0x109
    Actor State: 0x3d, LACP Activity, Aggregation, Synchronization, Collecting, Distributing
        .... ...1 = LACP Activity: Active
        .... ..0. = LACP Timeout: Long Timeout
        .... .1.. = Aggregation: Aggregatable
        .... 1... = Synchronization: In Sync
        ...1 .... = Collecting: Enabled
        ..1. .... = Distributing: Enabled
        .0.. .... = Defaulted: No
        0... .... = Expired: No
    [Actor State Flags: **DCSG*A]
    Reserved: 000000
    TLV Type: Partner Information (0x02)
    TLV Length: 0x14
    Partner System Priority: 32768
    Partner System: 00:05:73:d9:9c:01
    Partner Key: 1
    Partner Port Priority: 32768
    Partner Port: 260
    Partner State: 0x3d, LACP Activity, Aggregation, Synchronization, Collecting, Distributing
        .... ...1 = LACP Activity: Active
        .... ..0. = LACP Timeout: Long Timeout
        .... .1.. = Aggregation: Aggregatable
        .... 1... = Synchronization: In Sync
        ...1 .... = Collecting: Enabled
        ..1. .... = Distributing: Enabled
        .0.. .... = Defaulted: No
        0... .... = Expired: No
    [Partner State Flags: **DCSG*A]
    Reserved: 000000
    TLV Type: Collector Information (0x03)
    TLV Length: 0x10
    Collector Max Delay: 0
    Reserved: 000000000000000000000000
    TLV Type: Terminator (0x00)
    TLV Length: 0x00
    Pad: 000000000000000000000000000000000000000000000000…

N9K-1# hex 16649
0x4109

N9K-1# hex 265
0x109

From a protocol design perspective, this makes sense for two reasons:

1. vPC is designed such that a device connected to both vPC peers through a vPC does not know it's connected to two separate logical Nexus switches. From the device's perspective, both Nexus switches operate as one. Therefore, both Nexus switches cannot send identical LACPDUs - the Actor Port ID field must be different between the two, or else the vPC-connected device will appear to receive an LACPDU from the same remote physical port on two different interfaces. This would most likely cause the device to suspend its LACP port-channel due to misconfiguration.
2. The vPC peer that modifies its Actor Port ID field must remain consistent and be agnostic of the vPC role. If this action was tied to the vPC role of the switch (such as the vPC Primary/Operational Primary), then in failure scenarios where the vPC Primary/Operational Primary goes offline (due to a reload, power outage, etc.), the vPC Secondary would suddenly start modifying the Actor Port ID of LACPDUs it sends. From the vPC-connected device's perspective, it would start receiving a completely different LACPDU after one member of its port-channel goes down (facing the reloaded vPC peer). This would probably cause the LACP finite state machine on that device to restart, thus causing the vPC to go down and interrupting traffic.

I hope this helps - thank you!

[Difan_Zhao]

This Christopher, thank you very much for the detailed explanation! It is very clear. So just to confirm

1. It is totally up to the system-mac, and not the configured "role priority" for adding the 0x4000 (or setting the highest bit) for the Port ID? I totally understand that it shouldn't change with VPC role changes, but just wondering if it would still have an influence on the initial assignment, but will stay after even with VPC failover

2. There is no command other than the "ethanalyzer" to verify the port ID? lol

Thanks again for the research! 

Difan

[Christopher Hart]

Hi Difan,

Answers to your questions are below:

1. Correct - this behavior is totally up to the system MAC address. The vPC role and role priority have no factor into modifying this behavior.
2. Also correct - I am not aware of any command that shows the true Actor Port ID other than through an Ethanalyzer control plane packet capture. It might be buried deep within an internal LACP command somewhere, but I wasn't able to find any commands that show it after a brief search. 

Thank you!

-Christopher