Linux Bonding Mode 1 to a Pair Nexus Switch

[mrdlnf]

Hi Experts,

I am trying to connect a Linux Server with 2x10Gbps to a pair Nexus switch as shown below :

The bonding itself is using Mode 1 Active-Backup, between Nexus switches there is a vPC peer-link, and these Nexus switches are connected to a pair Aggregation Switch. But somehow i saw both Nexus switches are learning the same mac address (mac address from the active one) from both 10Gbps port, the result : the traffic is stopped. Is there any clue how to solve this issue?

Thanks alot

[Manoj Papisetty]

Hello there,

It would be easier if you can share the configuration from the Nexus switches as well. If you are configuring them in vpc and the ports towards the server are also configured for vpc, it is expected to learn the mac address on both the switches.

If the intention is to use active-standby configuration, then you can simply connect both the links on regular trunk interfaces and not configure anything related to vpc on N9ks. When the traffic fails over, the mac gets learnt on the other switch and should cause no problems.

[mrdlnf]

Hi Manoj,

Thank you very much for your reply, the configuration is simple :

on Nexus, ports that connected to the server is only access port and assign to one VLAN.

Nexus-1

interface Ethernet1/16
  description
  switchport
  switchport access vlan 100
  no shutdown

interface port-channel1
  description to-Nexus-2
  switchport
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel28
  description PortChannel to Aggregation
  switchport
  switchport mode trunk
  spanning-tree port type network
  vpc 28

Nexus-2

interface Ethernet1/16
  description
  switchport
  switchport access vlan 100
  no shutdown

interface port-channel1
  description to-Nexus-1
  switchport
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel28
  description PortChannel to Aggregation
  switchport
  switchport mode trunk
  spanning-tree port type network
  vpc 28

Thanks alot

[Manoj Papisetty]

This configuration should work for an active standby setup, unless the standby link is sending some traffic for the switches to learn the mac.

Can you share the outputs for "show mac add interface eth1/16" from both the links? Unless there is some traffic, we should not learn the mac. On this Nexus 9k, you will be able to capture packets on the standby interface using SPAN to cpu. A sample configuration can be found here - https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_...

[mrdlnf]

Hi Manoj,

Here the result from both switch :

Nexus 1

sh mac address-table interface eth1/16
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 100     ac1f.6bab.74c3   dynamic  0         F      F    Eth1/16

Nexus 2

sh mac address-table interface eth1/16
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 100     ac1f.6bab.74c3   dynamic  0         F      F    Eth1/16

Thanks alot

[Manoj Papisetty]

This output shows that there are packets with this mac address coming in on both standby and active links. You would ideally want to check with the server team as to why it is sending such packets.

[mrdlnf]

Hi Manoj,

Sorry for late response, just want to check with you. In my case, if the Active NIC connected to port 16 Nexus-1, this port 16 of Nexus-1 will learn the mac address ac1f.6bab.74c3 meanwhile in Nexus-2, mac address ac1f.6bab.74c3 will learn at port-channel1 but port 16 of Nexus-2 will not learn that mac address. This is expected behavior, right?

Thanks alot

[Manoj Papisetty]

Yes, that should be it.

[mrdlnf]

Hi Manoj,

Thank you very much for your info. Just want to check one more thing, even though the passive nic status is disconnected, does port-channel 1 of Nexus-2 still learn the mac address ac1f.6bab.74c3? Where i can find the article/doc that explain this behavior?

Thanks alot

[kumarpadala]

Hi,

We recently bought a Nexus 9348 pair, and tried to establish the VPC with linux server (Active/backup bonding), but it didnt work, when both interfaces are up in Nexus pair, attached the scenario overview diagram & configuration, I am trying to find, did I miss any configuration.

VPC's are up, no vpc suspend interfaces, and port-channels are up & no spanning-tree blocked ports on all 3switches, am I missing something here.

NOTE : If I shutdown one interface on any of Nexus device, server started responding.

[Joshua Walton]

check the following guide

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/layer2/configuration/guide/b_Cisco_Nexus_7000_Series_NX-OS_Layer_2_Switching_Configuration_Guide/config_stp.html

[f00z]

Best practice is that anything connected to VPC pair should be in port channel , preferably using LACP, which is mode 4 on linux.  It works fine in this manner. I would suggest configuring the server links as port channels and using mode 4 and also use layer3+4 hash on linux server.   

Active/backup will work on the linux server but why do that when both ports can be active at the same time and follow best practice procedures?

Only time this changes is when using the new VPC fabric peering, then the orphan ports are the same as the port channel ports and it does not matter any more.