Nexus 9k - traffic between orphan ports not forwarded via vpc peer-link

[lacyk]

Hi,

we are trying to attach windows and redhat servers on vpc portchannels to FEXes (n2248 hooked on N9K-C9372PX-E) (servers set to active-active/LACP). Even if switch looks fine - lacp neighbor info is correct, MAC addresses are learnt on portchannels (all L2 access), some servers are not able to ping each other (they are all on the same vlan) - they even don't learn ARP from the others.

For troubleshooting purpose we turned servers to active-standby and configured switchports on FEXes as standard access ports without portchannel - they effectively all became orphan ports in the vlan. With this setup servers can ping each other only if all NICs are active on the same switch. Once we move active NIC to another switch - servers are not able to ping each other. In all cases MAC address table looks correctly and MAC addresses are learnt where they should be (including entries for vpc-peerlink). We use nx-os 7.0(3)I5(1)

[Rick1776]

Can you send the output from show vpc? It sounds like the VLANs aren’t allowed on the VPC

[Rick1776]

Also make sure that you only have the Port Channel configured with the options and under the physical interface only has channel-group xxx and no shut.

[lacyk]

Hi Rick,

when vPC was not working we wanted to have at least some resiliency and went to active-standby. Diagram shows what doesn't work - when both servers have active NICs on one FEX, they can reach each other, if active NICs are on different switches, they can't. All four FEX ports are simple access ports all in the same vlan. Vlan is allowed on peerlink and MAC addresses of servers are seen on the switches.

leaf03# sh run vpc
version 7.0(3)I5(1)
feature vpc
vpc domain 2
  peer-switch
  peer-keepalive destination 10.0.0.4 source 10.0.0.3
  peer-gateway
  layer3 peer-router
  no layer3 peer-router syslog
  auto-recovery
  ipv6 nd synchronize
  ip arp synchronize

interface port-channel1
  vpc peer-link

....show vpc doesn't show related vlan as we have many vlans in campus. It shows first 6 rows of vlans and then "...". But show int trunk displays vlan in forwarding not-pruned state on vpc peerlink. Also MAC address learning across peelink works fine for that vlan.

[Rick1776]

What the member ports look like as far as the configuration? Can you send the show int for the member ports.

Thanks.

[lacyk]

Hi Rick,

attaching in the file.

ports e106/1/6 on leaf3 and e106/1/4 on leaf4 are facing one server with NICs teamed in active/standby mode. There are several more servers dualhomed in exactly the same way with the same port config.

[Rick1776]

You don't have any orphan ports configured on the VPC domain 

Your configuration should look like the following. It will only be active at one N9K switch at a time and once the primary VPC switche falls the other port will kick in.

configure terminal
switch(config)# interface ethernet 3/1
switch(config-if)# vpc orphan-ports suspend
switch(config-if)# exit

Not sure why you want to set it up this way. You would be better off setting it up like the image I attached.

So basically it doesn't know that it's an orphan port unless you configure it that way.

Also CFS is running on the switches with the "show cfs status" command.

These are pretty good links...

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/interfaces/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_Interfaces_Configuration_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Interfaces_Configuration_Guide_chapter_0111.html#...

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/design_guide_c07-625857.html#_Toc271759441

[lacyk]

what you suggest here is "You can explicitly declare physical interfaces as orphan ports to be suspended (shut down) by the secondary peer when it suspends its vPC ports in response to a peer link or peer-keepalive failure"

this is not what I need. I don't need to react on vpc failures, vpc is stable an up.
We have another issue now where singlehomed server (obviously an orphan port)can't be pinged from other vpc peer. MAC addresses and ARP tables are correctly populated, but when trying to ping server, TX unicast counter doesn't increase on server facing port.

Your diagram is how we connected it originally, but even that was not working correctly - sometimes servers could ping each other, sometimes they couldn't. So we decided to rule out portchannel from troubleshooting and came to issue with orphaned ports - which seems to be growing

[MambaRod16]

Hello Cizmart.att,

If you configured port- channel 106 as a vPC port member I think you have a design and configuration issue. If that is the case the issue is that the traffic between vPC member ports accross the Peer Link are dropped because in a correct vPC design this would be a duplicated traffic. Also It is not correct to configure the same port-channel with two independent devices unless they are running vPC (the access switches of your diagram aren't running vPC).

The recommended design is to connect both access switches to each Leafs switches using diferrent Port-channel per access switch.  Both port-channel have to be configured on Leafs as a vPC member port.

HTH

[lacyk]

short update - it seems to be caused by cisco bug CSCvc12950. 

this log entry pointed us to that:

%DAEMON-2-SYSTEM_MSG: Table Full/Hash collision Vxlan VP 4885, vlan 829, Please reduce the number of Vxlan VPs  - pixmc[26701]

we upgraded to 7.0.3.I7.3 and testing, seems to be fixed

[Vl@d@Ni]

I am experiencing exactly the same problem but without the FEXs. 

In my case upgrade helped just for 2 days and then again the same problem happen.