Nexus 9396px: Unable to make xconnect work

[ss1]

Hello,

I'm trying to do a xconnect between two 10G ports on two different Nexus 9396PX switches, however, unable to do it in spite of following this document for guidelines: https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/213958-nexus-9000-vxlan-xconnect-configuration.html

So far I have completed the steps for enabling and configuring the ngoam feature as recommended on the above link, however I'm struggling to complete the xconnect peering due to the following issue:

# show ngoam xconnect session 3092
Vlan ID: 3092
Peer IP: 10.128.164.139  VNI : 9003092
State: Heartbeat loss, 
Last state update: 11/29/2021 17:21:00.321
Local interface: Eth1/16  State:  ERR
Local vpc interface Unknown  State:  DOWN
Remote interface: Unknown  State:  DOWN
Remote vpc interface: Unknown  State:  DOWN

The heartbeat loss is also seen on the peer switch, however all other diagnostic effort don't seem to show any issue evidence:

# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 5, Local Router ID is 10.128.44.182
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 10.128.44.182:35859    (L2VNI 9003092)
*>l[3]:[0]:[32]:[10.128.144.182]/88
                      10.128.144.182                    100      32768 i
*>i[3]:[0]:[32]:[10.128.164.139]/88
                      10.128.164.139                    100          0 i

Route Distinguisher: 10.128.64.139:35859
* i[3]:[0]:[32]:[10.128.164.139]/88
                      10.128.164.139                    100          0 i
*>i                   10.128.164.139                    100          0 i


# show nve peers
Interface Peer-IP                                 State LearnType Uptime   Router-Mac       
--------- --------------------------------------  ----- --------- -------- -----------------
nve1      10.128.164.139                          Up    CP        17:32:39 n/a              

# show nve vni 9003092
Codes: CP - Control Plane        DP - Data Plane          
       UC - Unconfigured         SA - Suppress ARP        
       SU - Suppress Unknown Unicast 
       Xconn - Crossconnect      
       MS-IR - Multisite Ingress Replication
 
Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1      9003092  225.30.9.2        Up    CP   L2 [3092]          Xconn 

# show ip bgp all summary 
BGP summary information for VRF default, address family IPv4 Unicast

BGP summary information for VRF default, address family IPv6 Unicast

BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.128.44.182, local AS number xxxxxxx
BGP table version is 5, L2VPN EVPN config peers 2, capable peers 2
3 network entries and 4 paths using 732 bytes of memory
BGP attribute entries [3/516], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.128.2.183    4 xxxxxxx   14037    1058        5    0    0 17:33:31 1         
10.128.2.184    4 xxxxxxx   14033    1058        5    0    0 17:33:29 1         
# show running-config bgp 
version 9.3(8) Bios:version 07.69 
feature bgp

router bgp xxxxxxx
  router-id 10.128.44.182
  address-family l2vpn evpn
    maximum-paths 64
    maximum-paths ibgp 64
  neighbor 10.128.2.183
    remote-as xxxxxxx
    description SPINEswitch1
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.128.2.184
    remote-as xxxxxxx
    description SPINEswitch2
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
evpn
  vni 9003092 l2
    rd auto
    route-target import auto
    route-target export auto

PIM-RP has been also configured all along the path, however, not sure about the following:
- the 9396px is not on the xconnect supported list, however, the 9372px is, I thought they are almost identical by means of supported features, were I wrong in assuming this?
- do I need to enable ngoam all along the spine paths between all switches, I didn't do it yet, the spines are BGP RR, PIM and OSPF routers so far?
- last but not least, is there any tcam carving necessary in order to make it work?

Any ideas on how to make it work will be appreciated greatly.

Thank you.

[ss1]

Hello,

I have managed to progress this a bit, I feel I'm closer to the expected end result however there's no communication yet over the xconnect...

I did some tcam carving as follows:

hardware access-list tcam region ifacl 256 
hardware access-list tcam region vacl 0 
hardware access-list tcam region racl 256 
hardware access-list tcam region e-racl 0 
hardware access-list tcam region span 0 
hardware access-list tcam region arp-ether 256 double-wide

This brought an end result where the xconnect session shows as up and running on both switches in the lab:

States: LD = Local interface down, RD = Remote interface Down
          HB = Heartbeat lost, DB = Database/Routes not present
          * - Showing Vpc-peer interface info
Vlan          Peer-ip/vni      XC-State       Local-if/State    Rmt-if/State
===============================================================================
309210.128.144.182 / 9003092      Active       Eth1/37 / UP       Eth1/37 / UP

However no transport over the line. I'm pushing some multicast traffic to the Eth1/37.

interface Ethernet1/37 
 switchport mode dot1q-tunnel 
 switchport access vlan 3092 
 spanning-tree bpdufilter enable 
 mtu 9216

Ethernet1/37 is up
    30 seconds input rate 1538270328 bits/sec, 140794 packets/sec
    30 seconds output rate 0 bits/sec, 0 packets/sec

However nothing is going out through the (only) underlay port I have at the moment:

Ethernet1/28 is up
    30 seconds input rate 30488 bits/sec, 41 packets/sec
    30 seconds output rate 24976 bits/sec, 41 packets/sec

Here's a list of type 3 routes exchanged:

# show bgp l2vpn evpn route-type 3
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.128.44.182:35859
BGP routing table entry for [3]:[0]:[32]:[10.128.144.182]/88, version 13
Paths: (2 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Multipath: eBGP iBGP

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-9003092
  AS-Path: NONE, path sourced internal to AS
    10.128.144.182 (metric 9) from 10.128.2.183 (10.128.2.183)
      Origin IGP, MED not set, localpref 100, weight 0
      Extcommunity: RT:65000:9003092 ENCAP:8
      Originator: 10.128.44.182 Cluster list: 10.128.2.183 
      PMSI Tunnel Attribute:
        flags: 0x00, Tunnel type: Ingress Replication
        Label: 9003092, Tunnel Id: 10.128.144.182

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    10.128.144.182 (metric 9) from 10.128.2.184 (10.128.2.184)
      Origin IGP, MED not set, localpref 100, weight 0
      Extcommunity: RT:65000:9003092 ENCAP:8
      Originator: 10.128.44.182 Cluster list: 10.128.2.184 
      PMSI Tunnel Attribute:
        flags: 0x00, Tunnel type: Ingress Replication
        Label: 9003092, Tunnel Id: 10.128.144.182

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.128.54.168:35859    (L2VNI 9003092)
BGP routing table entry for [3]:[0]:[32]:[10.128.144.182]/88, version 14
Paths: (1 available, best #1)
Flags: (0x000012) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW
Multipath: eBGP iBGP

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported from 10.128.44.182:35859:[3]:[0]:[32]:[10.128.144.182]/88 
  AS-Path: NONE, path sourced internal to AS
    10.128.144.182 (metric 9) from 10.128.2.183 (10.128.2.183)
      Origin IGP, MED not set, localpref 100, weight 0
      Extcommunity: RT:65000:9003092 ENCAP:8
      Originator: 10.128.44.182 Cluster list: 10.128.2.183 
      PMSI Tunnel Attribute:
        flags: 0x00, Tunnel type: Ingress Replication
        Label: 9003092, Tunnel Id: 10.128.144.182

  Path-id 1 not advertised to any peer
BGP routing table entry for [3]:[0]:[32]:[10.128.154.168]/88, version 2
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn
Multipath: eBGP iBGP

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    10.128.154.168 (metric 0) from 0.0.0.0 (10.128.54.168)
      Origin IGP, MED not set, localpref 100, weight 32768
      Extcommunity: RT:65000:9003092 ENCAP:8
      PMSI Tunnel Attribute:
        flags: 0x00, Tunnel type: Ingress Replication
        Label: 9003092, Tunnel Id: 10.128.154.168

  Path-id 1 advertised to peers:
    10.128.2.183       10.128.2.184   

Switch 2:

# show bgp l2vpn evpn route-type 3
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.128.44.182:35859    (L2VNI 9003092)
BGP routing table entry for [3]:[0]:[32]:[10.128.144.182]/88, version 2
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn
Multipath: eBGP iBGP

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    10.128.144.182 (metric 0) from 0.0.0.0 (10.128.44.182)
      Origin IGP, MED not set, localpref 100, weight 32768
      Extcommunity: RT:65000:9003092 ENCAP:8
      PMSI Tunnel Attribute:
        flags: 0x00, Tunnel type: Ingress Replication
        Label: 9003092, Tunnel Id: 10.128.144.182

  Path-id 1 advertised to peers:
    10.128.2.183       10.128.2.184   
BGP routing table entry for [3]:[0]:[32]:[10.128.154.168]/88, version 376
Paths: (1 available, best #1)
Flags: (0x000012) (high32 00000000) on xmit-list, is in l2rib/evpn, is not in HW
Multipath: eBGP iBGP

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported from 10.128.54.168:35859:[3]:[0]:[32]:[10.128.154.168]/88 
  AS-Path: NONE, path sourced internal to AS
    10.128.154.168 (metric 13) from 10.128.2.183 (10.128.2.183)
      Origin IGP, MED not set, localpref 100, weight 0
      Extcommunity: RT:65000:9003092 ENCAP:8
      Originator: 10.128.54.168 Cluster list: 10.128.2.183 
      PMSI Tunnel Attribute:
        flags: 0x00, Tunnel type: Ingress Replication
        Label: 9003092, Tunnel Id: 10.128.154.168

  Path-id 1 not advertised to any peer

Route Distinguisher: 10.128.54.168:35859
BGP routing table entry for [3]:[0]:[32]:[10.128.154.168]/88, version 195
Paths: (2 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Multipath: eBGP iBGP

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-9003092
  AS-Path: NONE, path sourced internal to AS
    10.128.154.168 (metric 13) from 10.128.2.183 (10.128.2.183)
      Origin IGP, MED not set, localpref 100, weight 0
      Extcommunity: RT:65000:9003092 ENCAP:8
      Originator: 10.128.54.168 Cluster list: 10.128.2.183 
      PMSI Tunnel Attribute:
        flags: 0x00, Tunnel type: Ingress Replication
        Label: 9003092, Tunnel Id: 10.128.154.168

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    10.128.154.168 (metric 13) from 10.128.2.184 (10.128.2.184)
      Origin IGP, MED not set, localpref 100, weight 0
      Extcommunity: RT:65000:9003092 ENCAP:8
      Originator: 10.128.54.168 Cluster list: 10.128.2.184 
      PMSI Tunnel Attribute:
        flags: 0x00, Tunnel type: Ingress Replication
        Label: 9003092, Tunnel Id: 10.128.154.168

  Path-id 1 not advertised to any peer

Any ideas on how to continue the troubleshooting efforts will be appreciated.

Thank you everybody for the time spent on this. 

[ss1]

Hello,

I just had the opportunity to test the same configuration on a 9364C switch and it worked right away.
However, 9372 and 9396 continue to behave as if there's absolutely no traffic over the xconnect circuit. The ngoam session is established and all ports are UP on both sides however interface counters remain zero. 

I think I'm missing some tcam carving in addition to the arp-ether 256 double-wide. I'm continuing to spend a lot of time in trying out lots of stuff however no luck so far.

[ss1 · Accepted solution]

It appears I'm talking to myself here lol
Let me share the final outcome -> the case is now resolved and the xconnect works as resolved <-

There is a requirement to do tcam carving on vxlan-p2p to 256 or more, nothing of this kind is documented anywhere however I found it in my tryouts.

Example setting:

hardware access-list tcam region vacl 0
hardware access-list tcam region racl 0
hardware access-list tcam region span 0
hardware access-list tcam region arp-ether 256 double-wide
hardware access-list tcam region vxlan-p2p 256

Applies to 9372 and 9396 /confirmed working with the above carving/ and also 9364C /confirmed working without any tcam carving at all/.

[Channels]

Hello everyone
I allow myself to post this topic because I have exactly the same problem, even by applying the hardware commands.
We have C9372PX in 9.3.11.

I don't know how to debug?

Vlan ID: 1103
Peer IP: 10.2.4.10 VNI: 1103
State: Heartbeat loss,
Last state update: 04/14/2023 12:26:46.932
Local interface: Unknown State: ERR
Local vpc interface Unknown State: DOWN
Remote interface: Unknown State: DOWN
Remote vpc interface: Unknown State: DOWN

[ss1]

Dear friend,

it seems that your gear is not even reporting what the local interface is. Hence I suspect that you might not have configured the dot1q access vlan on the interface and the VLAN does not have xconnect enabled on it. 

vlan 1103
vn-segment xxxxxx
xconnect

Please double check that, thank you. 

[Channels]

Hello

I have this in both side :

vlan 1103
vn-segment 1103
xconnect

interface nve1
member vni 1103

evpn
vni 1103 l2
rd 65500:1103
route-target import auto
route-target export auto

interface Ethernet1/9
switchport mode dot1q-tunnel
switchport access vlan 1103