nve peers not coming up

[richbotham]

Hello All,

I would be greatful for some help with my L3VNI BGP EVP setup.

I have a very simple setup with 1 spine and 2 leafs as well as 2 Ubuntu VM's running on eve-ng in GCP

The version on all devices is nxos.7.0.3.I7.4.bin

This setup is to setup a L3VNI only - so no intra subnet traffic required.

There is no multicast configured - instead I have "ingress-replication protocol bgp"

The left hand leaf has vlan 10 - 172.16.10.0/24
The right hand leaf has vlan 20 - 172.16.20.0/24

All devices are running OSPF and iBGP, using route-reflectors.

Whilst I have investigated all possible avenues I must have missed something.

1 - my nve peers do not come up
2 - there is nothing learned in "show bgp l2vp evpn" although the peers are up and the leafs are route-reflector clients.
3 - I have attached a diagram, the 3 configs and show commands from each device

If anyone can assist and tell me what i have obviously missed i would be very grateful

Thanks
Rich

[MHM Cisco World]

Sorry but NVI interface is UP but each peer dont see other' this indeed issue

MHM

[Stefan Mihajlov]

Hi @richbotham 

Your BGP EVPN sessions are established, but the leafs are not actually advertising their VRF routes into EVPN. Because of that, no EVPN routes are exchanged and the NVE peers stay down. Once you enable the VRF routes to be advertised into EVPN, the peers will come up and you’ll start seeing entries in show bgp l2vpn evpn.

Best regards,
Stefan Mihajlov

Mark this post as Helpful if it helped you, and Accept as Solution if it resolved your question.

[MHM Cisco World]

I get issue here 

vlan 10
  vn-segment 10010

Each leaf different vlan vni under nve and hence there is no evpn between it

You need to add same vlan with same vni in each leaf

Other config as I see it OK

MHM

[Harold Ritter]

Hi @richbotham ,

It looks like you are missing the VRF configuration under BGP on leaf2:

router bgp 65123

  vrf CUSTOMER

    address-family ipv4 unicast

      network 172.16.20.0/24

[richbotham]

Thanks Harold

I now have this

leaf1

router bgp 65123
neighbor 11.11.11.1
remote-as 65123
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf CUSTOMER
address-family ipv4 unicast
network 172.16.10.0/24

leaf2

router bgp 65123
  address-family l2vpn evpn
  neighbor 11.11.11.1
    remote-as 65123
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf CUSTOMER
    address-family ipv4 unicast
      network 172.16.20.0/24

[MHM Cisco World · Accepted solution]

You edit the post?

Anyway 

vlan 1020
  vn-segment 101020

vrf context CUSTOMER
  vni 101020

You cannot use same VNI for vlan and l3 VRF 

You need to use different VNI

Also you need to advertise network under bgp via 

Network x.x.x.x mask x.x.x.x

[richbotham]

All,

I may have confused the issue a little - apologies.

In my leaf configs was un-needed config, shown below which has now been removed.

evpn
  vni 10010 l2
    rd auto
    route-target import auto
    route-target export auto

evpn
  vni 10020 l2
    rd auto
    route-target import auto
    route-target export auto

My aim is a L3 VNI and not a L2 VNI

 I'm afraid I don't understand this comment - apologies, as i'm trying to get a L3VNI working

Each leaf different vlan vni under nve and hence there is no evpn between it
You need to add same vlan with same vni in each leaf

on both leafs I understand that for a L2 VNI I "need to add same vni in each leaf under the nve interface " which i have labbed before and works fine.

Each leaf has 2 different subnets and vlans

leaf 1 is vlan 10, 172.16.10.0/24

leaf 2 is vlan 10, 172.16.20.0/24

Leaf 1 is as per below, what should i be changing for a L3 VNI to work

vlan 10
  vn-segment 10010
vlan 1020
  vn-segment 101020

vrf context CUSTOMER
  vni 101020
  address-family ipv4 unicast
vrf context management

interface Vlan1

interface Vlan10
  no shutdown
  vrf member CUSTOMER
  ip address 172.16.10.254/24

interface Vlan1020
  no shutdown
  vrf member CUSTOMER
  ip forward

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback0
  member vni 10010
    ingress-replication protocol bgp
  member vni 101020 associate-vrf

Leaf 2 is as per below, what should i be changing for a L3 VNI to work

vlan 20
  vn-segment 10020
vlan 1020
  vn-segment 101020

vrf context CUSTOMER
  vni 101020
  address-family ipv4 unicast
vrf context management

interface Vlan1

interface Vlan20
  no shutdown
  vrf member CUSTOMER
  ip address 172.16.20.254/24

interface Vlan1020
  no shutdown
  vrf member CUSTOMER
  ip forward

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback0
  member vni 10020
    ingress-replication protocol bgp
  member vni 101020 associate-vrf

[richbotham]

All,

so a few things happened using a combination nxos.7.0.3.I7.4.bin and eve-ng

1. The linux host could not even ping it's default gateway for some reason
2. Deleting the vlan 10 SVI and re configuring it fixed it (???)
3. A reload completely removed the 'nve' interface for some reason.
4. It would appear that this missing config fixes the Layer 3 issue for the communication between 172.16.10.0/24 and 172.16.20.0/24
   1. thanks Rick Donato at https://www.packetcoders.io/how-to-build-a-nxos-9000v-based-evpn-vxlan-fabric/ 

vrf context CUSTOMER
  vni 101020
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn