Local VLAN in leaf switch with VXLAN EVPN

[Christer Lind]

Hello!

I have bought a few Cisco C9500 (C9500-24Y4C) that I want to be used as leafs in a VXLAN EVPN setup (Multi vendor, different spines) 

I'm trying to get the local VLANs in the C9500 to get encapsulated and sent across the tunnels to the nve peers. But I can't seem to get it to work. Does anyone have any configuration examples how one goes about for this to work?

[MHM Cisco World]

in which step you stop ?
can you share the config 

[Christer Lind]

I've included the relevant part of the configuration. There's also OSPF running for the loopback addresses.

So what I *want* is the local VLAN95 (In the example) to be encapsulated and sent across L3 links, or rather, populate the BGP tables.

vlan configuration 95
member evpn-instance 95 vni 10095
!
interface Vlan95
ip address 192.168.95.1 255.255.255.0
no autostate
end
!
l2vpn evpn
logging peer state
replication-type ingress
router-id Loopback0
!
l2vpn evpn instance 95 vlan-based
encapsulation vxlan
route-target export 65333:95
route-target import 65333:95
!
router bgp 65333
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 10.111.0.7 remote-as 65333
neighbor 10.111.0.7 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family l2vpn evpn
neighbor 10.111.0.7 activate
neighbor 10.111.0.7 send-community both
exit-address-family
!

[sidshas03]

interface nve1
host-reachability protocol bgp
source-interface loopback0
member vni 10095
ingress-replication protocol bgp
suppress-arp
suppress-arp
no shutdown

I hope this helps you in troubleshooting the issue. If you have any further questions, feel free to ask!

[rhallinan7387]

It looks like you're missing the configuration of the NVE interface - which ties everything you have above together. For example:

int nve 1
  no ip address
  source-interface LoopbackX  (suggest to make this a loopback advertised in underlay but different from loopback for mgmt)
  host-reachability protocol bgp
  member vni 10095 (and then 'ingress replication' or either 'mcast-group A.B.C.D' depending on how you are handling BUM traffic)

You'd have a 'member vni ...' line per VLAN you want to tie to VXLAN across the L3 boundary.

Please mark helpful replies.

[Christer Lind]

Hello,

I forgot to include the configuration for the nve1 interface, but it is present, and follows
interface nve1
no ip address
source-interface Loopback0
host-reachability protocol bgp
member vni 10095 ingress-replication
end

[rhallinan7387]

Some other things to check - make sure you have jumbo MTU across your underlay. Also - do you see any NVE peers (sh nve peers)? Can you check the BGP table and make sure you're getting information from the other VTEP: sh bgp l2vpn evpn
Are there any actual host addresses live on either side (make sure the SVI is up/up with host addresses that would be shared)?

[Christer Lind]

Hello, thank you for your reply.

MTU is set to 9216. IP MTU is default at 1500.

Peers are up, and active;
Interface VNI Type Peer-IP RMAC/Num_RTs eVNI state flags UP time
nve1 10095 L2CP 10.111.0.4 1 10095 UP N/A 09:10:01
nve1 10095 L2CP 10.111.0.6 1 10095 UP N/A 09:42:16

show bgp l2vpn evpn
BGP table version is 75, local router ID is 10.111.0.5
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.111.0.4:95
*>i [3][10.111.0.4:95][0][32][10.111.0.4]/17
10.111.0.4 0 100 0 ?
Route Distinguisher: 10.111.0.5:95
*>i [3][10.111.0.5:95][0][32][10.111.0.4]/17
10.111.0.4 0 100 0 ?
*> [3][10.111.0.5:95][0][32][10.111.0.5]/17
:: 32768 ?
*>i [3][10.111.0.5:95][0][32][10.111.0.6]/17
10.111.0.6 0 100 0 ?
Route Distinguisher: 10.111.0.6:95
*>i [3][10.111.0.6:95][0][32][10.111.0.6]/17
10.111.0.6 0 100 0 ?

Right now, there are no hosts connected to the switches.
But I've had hosts connected, and verified connectivity, and the host can ping every switch.
However, the switches can't ping eachother (Across the tunnel). SW1 trying to ping SW2 won't even try to egress the package over the tunnel. So im wondering what im missing.

[rhallinan7387]

Thanks for the additional clarification - that actually makes more sense now since you're configs looked good once you had posted the NVE interface. So to make sure I'm understanding - a summary:

• When hosts are connected on either VTEP - then can ping across to hosts on the other VTEP
• Those hosts can also ping the local VTEP SVI as well as the remote VTEP SVI
• The issue is that from VTEP to VTEP - you can't ping the destination SVI from the source SVI?

[Christer Lind]

That's a great summary and describes my problem exactly. My apologies for being a bit unclear at the beginning.

[rhallinan7387]

No problem. What is the intent of having different SVI IPs on the two different VTEPs? You're not doing a distribute anycast gateway - are you instead going with a centralized default gateway (see https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-3/configuration_guide/vxlan/b_173_bgp_evpn_vxlan_9500_cg/configuring_evpn_vxlan_integrated_routing_and_bridging.html)? I'm wondering if you're coming into one of the scenarios under the section 'Default Gateway MAC Address Assignment' - specifically the third (If the leaf switch has an SVI with no matching IP address, it invalidates the MAC-IP route and issues an error.) The host ping may work because it becomes an unknown unicast. 

[Christer Lind]

To be fair, I don't really need VTEP-to-VTEP communication. It wont be used in any gateway application at all. Just bridging.


It all started with me wanting to use a local vlan just to verify connectivity and to see that it was working in my lab.

But then I got invested in trying to figure out why I couldn't get the CPU to encapsulate the package and send it across the wire and wether I missed something obvious, or not. I know other vendors has to pull of some tricks to do it. It might just not be possible to encap and send from the control-plane.