Hello everyone!
Please help to get rid of the VXLAN over internet issues.
We have many sites with Nexuses 93180YC-FX (in vPC chassis) and want to combine them into single L2 segment at least for management network.
2 of sites should be route-reflectors the others will be route-reflector-clients.
But current goal is to get VXLAN operational between 2 sites only.
Constraints and pre-requisites:
- VXLAN:
- VXLAN requires to be run loopback interface.
- We need that IPs to be visible on the internet.
- To run on vPC I have used the second any-cast loopback.
- I decided to use /128 IPv6 addresses.
- BGP EVPN:
- Both sites use the same AS number and there are configured some IPv4/IPv6 networks.
- Neighbors are located far from each other, so we need configure eBGP and multi-hop.
- I decided to use private ASN for peering for neighbors to be not considered as iBGP.
Common part of the config:
nv overlay evpn
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature vpc
feature nv overlay
vlan 3
name VXLANTEST
vn-segment 10003
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback0 anycast loopback1
member vni 10003
ingress-replication protocol bgp
router bgp <public ans>
! router-id <local ipv4>
address-family l2vpn evpn
allow-vni-in-ethertag
! neighbors configs...
evpn
vni 10003 l2
rd auto
route-target import auto
route-target export auto
We have sites #6 and #8, so neighbors config are similar except digits 6 & 8. Below is an example of site #6 config:
interface Vlan3
ip address 10.1.6.1/16
interface loopback0
ipv6 address bad:face:6::aaaa/128
interface loopback1
ipv6 address bad:face:6::ba/128
! BGP section below
neighbor bad:face:8::ba
remote-as 65008
local-as 65006
update-source loopback1
disable-connected-check
ebgp-multihop 32
address-family l2vpn evpn
allowas-in 3
send-community
send-community extended
vPC peer has the same config except interface loopback0 — ipv6 address bad:face:6::bbbb/128.
Debug commands and their outputs
# show bgp session
bad:face:8::ba 65008 0 16:14:07|00:00:48|00:00:16 E 179/20558 0/0
SUCCESS: BGP Sessions established
# show bgp l2vpn evpn vni-id 10003
Route Distinguisher: <local bgp router-id>:32770 (L2VNI 10003)
*>l[2]:[0]:[0]:[48]:[062c.2bd4.89dc]:[0]:[0.0.0.0]/216
bad:face:6::ba 100 32768 i
*>e[2]:[0]:[0]:[48]:[c27e.3bf4.23fd]:[0]:[0.0.0.0]/216
bad:face:8::ba 0 65006 65008 202656 i
*>l[3]:[0]:[128]:[bad:face:6::ba]/184
bad:face:6::ba 100 32768 i
*>e[3]:[0]:[128]:[bad:face:8::ba]/184
bad:face:8::ba 0 65006 65008 202656 i
SUCCESS: we see local and remote MAC addresses.
FAIL: We see only MAC route and no MAC-IP routes.
# show mac address-table vlan 3
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C 3 c27e.3bf4.23fd dynamic NA F F nve1(bad:face:8::ba)
+ 3 062c.2bd4.89dc dynamic NA F F vPC Peer-Link
SUCCESS: We see remote MAC addresses in the mac address-table.
# show ip arp vlan 3
Address Age MAC Address Interface Flags
10.1.6.2 00:12:07 003a.9c1c.6ae7 Vlan3
10.1.6.253 00:00:03 062c.2bd4.89dc Vlan3 +
10.1.8.2 00:00:19 INCOMPLETE Vlan3
FAIL: we don't see here any entries with remote MAC addresses.
# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- -------------------------------------- ----- --------- -------- -----------------
nve1 bad:face:8::ba Up CP 00:19:09 n/a
SUCCESS: we see raised up peer.
FAIL: Router-Mac is n/a (probably that is the reason).
I tried to specify virtual-rmac and advertise it (along with advertise-pip in BGP scope) but have no luck.
Neither switches nor hosts can't ping each other.
Can anyone please advice anything...
