12-15-2021 05:36 AM - edited 12-15-2021 05:56 AM
Hi guys,
I'm going to configure VXLAN on nexus N9K-C93108TC-EX but L2VNI no connectivity.
Network topology diagram:
At the all I used VLAN with EVPN
two leaf switch can learn Mac addresses from each other through EVPN but L2VNI no connectivity.
The same configuration works perfectly on nx-osv So I believe that the configuration is correct, but it will not work if it is moved to the hardware switch. there have some policies on the hardware blocking vxlan UDP traffic?
I used Ethanalyzer and I can see the VXLAN UDP traffic send to Leaf2 switch, but using Ethanalyzer on another Leaf 2 switch I can't see VXLAN traffic incoming.
so Both Leaf are sending out VXLAN traffic, but neither has received any VLXAN traffic.
There is no problem with MTU between Leaf1 and Leaf2
ping 192.168.1.202 source 192.168.1.201 df-bit packet-size 8000
ping 192.168.1.202 source 192.168.1.201 df-bit packet-size 8000
PING 192.168.1.202 (192.168.1.202) from 192.168.1.201: 8000 data bytes
8008 bytes from 192.168.1.202: icmp_seq=0 ttl=253 time=1.326 ms
8008 bytes from 192.168.1.202: icmp_seq=1 ttl=253 time=1.107 ms
8008 bytes from 192.168.1.202: icmp_seq=2 ttl=253 time=1.131 ms
Configuration
==================================================================================
Leaf1:
vlan 1001
vn-segment 10001
vlan 3000
vn-segment 8000
interface Vlan1001
no shutdown
vrf member vxlan-user1
ip address 10.0.0.1/24
fabric forwarding mode anycast-gateway
interface Vlan3000
no shutdown
vrf member vxlan-user1
no ip redirects
ip forward
ipv6 address use-link-local-only
no ipv6 redirects
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 8000 associate-vrf
member vni 10001
suppress-arp
ingress-replication protocol bgp
==================================================================================
Leaf2:
vlan 1001
vn-segment 10001
vlan 3000
vn-segment 800
interface Vlan1001
no shutdown
vrf member vxlan-user1
ip address 10.0.0.1/24
fabric forwarding mode anycast-gateway
interface Vlan3000
no shutdown
vrf member vxlan-user1
no ip redirects
ip forward
ipv6 address use-link-local-only
no ipv6 redirects
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 8000 associate-vrf
member vni 10001
suppress-arp
ingress-replication protocol bgp
==================================================================================
Leaf1(config)# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 8000 n/a Up CP L3 [vxlan-user1]
nve1 10001 UnicastBGP Up CP L2 [1001]
Leaf2(config)# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 8000 n/a Up CP L3 [vxlan-user1]
nve1 10001 UnicastBGP Up CP L2 [1001]
==================================================================================
Leaf1(config)# show nve peers
Interface Peer-IP State LearnType Uptime Route
r-Mac
--------- -------------------------------------- ----- --------- -------- -----
------------
nve1 192.168.1.202 Up CP 00:02:40 0027.
e3d1.e3c1
Leaf2(config)# show nve peers
Interface Peer-IP State LearnType Uptime Route
r-Mac
--------- -------------------------------------- ----- --------- -------- -----
------------
nve1 192.168.1.201 Up CP 00:02:41 00a3.
8e6d.5981
==================================================================================
Leaf1(config)# show l2route mac-ip all
Topology Mac Address Host IP Prod Flags
Seq No Next-Hops
----------- -------------- --------------------------------------- ------ ------
---- ---------- ---------------------------------------
1001 ecf4.bbe2.5bf8 10.0.0.2 HMM L,
0 Local
1001 246e.966a.c110 10.0.0.3 HMM L,
0 Local
1001 246e.966a.a878 10.0.0.4 BGP --
0 192.168.1.202 (Label: 10001)
1001 ecf4.bbdb.29e8 10.0.0.5 BGP --
0 192.168.1.202 (Label: 10001)
==================================================================================
Leaf2(config)# show l2route mac-ip all
Topology Mac Address Host IP Prod Flags
Seq No Next-Hops
----------- -------------- --------------------------------------- ------ ------
---- ---------- ---------------------------------------
1001 ecf4.bbe2.5bf8 10.0.0.2 BGP --
0 192.168.1.201 (Label: 10001)
1001 246e.966a.c110 10.0.0.3 BGP --
0 192.168.1.201 (Label: 10001)
1001 246e.966a.a878 10.0.0.4 HMM L,
0 Local
1001 ecf4.bbdb.29e8 10.0.0.5 HMM L,
0 Local
=========================TEST=========================================================
Leaf1(config)# ping 10.0.0.2 vrf vxlan-user1
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=63 time=0.828 ms
64 bytes from 10.0.0.2: icmp_seq=1 ttl=63 time=0.538 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=63 time=0.454 ms
Leaf1(config)# ping 10.0.0.4 vrf vxlan-user1
PING 10.0.0.4 (10.0.0.4): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out
--- 10.0.0.4 ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss
Leaf2(config)# ping 10.0.0.4 vrf vxlan-user1
PING 10.0.0.4 (10.0.0.4): 56 data bytes
64 bytes from 10.0.0.4: icmp_seq=0 ttl=63 time=0.758 ms
64 bytes from 10.0.0.4: icmp_seq=1 ttl=63 time=0.534 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=63 time=0.481 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=63 time=0.46 ms
64 bytes from 10.0.0.4: icmp_seq=4 ttl=63 time=0.474 ms
^C
--- 10.0.0.4 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.46/0.541/0.758 ms
Leaf2(config)# ping 10.0.0.2 vrf vxlan-user1
PING 10.0.0.2 (10.0.0.2): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out
Thanks
12-15-2021 05:57 AM
Good Morning
Do you have the LEAF2 outputs from the show bgp l2vpn evpn? Is the RT/RD from LEAF2 to send traffic to LEAF1 correct?
Maybe LEAF2 doesn't know who to forward the traffic to. Check that the MAC of the host that is on LEAF1 is arriving in the EVPN table of LEAF2. HOST1 pings HOST2 normally?
12-15-2021 09:26 PM - edited 12-15-2021 09:28 PM
Hi Pedroxh,
the MAC of the host that is on LEAF1 is arriving in the EVPN table of LEAF2.
Yes LEAF learned host's mac addresses from each other.
HOST1 pings HOST2 normally?
No Connectivity
LEAF1 lo0 192.168.1.80
LEAF2 lo0 192.168.1.81
LEAF1 lo1 192.168.1.201
LEAF2 lo1 192.168.1.202
I have tried using lo0 as source-interface for interface nve but it doesn't work too
LEAF1#lo0 > LEAF2#lo0 worked
LEAF1#lo1 > LEAF2#lo1 worked
LEAF1#lo0 > LEAF2#lo2 worked
ping 192.168.1.81 source 192.168.1.80 df-bit packet-size 8000
ping 192.168.1.81 source 192.168.1.80 df-bit packet-size 8000
PING 192.168.1.81 (192.168.1.81) from 192.168.1.80: 8000 data bytes
8008 bytes from 192.168.1.81: icmp_seq=0 ttl=253 time=1.292 ms
8008 bytes from 192.168.1.81: icmp_seq=1 ttl=253 time=1.125 ms
8008 bytes from 192.168.1.81: icmp_seq=2 ttl=253 time=1.116 ms
8008 bytes from 192.168.1.81: icmp_seq=3 ttl=253 time=1.045 ms
=================================================================================
LEAF1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 476, Local Router ID is 192.168.1.80
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.1.80:33768 (L2VNI 10001)
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[0]:[0.0.0.0]/216
192.168.1.202 100 0 i
*>l[2]:[0]:[0]:[48]:[246e.966a.c110]:[0]:[0.0.0.0]/216
192.168.1.201 100 32768 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[0]:[0.0.0.0]/216
192.168.1.202 100 0 i
*>l[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[0]:[0.0.0.0]/216
192.168.1.201 100 32768 i
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[32]:[10.0.0.4]/272
192.168.1.202 100 0 i
*>l[2]:[0]:[0]:[48]:[246e.966a.c110]:[32]:[10.0.0.3]/272
192.168.1.201 100 32768 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[32]:[10.0.0.5]/272
192.168.1.202 100 0 i
*>l[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[32]:[10.0.0.2]/272
192.168.1.201 100 32768 i
*>l[3]:[0]:[32]:[192.168.1.201]/88
192.168.1.201 100 32768 i
*>i[3]:[0]:[32]:[192.168.1.202]/88
192.168.1.202 100 0 i
Route Distinguisher: 192.168.1.81:3
*>i[5]:[0]:[0]:[24]:[10.0.0.0]/224
192.168.1.202 0 100 0 ?
* i 192.168.1.202 0 100 0 ?
Route Distinguisher: 192.168.1.81:33768
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[0]:[0.0.0.0]/216
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[0]:[0.0.0.0]/216
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[32]:[10.0.0.4]/272
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[32]:[10.0.0.5]/272
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
*>i[3]:[0]:[32]:[192.168.1.202]/88
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
Route Distinguisher: 192.168.1.80:3 (L3VNI 8000)
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[32]:[10.0.0.4]/272
192.168.1.202 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[32]:[10.0.0.5]/272
192.168.1.202 100 0 i
* i[5]:[0]:[0]:[24]:[10.0.0.0]/224
192.168.1.202 0 100 0 ?
*>l 192.168.1.201 0 100 32768 ?
=================================================================================
LEAF2# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 434, Local Router ID is 192.168.1.81
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.1.80:3
*>i[5]:[0]:[0]:[24]:[10.0.0.0]/224
192.168.1.201 0 100 0 ?
* i 192.168.1.201 0 100 0 ?
Route Distinguisher: 192.168.1.80:33768
* i[2]:[0]:[0]:[48]:[246e.966a.c110]:[0]:[0.0.0.0]/216
192.168.1.201 100 0 i
*>i 192.168.1.201 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[0]:[0.0.0.0]/216
192.168.1.201 100 0 i
* i 192.168.1.201 100 0 i
* i[2]:[0]:[0]:[48]:[246e.966a.c110]:[32]:[10.0.0.3]/272
192.168.1.201 100 0 i
*>i 192.168.1.201 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[32]:[10.0.0.2]/272
192.168.1.201 100 0 i
* i 192.168.1.201 100 0 i
*>i[3]:[0]:[32]:[192.168.1.201]/88
192.168.1.201 100 0 i
* i 192.168.1.201 100 0 i
Route Distinguisher: 192.168.1.81:33768 (L2VNI 10001)
*>l[2]:[0]:[0]:[48]:[246e.966a.a878]:[0]:[0.0.0.0]/216
192.168.1.202 100 32768 i
*>i[2]:[0]:[0]:[48]:[246e.966a.c110]:[0]:[0.0.0.0]/216
192.168.1.201 100 0 i
*>l[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[0]:[0.0.0.0]/216
192.168.1.202 100 32768 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[0]:[0.0.0.0]/216
192.168.1.201 100 0 i
*>l[2]:[0]:[0]:[48]:[246e.966a.a878]:[32]:[10.0.0.4]/272
192.168.1.202 100 32768 i
*>i[2]:[0]:[0]:[48]:[246e.966a.c110]:[32]:[10.0.0.3]/272
192.168.1.201 100 0 i
*>l[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[32]:[10.0.0.5]/272
192.168.1.202 100 32768 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[32]:[10.0.0.2]/272
192.168.1.201 100 0 i
*>i[3]:[0]:[32]:[192.168.1.201]/88
192.168.1.201 100 0 i
*>l[3]:[0]:[32]:[192.168.1.202]/88
192.168.1.202 100 32768 i
Route Distinguisher: 192.168.1.81:3 (L3VNI 8000)
*>i[2]:[0]:[0]:[48]:[246e.966a.c110]:[32]:[10.0.0.3]/272
192.168.1.201 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[32]:[10.0.0.2]/272
192.168.1.201 100 0 i
* i[5]:[0]:[0]:[24]:[10.0.0.0]/224
192.168.1.201 0 100 0 ?
*>l 192.168.1.202 0 100 32768 ?
Thank you
12-15-2021 09:47 PM
Did you test ping from Host1 to Host2?
In anycast gw mode, all leafs use the same ip address, so the icmp reply will terminate by local leaf.
12-17-2021 01:54 AM
Did you test ping from Host1 to Host2?
I did but no connectivity too.
After a long in troubleshooting, we found that it was ESXI vSwitch problem, After the disable/enable the port on leaf it was normal now.
not sure what's the problem inside ESXI vSwitch, I think uplink failover some thing
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide