cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

537
Views
0
Helpful
4
Replies
louisng
Beginner

VXLAN L2VNI no connectivity on nexus 9300-EX

Hi guys,

 

I'm going to configure VXLAN on nexus N9K-C93108TC-EX but L2VNI no connectivity.

 

Network topology diagram:

https://prnt.sc/2346bcn

 

 

At the all I used VLAN with EVPN

two leaf switch can learn Mac addresses from each other through EVPN but L2VNI no connectivity.

 

The same configuration works perfectly on nx-osv So I believe that the configuration is correct, but it will not work if it is moved to the hardware switch. there have some policies on the hardware blocking vxlan UDP traffic?

 

I used Ethanalyzer and I can see the VXLAN UDP traffic send to Leaf2 switch, but using Ethanalyzer on another Leaf 2 switch I can't see VXLAN traffic incoming.

so Both Leaf are sending out VXLAN traffic, but neither has received any VLXAN traffic.

 

There is no problem with MTU between Leaf1 and Leaf2

ping 192.168.1.202 source 192.168.1.201 df-bit packet-size 8000
ping 192.168.1.202 source 192.168.1.201 df-bit packet-size 8000
PING 192.168.1.202 (192.168.1.202) from 192.168.1.201: 8000 data bytes
8008 bytes from 192.168.1.202: icmp_seq=0 ttl=253 time=1.326 ms
8008 bytes from 192.168.1.202: icmp_seq=1 ttl=253 time=1.107 ms
8008 bytes from 192.168.1.202: icmp_seq=2 ttl=253 time=1.131 ms

 

 

Configuration
==================================================================================
Leaf1:
vlan 1001
vn-segment 10001
vlan 3000
vn-segment 8000

interface Vlan1001
no shutdown
vrf member vxlan-user1
ip address 10.0.0.1/24
fabric forwarding mode anycast-gateway

interface Vlan3000
no shutdown
vrf member vxlan-user1
no ip redirects
ip forward
ipv6 address use-link-local-only
no ipv6 redirects

interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 8000 associate-vrf
member vni 10001
suppress-arp
ingress-replication protocol bgp

==================================================================================
Leaf2:
vlan 1001
vn-segment 10001
vlan 3000
vn-segment 800

interface Vlan1001
no shutdown
vrf member vxlan-user1
ip address 10.0.0.1/24
fabric forwarding mode anycast-gateway

interface Vlan3000
no shutdown
vrf member vxlan-user1
no ip redirects
ip forward
ipv6 address use-link-local-only
no ipv6 redirects

interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 8000 associate-vrf
member vni 10001
suppress-arp
ingress-replication protocol bgp
==================================================================================

Leaf1(config)# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication

Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 8000 n/a Up CP L3 [vxlan-user1]
nve1 10001 UnicastBGP Up CP L2 [1001]

 

Leaf2(config)# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Xconn - Crossconnect
MS-IR - Multisite Ingress Replication

Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 8000 n/a Up CP L3 [vxlan-user1]
nve1 10001 UnicastBGP Up CP L2 [1001]

==================================================================================
Leaf1(config)# show nve peers

Interface Peer-IP State LearnType Uptime Route
r-Mac
--------- -------------------------------------- ----- --------- -------- -----
------------
nve1 192.168.1.202 Up CP 00:02:40 0027.
e3d1.e3c1

Leaf2(config)# show nve peers
Interface Peer-IP State LearnType Uptime Route
r-Mac
--------- -------------------------------------- ----- --------- -------- -----
------------
nve1 192.168.1.201 Up CP 00:02:41 00a3.
8e6d.5981
==================================================================================
Leaf1(config)# show l2route mac-ip all

Topology Mac Address Host IP Prod Flags
Seq No Next-Hops
----------- -------------- --------------------------------------- ------ ------
---- ---------- ---------------------------------------
1001 ecf4.bbe2.5bf8 10.0.0.2 HMM L,
0 Local
1001 246e.966a.c110 10.0.0.3 HMM L,
0 Local
1001 246e.966a.a878 10.0.0.4 BGP --
0 192.168.1.202 (Label: 10001)
1001 ecf4.bbdb.29e8 10.0.0.5 BGP --
0 192.168.1.202 (Label: 10001)


==================================================================================
Leaf2(config)# show l2route mac-ip all

Topology Mac Address Host IP Prod Flags
Seq No Next-Hops
----------- -------------- --------------------------------------- ------ ------
---- ---------- ---------------------------------------
1001 ecf4.bbe2.5bf8 10.0.0.2 BGP --
0 192.168.1.201 (Label: 10001)
1001 246e.966a.c110 10.0.0.3 BGP --
0 192.168.1.201 (Label: 10001)
1001 246e.966a.a878 10.0.0.4 HMM L,
0 Local
1001 ecf4.bbdb.29e8 10.0.0.5 HMM L,
0 Local


=========================TEST=========================================================
Leaf1(config)# ping 10.0.0.2 vrf vxlan-user1
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=63 time=0.828 ms
64 bytes from 10.0.0.2: icmp_seq=1 ttl=63 time=0.538 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=63 time=0.454 ms

Leaf1(config)# ping 10.0.0.4 vrf vxlan-user1
PING 10.0.0.4 (10.0.0.4): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out

--- 10.0.0.4 ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss


Leaf2(config)# ping 10.0.0.4 vrf vxlan-user1
PING 10.0.0.4 (10.0.0.4): 56 data bytes
64 bytes from 10.0.0.4: icmp_seq=0 ttl=63 time=0.758 ms
64 bytes from 10.0.0.4: icmp_seq=1 ttl=63 time=0.534 ms
64 bytes from 10.0.0.4: icmp_seq=2 ttl=63 time=0.481 ms
64 bytes from 10.0.0.4: icmp_seq=3 ttl=63 time=0.46 ms
64 bytes from 10.0.0.4: icmp_seq=4 ttl=63 time=0.474 ms
^C
--- 10.0.0.4 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.46/0.541/0.758 ms
Leaf2(config)# ping 10.0.0.2 vrf vxlan-user1
PING 10.0.0.2 (10.0.0.2): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out

 

Thanks

 

4 REPLIES 4
Pedroxh
Beginner

Good Morning

Do you have the LEAF2 outputs from the show bgp l2vpn evpn? Is the RT/RD from LEAF2 to send traffic to LEAF1 correct?

Maybe LEAF2 doesn't know who to forward the traffic to. Check that the MAC of the host that is on LEAF1 is arriving in the EVPN table of LEAF2. HOST1 pings HOST2 normally?

 

 

Hi Pedroxh,

 

the MAC of the host that is on LEAF1 is arriving in the EVPN table of LEAF2.
Yes LEAF learned host's mac addresses from each other.

HOST1 pings HOST2 normally?
No Connectivity

 

LEAF1 lo0 192.168.1.80
LEAF2 lo0 192.168.1.81

LEAF1 lo1 192.168.1.201
LEAF2 lo1 192.168.1.202

I have tried using lo0 as source-interface for interface nve but it doesn't work too
LEAF1#lo0 > LEAF2#lo0 worked
LEAF1#lo1 > LEAF2#lo1 worked
LEAF1#lo0 > LEAF2#lo2 worked


ping 192.168.1.81 source 192.168.1.80 df-bit packet-size 8000
ping 192.168.1.81 source 192.168.1.80 df-bit packet-size 8000
PING 192.168.1.81 (192.168.1.81) from 192.168.1.80: 8000 data bytes
8008 bytes from 192.168.1.81: icmp_seq=0 ttl=253 time=1.292 ms
8008 bytes from 192.168.1.81: icmp_seq=1 ttl=253 time=1.125 ms
8008 bytes from 192.168.1.81: icmp_seq=2 ttl=253 time=1.116 ms
8008 bytes from 192.168.1.81: icmp_seq=3 ttl=253 time=1.045 ms

 

=================================================================================
LEAF1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 476, Local Router ID is 192.168.1.80
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.1.80:33768 (L2VNI 10001)
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[0]:[0.0.0.0]/216
192.168.1.202 100 0 i
*>l[2]:[0]:[0]:[48]:[246e.966a.c110]:[0]:[0.0.0.0]/216
192.168.1.201 100 32768 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[0]:[0.0.0.0]/216
192.168.1.202 100 0 i
*>l[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[0]:[0.0.0.0]/216
192.168.1.201 100 32768 i
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[32]:[10.0.0.4]/272
192.168.1.202 100 0 i
*>l[2]:[0]:[0]:[48]:[246e.966a.c110]:[32]:[10.0.0.3]/272
192.168.1.201 100 32768 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[32]:[10.0.0.5]/272
192.168.1.202 100 0 i
*>l[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[32]:[10.0.0.2]/272
192.168.1.201 100 32768 i
*>l[3]:[0]:[32]:[192.168.1.201]/88
192.168.1.201 100 32768 i
*>i[3]:[0]:[32]:[192.168.1.202]/88
192.168.1.202 100 0 i

Route Distinguisher: 192.168.1.81:3
*>i[5]:[0]:[0]:[24]:[10.0.0.0]/224
192.168.1.202 0 100 0 ?
* i 192.168.1.202 0 100 0 ?

Route Distinguisher: 192.168.1.81:33768
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[0]:[0.0.0.0]/216
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[0]:[0.0.0.0]/216
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[32]:[10.0.0.4]/272
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[32]:[10.0.0.5]/272
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i
*>i[3]:[0]:[32]:[192.168.1.202]/88
192.168.1.202 100 0 i
* i 192.168.1.202 100 0 i

Route Distinguisher: 192.168.1.80:3 (L3VNI 8000)
*>i[2]:[0]:[0]:[48]:[246e.966a.a878]:[32]:[10.0.0.4]/272
192.168.1.202 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[32]:[10.0.0.5]/272
192.168.1.202 100 0 i
* i[5]:[0]:[0]:[24]:[10.0.0.0]/224
192.168.1.202 0 100 0 ?
*>l 192.168.1.201 0 100 32768 ?

=================================================================================
LEAF2# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 434, Local Router ID is 192.168.1.81
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.1.80:3
*>i[5]:[0]:[0]:[24]:[10.0.0.0]/224
192.168.1.201 0 100 0 ?
* i 192.168.1.201 0 100 0 ?

Route Distinguisher: 192.168.1.80:33768
* i[2]:[0]:[0]:[48]:[246e.966a.c110]:[0]:[0.0.0.0]/216
192.168.1.201 100 0 i
*>i 192.168.1.201 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[0]:[0.0.0.0]/216
192.168.1.201 100 0 i
* i 192.168.1.201 100 0 i
* i[2]:[0]:[0]:[48]:[246e.966a.c110]:[32]:[10.0.0.3]/272
192.168.1.201 100 0 i
*>i 192.168.1.201 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[32]:[10.0.0.2]/272
192.168.1.201 100 0 i
* i 192.168.1.201 100 0 i
*>i[3]:[0]:[32]:[192.168.1.201]/88
192.168.1.201 100 0 i
* i 192.168.1.201 100 0 i

Route Distinguisher: 192.168.1.81:33768 (L2VNI 10001)
*>l[2]:[0]:[0]:[48]:[246e.966a.a878]:[0]:[0.0.0.0]/216
192.168.1.202 100 32768 i
*>i[2]:[0]:[0]:[48]:[246e.966a.c110]:[0]:[0.0.0.0]/216
192.168.1.201 100 0 i
*>l[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[0]:[0.0.0.0]/216
192.168.1.202 100 32768 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[0]:[0.0.0.0]/216
192.168.1.201 100 0 i
*>l[2]:[0]:[0]:[48]:[246e.966a.a878]:[32]:[10.0.0.4]/272
192.168.1.202 100 32768 i
*>i[2]:[0]:[0]:[48]:[246e.966a.c110]:[32]:[10.0.0.3]/272
192.168.1.201 100 0 i
*>l[2]:[0]:[0]:[48]:[ecf4.bbdb.29e8]:[32]:[10.0.0.5]/272
192.168.1.202 100 32768 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[32]:[10.0.0.2]/272
192.168.1.201 100 0 i
*>i[3]:[0]:[32]:[192.168.1.201]/88
192.168.1.201 100 0 i
*>l[3]:[0]:[32]:[192.168.1.202]/88
192.168.1.202 100 32768 i

Route Distinguisher: 192.168.1.81:3 (L3VNI 8000)
*>i[2]:[0]:[0]:[48]:[246e.966a.c110]:[32]:[10.0.0.3]/272
192.168.1.201 100 0 i
*>i[2]:[0]:[0]:[48]:[ecf4.bbe2.5bf8]:[32]:[10.0.0.2]/272
192.168.1.201 100 0 i
* i[5]:[0]:[0]:[24]:[10.0.0.0]/224
192.168.1.201 0 100 0 ?
*>l 192.168.1.202 0 100 32768 ?

 

Thank you

Dawei
Cisco Employee

Did you test ping from Host1 to Host2?

In anycast gw mode, all leafs use the same ip address, so the icmp reply will terminate by local leaf. 

Did you test ping from Host1 to Host2?

I did but no connectivity too.

After a long in troubleshooting, we found that it was ESXI vSwitch problem, After the disable/enable the port on leaf  it was normal now.

not sure what's the problem inside ESXI vSwitch, I think uplink failover some thing