cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2235
Views
5
Helpful
3
Replies

Nexus 9000v VXLAN issue

Istvan kelemen
Level 1
Level 1

Hello Dears,

 

I am playing around with the latest Nexus 9000v release 9.3(1).

The setup is the following:

  The images are deployed on  ESXI 6.7.0, 9484548.

  VM resources: 4*CPUs( E7- 4870 @ 2.40GHz ) 8GB RAM.

  Spines are CSR1000v 16.12.01a, they are the RRs for BGP scenario.

  R1 and R2 are the BGP RRs. R1 is the PIM RP in case of multicast deployment.

  OSPF is used as IGP, links are P2P. IGP and PIM peering is properly established. BGP peering is OK too.

  Transit links are .1q sub-interfaces, connected to the same vSwitch. Peer-Links and host facing links are all separated vSwitches without VLAN tagging.

 

I have tried 3 scenarios so far, vPC is always deployed. Peer-Link and Host facing links are LACP Port-Channels.

 

VXLAN with static peers (ingress replication)

  - MAC addresses are learned via Port-Channel 10 and 20 (host facing interfaces).

 

VXLAN with iBGP EVPN control-plane with Multicast

  - MAC addresses are advertised from Nexus 5-6 switches to the RRs, the RRs learn, but do not advertise the routes to the spines. Data-plane is always using the Multicast transport.

 

VXLAN with Flood & Learn (Multicast)

  - This works well.

 

image.png

 

VXLAN with static peers (ingress replication)

- Config attached: ingress_repl.rar

- Pinging from both R3 and R4. MACs only learned on the local port-channel and not from the VTEP.

- LACP, OSPF, vPC are functioning properly. VTEP to VTEP ping is working.

 

NXv9k5# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
G - 0050:56b8:d3b6 static - F F sup-eth1(R)
G - 0005:0010:0001 static - F F sup-eth1(R)
* 10 0000.0000.0003 dynamic 00:00:14 F F Po10

 

NXv9k6# ---------- Same .-------------


NXv9k7# show system internal l2fwder mac
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 20 0000.0000.0004 dynamic 00:00:05 F F Po20
G - 0007:0013:0001 static - F F sup-eth1(R)
G - 0050:56b8:1429 static - F F sup-eth1(R)

 

NXv9k8# -------- Same ---------

 


NXv9k5# ping 192.168.1.3 source 192.168.1.1
PING 192.168.1.3 (192.168.1.3) from 192.168.1.1: 56 data bytes
64 bytes from 192.168.1.3: icmp_seq=0 ttl=253 time=2.939 ms

 

NXv9k6# ping 192.168.1.3 source 192.168.1.1
PING 192.168.1.3 (192.168.1.3) from 192.168.1.1: 56 data bytes
64 bytes from 192.168.1.3: icmp_seq=0 ttl=253 time=2.49 ms

 

NXv9k5# show nve vni
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 101010 UnicastStatic Up DP L2 [10]

 

NXv9k6# ----------- Same -----------

 

NXv9k7# show nve vni
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 101010 UnicastStatic Up DP L2 [20]

 

NXv9k8# -------- Same ----------

 

 

 

 

 

iBGP EVPN

 

Data-plane is keep using the Multicast transport, because RRs do not advertise any routes.

NXv9k5# sh bgp l2 e s
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 192.168.0.1, local AS number 1
BGP table version is 27, L2VPN EVPN config peers 2, capable peers 2
1 network entries and 1 paths using 240 bytes of memory
BGP attribute entries [1/164], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.10.1 4 1 119 122 27 0 0 00:07:35 0
192.168.10.2 4 1 115 117 27 0 0 00:07:30 0
NXv9k5# sh bgp l2 e
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 27, Local Router ID is 192.168.0.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.0.1:32777 (L2VNI 101010)
*>l[2]:[0]:[0]:[48]:[0000.0000.0003]:[0]:[0.0.0.0]/216
192.168.1.1 100 32768 i

 

------------ same output on the other 3 Nexus switches, only the local MAC route is imported to BGP.

 and one of the RRs BGP table...

 

"Not advertised to any peer". --- I did not meet this issue in the other (924). In that version I had serious issues with LACP and vPC peer establishment, so I moved to 931.

 

R2#sh bgp l2 e su
BGP router identifier 192.168.10.2, local AS number 1
BGP table version is 29, main routing table version 29
4 network entries using 1376 bytes of memory
4 path entries using 832 bytes of memory
2/2 BGP path/bestpath attribute entries using 576 bytes of memory
2 BGP extended community entries using 80 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2864 total bytes of memory
BGP activity 16/12 prefixes, 16/12 paths, scan interval 60 secs
4 networks peaked at 13:24:52 Sep 3 2019 UTC (1d00h ago)

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.0.1 4 1 17 14 29 0 0 00:10:38 1
192.168.0.2 4 1 13 12 29 0 0 00:08:38 1
192.168.0.3 4 1 17 15 29 0 0 00:10:42 1
192.168.0.4 4 1 13 12 29 0 0 00:08:38 1

 

R2#sh bgp l2 e detail

Route Distinguisher: 192.168.0.1:32777
BGP routing table entry for [2][192.168.0.1:32777][0][48][000000000003][0][*]/20, version 26
Paths: (1 available, best #1, table EVPN-BGP-Table)
Flag: 0x100
Not advertised to any peer
Refresh Epoch 1
Local
192.168.1.1 (metric 2) (via default) from 192.168.0.1 (192.168.0.1)
Origin IGP, localpref 100, valid, internal, best
EVPN ESI: 00000000000000000000, Label1 101010
Extended Community: SoO:192.168.1.1:0 RT:1:101010 ENCAP:8
rx pathid: 0, tx pathid: 0x0
Updated on Sep 4 2019 13:33:55 UTC

Route Distinguisher: 192.168.0.2:32777
BGP routing table entry for [2][192.168.0.2:32777][0][48][000000000003][0][*]/20, version 27
Paths: (1 available, best #1, table EVPN-BGP-Table)
Flag: 0x100
Not advertised to any peer
Refresh Epoch 1
Local
192.168.1.1 (metric 2) (via default) from 192.168.0.2 (192.168.0.2)
Origin IGP, localpref 100, valid, internal, best
EVPN ESI: 00000000000000000000, Label1 101010
Extended Community: SoO:192.168.1.1:0 RT:1:101010 ENCAP:8
rx pathid: 0, tx pathid: 0x0
Updated on Sep 4 2019 13:33:55 UTC

Route Distinguisher: 192.168.0.3:32787
BGP routing table entry for [2][192.168.0.3:32787][0][48][000000000004][0][*]/20, version 28
Paths: (1 available, best #1, table EVPN-BGP-Table)
Flag: 0x100
Not advertised to any peer
Refresh Epoch 1
Local
192.168.1.3 (metric 2) (via default) from 192.168.0.3 (192.168.0.3)
Origin IGP, localpref 100, valid, internal, best
EVPN ESI: 00000000000000000000, Label1 101010
Extended Community: SoO:192.168.1.3:0 RT:1:101010 ENCAP:8
rx pathid: 0, tx pathid: 0x0
Updated on Sep 4 2019 13:33:55 UTC

Route Distinguisher: 192.168.0.4:32787
BGP routing table entry for [2][192.168.0.4:32787][0][48][000000000004][0][*]/20, version 29
Paths: (1 available, best #1, table EVPN-BGP-Table)
Flag: 0x100
Not advertised to any peer
Refresh Epoch 1
Local
192.168.1.3 (metric 2) (via default) from 192.168.0.4 (192.168.0.4)
Origin IGP, localpref 100, valid, internal, best
EVPN ESI: 00000000000000000000, Label1 101010
Extended Community: SoO:192.168.1.3:0 RT:1:101010 ENCAP:8
rx pathid: 0, tx pathid: 0x0
Updated on Sep 4 2019 13:33:55 UTC

 

Thanks for your help in advance!

1 Accepted Solution

Accepted Solutions

Istvan kelemen
Level 1
Level 1

I have found the answers to my issues.

 

Rule1: Always use a dedicated VMware vSwitch which allows all VLANs (4095) -> Peer-link can pass VLANs, Vlanif backup works with automatic VXLAN encapsulation. No extra command is needed, just enable PIM and IGP.

 

Rule2: VTEPs have to run PIM.

 

I managed to build ad advanced scenario. VXLAN L2&L3 GW with BGP EVPN + Flood&Learn for BUM.

The current topology:

vxlan L3.jpg

View solution in original post

3 Replies 3

Hi @Istvan kelemen,

What is the result of replacing the CSR1000v with Nexus 9000v as Spines/BGP Route Reflectors while maintaining the same configuration on the Leaf Switches?

Regards.

CRS1000v works, but currently, I am using XRv9000 as spines to be able to filter MAC address routes.

I think I had duplicated OSPF RID due to VTEPs or mismatching OSPF link subnets.

Istvan kelemen
Level 1
Level 1

I have found the answers to my issues.

 

Rule1: Always use a dedicated VMware vSwitch which allows all VLANs (4095) -> Peer-link can pass VLANs, Vlanif backup works with automatic VXLAN encapsulation. No extra command is needed, just enable PIM and IGP.

 

Rule2: VTEPs have to run PIM.

 

I managed to build ad advanced scenario. VXLAN L2&L3 GW with BGP EVPN + Flood&Learn for BUM.

The current topology:

vxlan L3.jpg

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: