EVPN VXLAN NVE Peers only showing up on one side and are inconsistent

[shannonr]

I am setting up EVPN-VXLAN as an overlay to provide L2 reachability.

I have 2 spines and 4 leaves. Spines are Cat 9500-16x and Leaves are a mix of Cat 9500-16x and Cat 9500-48y4c (2 of each).

Looking at the NVE peers on all the switches:

• Spine1 peers with Leaf 1-2
• Spine 2 peers with Leaf 1-2
• Leaf 1 peers with Spine 1,2, Leaf 2
• Leaf 2 peers with Spine 1,2, Leaf 1.
• Leaf 3 thinks it peers with Spine 1-2, and Leaf 1-2
• Leaf 4 thinks it peers with Spine1-2 and Leaf 1-2.

I'm not sure why leaves 1-2 peer with each other. And I'm not sure why leaves 3-4 think they are peered with everything (except with each other), when the other peer doesn't recognise the peering.

I also receive these errors in the logs on the 9500-16x switches:
*Jul 9 01:42:31.759: NVE-MGR-EI: L2FIB rvtep 100102:UNKNOWN cfg type 2 for BD 102
*Jul 9 01:42:31.759: NVE-MGR-EI ERROR: Invalid peer address for bd 102

These errors are definitely related as, as, if I disable the NVE interfaces on the 9500-48 switches (leaves 3-4) the errors stop.

Config is identical on all leaves.

• Spine1 = 10.254.1.1 (NVE IP, source interface loopback 1)
• Spine2 = 10.254.1.2
• Leaf 1 (9500-16) = 10.254.1.3
• Leaf 2 (9500-16) = 10.254.1.4
• Leaf 3 (9500-48) = 10.254.1.5
• Leaf 4 (9500-48) = 10.254.1.6

Output from Leaf 5 showing it has a peering to leaf 2:

RND-9500-48X_VTEP(config-if)#do sh nve peers | i 10.254.1.4
nve1 100100 L2CP 10.254.1.4 4 100100 UP N/A 00:03:49

Output from Leaf 2 showing no peering to leaf 5:

S4NR-9500-16X_VTEP#sh nve peers | i 10.254.1.5
S4NR-9500-16X_VTEP#

S4NR-9500-16X_VTEP#sh nve peers | i 10.254.1.
nve1 100100 L2CP 10.254.1.1 4 100100 UP N/A 23:49:47
nve1 100100 L2CP 10.254.1.2 4 100100 UP N/A 23:49:47
nve1 100100 L2CP 10.254.1.3 4 100100 UP N/A 23:49:47

Note: the loopbacks used by the NVE interfaces can ping each other so the underlay routing is working fine (OSPF).

NVE interface config (identical on all switches):

interface nve1
no ip address
source-interface Loopback1
host-reachability protocol bgp
end

What is going on?

I've not sure how to troubleshoot further.

Thanks!

[shannonr]

To test the different VLAN/EVIs the "host" is actually a L2 switch with multiple VLANs. The switch has SVI's for VLAN 100,101,102,103 (IP addresses 192.168.100.103, 192.168.101.103, 192.168.102.103, 192.168.103.103).

The concept is the same as the real-life scenario where access switches are connected to the leaf switches and each access switch has hosts connected in different VLANs. 

LEAF2-9500-16X_VTEP#show l2route evpn mac ip topology 10100
EVI ETag Prod Mac Address Host IP Next Hop(s)
----- ---------- ----- -------------- --------------------------------------- --------------------------------------------------
10100 0 BGP f839.182a.0d51 192.168.100.103 V:100100 10.254.1.3

LEAF2-9500-16X_VTEP#show l2route evpn mac ip topology 10101
EVI ETag Prod Mac Address Host IP Next Hop(s)
----- ---------- ----- -------------- --------------------------------------- --------------------------------------------------
10101 0 BGP f839.182a.0d41 192.168.101.103 V:100101 10.254.1.3

LEAF2-9500-16X_VTEP#show l2route evpn mac ip topology 10102
EVI ETag Prod Mac Address Host IP Next Hop(s)
----- ---------- ----- -------------- --------------------------------------- --------------------------------------------------
10102 0 BGP f839.182a.0d4d 192.168.102.103 V:100102 10.254.1.3

LEAF2-9500-16X_VTEP#show l2route evpn mac ip topology 10103
EVI ETag Prod Mac Address Host IP Next Hop(s)
----- ---------- ----- -------------- --------------------------------------- --------------------------------------------------
10103 0 BGP f839.182a.0d5d 192.168.103.103 V:100103 10.254.1.3

[MHM Cisco World]

 *>   [2][10.254.1.3:42867][0][48][F839182A0D0D][0][*]/20 <<- this MAC is appear in all EVI ??

please try connect one host to one VLAN in one leaf 
then add other host in other leaf in same VLAN 

other output is all correct 
there are four MAC one for each EVI and it advertise via bgp as type2 and type3 and that also OK 
and it success add to l2route except this mac is confuse me so if you can change SW to one host and do same show then add other host in other do same show 
MHM 

[shannonr]

Hi @MHM Cisco World thanks for that, and my apologies for the delay in getting back to you.

MAC address F839182A0D0D is the MAC address of the switch physical interface on the "client" switch connected to leaf 1 (.3). See below:

LEAF1_CLIENT#sh int g1/0/13
GigabitEthernet1/0/13 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is f839.182a.0d0d (bia f839.182a.0d0d)

LEAF1_CLIENT#sh run int g1/0/13
interface GigabitEthernet1/0/13
description TO_LEAF19500X
switchport trunk allowed vlan 100-103
switchport mode trunk
end

As instructed, I have changed the topology, so each "Client" switch is configured so that its uplink is in non-switchport mode and has an IP address configured on the physical interface connecting to the leaf switches. Example of the configuration for each "Client" below:

LEAF1_CLIENT(config-if)#do sh run int g1/0/13
interface GigabitEthernet1/0/13
description TO_LEAF19500X
no switchport
ip address 192.168.100.103 255.255.255.0
end

With this configuration the error appeared once when new NVE peers were brought up / added but after that there were no more errors.

I then moved each LEAF switch interface connecting to the "clients" back to a trunk port but left the "client" switches in no-switchport mode. This obviously broke the client connectivity as the VLAN tagging did not align but the nve peering came up and, again, the error appeared once and then stopped.

Next, I moved the "client" switch interfaces back to trunk mode adding one VLAN at a time and one "client" switch at a time but without any IP addressing on the "clients". The error did not re-appear.

At this point LEAF1 (.3) was once again advertising the physical interface MAC:

MNT-9500-16X_VTEP#show bgp l2vpn evpn neighbors 10.254.0.1 advertised-routes
BGP table version is 246158, local router ID is 10.254.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10.254.1.3:42867
*> [2][10.254.1.3:42867][0][48][F839182A0D0D][0][*]/20
0.0.0.0 32768 ?
Route Distinguisher: 10.254.1.3:42868
*> [2][10.254.1.3:42868][0][48][F839182A0D0D][0][*]/20
0.0.0.0 32768 ?
Route Distinguisher: 10.254.1.3:42869
*> [2][10.254.1.3:42869][0][48][F839182A0D0D][0][*]/20
0.0.0.0 32768 ?
Route Distinguisher: 10.254.1.3:42870
*> [2][10.254.1.3:42870][0][48][F839182A0D0D][0][*]/20
0.0.0.0 32768 ?
Route Distinguisher: 10.254.1.3:42867
*> [3][10.254.1.3:42867][0][32][10.254.1.3]/17
0.0.0.0 32768 ?
Route Distinguisher: 10.254.1.3:42868
*> [3][10.254.1.3:42868][0][32][10.254.1.3]/17
0.0.0.0 32768 ?
Route Distinguisher: 10.254.1.3:42869
*> [3][10.254.1.3:42869][0][32][10.254.1.3]/17
0.0.0.0 32768 ?
Route Distinguisher: 10.254.1.3:42870
*> [3][10.254.1.3:42870][0][32][10.254.1.3]/17
0.0.0.0 32768 ?

Total number of prefixes 8

LEAF1-9500-16X_VTEP#show l2route evpn mac ip topology 10100
EVI ETag Prod Mac Address Host IP Next Hop(s)
----- ---------- ----- -------------- --------------------------------------- --------------------------------------------------

LEAF1-9500-16X_VTEP#show l2vpn evpn evi 10100 det
EVPN instance: 10100 (VLAN Based)
Profile: pot-inter-dc
RD: 10.254.1.3:42867 (auto)
Import-RTs: 23456:10100
Export-RTs: 23456:10100
Per-EVI Label: none
State: Established
Replication Type: Ingress (profile)
Encapsulation: vxlan (profile)
IP Local Learn: Enabled (global)
Adv. Def. Gateway: Disabled (global)
Re-originate RT5: Disabled (profile)
Adv. Multicast: Enabled (profile)
AR Flood Suppress: Enabled (global)
Vlan: 100
Protected: False
Ethernet-Tag: 0
State: Established
Flood Suppress: Attached
Core If:
Access If:
NVE If: nve1
RMAC: 0000.0000.0000
Core Vlan: 0
L2 VNI: 100100
L3 VNI: 0
VTEP IP: 10.254.1.3
Pseudoports:
TenGigabitEthernet1/0/9 service instance 100
Routes: 1 MAC, 0 MAC/IP
Peers:
10.254.1.4
Routes: 0 MAC, 0 MAC/IP, 1 IMET, 0 EAD
10.254.1.5
Routes: 1 MAC, 0 MAC/IP, 1 IMET, 0 EAD
10.254.1.6
Routes: 0 MAC, 0 MAC/IP, 1 IMET, 0 EAD

Note: At this point, without any IP addressing on the VLAN interfaces on the "client" switch, there is no MAC addresses for the individual VLANs (no SVIs).

I then added an SVI to the "client" switch attached to LEAF1 (.3) for the VLAN 100 interface. The error has once again started again on all LEAF switches. 

So, the error only happens when SVI interfaces are created on the "client" switches which is when each VLAN/SVI has a MAC address associated with it. It also happens even when only one "client" switch has an SVI for a single VLAN.

Lastly, I tried removing the IP address off the SVI (but still having the SVI / VLAN Interface). This resulted in no errors but also no MAC associated when looking at:

LEAF1-9500-16X_VTEP#show l2route evpn mac ip topology 10100
EVI ETag Prod Mac Address Host IP Next Hop(s)
----- ---------- ----- -------------- --------------------------------------- --------------------------------------------------

I'm not sure what to make of that. Any ideas?

I have attached all the same output as previous for your review - this is at the point where there was no trunk configured between "client" switch and LEAF and the interface was configured on the "client" switch as "no switchport".

[MHM Cisco World]

Your last reply is excellent, your config is so correct if with client there is no error and only when you add SVI, that from my view good

For SVI I will check but I think we need to add each SVI into different vrf but let me check update you tomorrow 

Thanks for sharing new updates 

MHM

[shannonr]

Just to close this out - I ended up raising this with Cisco TAC as it seemed there was no configuration issue and maybe this was a bug. They have confirmed the error can be safely ignored and does not represent an underlying issue:

"The error log should be a cosmetic one, and there should not be any impact due to these logs. I’ve checked about this on similar cases internally as well and it does seem to be a cosmetic log with no impact."

@MHM Cisco World and M02@rt37 a huge thanks to both of you for your help in fixing up some of the original configuration I had, and for helping to validate and eliminate this as a configuration issue!! Really appreciate the time and effort you have spent in helping me.

[M02@rt37]

You're so welcome @shannonr 

Thanks for your feedback.