Re: VXLAN NVE L3 VNI Bridge Domain Operational state down

cmeyer3151 · ‎03-06-2019

I have 2 Nexus 7700 VDCs configured in a VPC running VXLAN BGP EVPN fabric. I seem to be having a problem getting the L3 VNI for the Tenant to come up. The problem appears to be due to the Bridge Domain tied to the vni being down.

N7K2-ADMIN-N7K2-LEAF2(config-router)# sh nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast

Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 20103 239.0.0.103 Up CP L2 [103] SA
nve1 20150 239.0.0.150 Up CP L2 [150] SA
nve1 20201 239.0.0.201 Up CP L2 [201] SA
nve1 21001 n/a Down CP L3 [--]
nve1 22048 239.0.0.248 Up CP L2 [2048] SA
nve1 22050 239.0.0.250 Up CP L2 [2050] SA
nve1 23092 239.0.0.92 Up CP L2 [3092] SA

I shut/no shut the interfaces and the bridge domain 1001 that is tied to vni 21001. When looking at the interface bdi 1001 for this Tenant, it is telling me the BD\VLAN is down.

N7K2-ADMIN-N7K2-LEAF2(config-router)# sh int bdi 1001
Bdi1001 is down (VLAN/BD is down), line protocol is down, autostate enabled
Hardware is EtherSVI, address is 00de.fb7b.b544
Description: L3VNI_for_TETRATION
MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA
Last clearing of "show interface" counters never
60 seconds input rate 0 bits/sec, 0 packets/sec
60 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
L3 Switched:
input: 0 pkts, 0 bytes - output: 0 pkts, 0 bytes
L3 in Switched:
ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes
L3 out Switched:
ucast: 0 pkts, 0 bytes - mcast: 0 pkts, 0 bytes

Bridge-domain 1001 (0 ports in all)
Name:: Bridge-Domain1001
Administrative State: UP Operational State: DOWN
vni21001

I can't for the life of me figure out why this L3 VNI for the Tenant will not come up. I have OSPF/BGP adjacency established. All of the VTEPs can reach each-other. Any help is appreciated.

Thanks

Chris

Andrea Testino · ‎03-07-2019

Hi Chris,

Can you share the outputs to the following CLI (do correct VNI/BDI ##s if needed):

show service instance vni detail 
show nve interface
show bridge-domain 1001
show spanning-tree bridge-domain 1001
show nve vni 21001
show nve vni 21001 detail 
show nve peers
show interface bdi 1001
show run vni
show run

In a working state, your VNI/BDI outputs should look similar to this:

N7K-Leaf# show service instance vni detail 

VSI: VSI-Ethernet9/1.4095(default)
If-index: 0x35400fff
Admin Status: Up
Oper Status: Up
Auto-configuration Mode: Data-frame snooping
encapsulation profile vni md0
  dot1q  vni 
Dot1q   VNI     BD
------------------
3001    30001   3010
3002    30002   3011
3003    30003   3012
3004    30004   3013
3005    30005   3014
3006    30006   3015
3007    30007   3016
3008    30008   3017
3009    30009   3018
3010    30010   3019
 
N7K-Leaf# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
 VPC Capability: VPC-VIP-Only [not-notified]
 Local Router MAC: 002a.6ab7.19c6
 Host Learning Mode: Control-Plane BGP
 IR Capability Mode: No
 Source-Interface: loopback1 (primary: 20.254.2.2, secondary: 0.0.0.0)

N7K-Leaf# show bridge-domain 3018

Bridge-domain 3018  (2 ports in all)
Name:: Bridge-Domain3018
 Administrative State: UP               Operational State: UP
        vni30009
        VSI-Eth9/1.4095
        nve1

N7K-Leaf# show spanning-tree bridge-domain 3018

BD3018
  Spanning tree enabled protocol rstp
  Root ID    Priority    35786
             Address     002a.6ab7.19c6
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    35786  (priority 32768 sys-id-ext 3018)
             Address     002a.6ab7.19c6
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VSI-Eth9/1.4095  Desg FWD 2         128.1153 P2p  
  
N7K-Leaf# show nve vni 30009
Codes: CP - Control Plane        DP - Data Plane          
       UC - Unconfigured         SA - Suppress ARP        
       SU - Suppress Unknown Unicast
 
Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1      30009    239.1.1.12        Up    CP   L2 [3018]          SA    

N7K-Leaf# show nve vni 30009 detail 
VNI: 30009 
  NVE-Interface       : nve1
  Mcast-Addr          : 239.1.1.12
  VNI State           : Up
  Mode                : control-plane
  VNI Type            : L2 [3018]
  VNI Flags           : SA    
  Provision State     : vni-add-complete
  Vlan-BD             : 3018
  SVI State           : n/a
  
N7K-Leaf# show nve peers
Interface Peer-IP          State LearnType Uptime   Router-Mac       
--------- ---------------  ----- --------- -------- -----------------
nve1      20.254.2.3       Up    CP        1w2d     8c60.4f32.5d44   
nve1      20.254.2.24      Up    CP        1w2d     a8b4.5670.cfaf   
nve1      20.254.2.200     Up    CP        1w2d     002a.6ab7.19c7   
nve1      20.254.2.201     Up    CP        1w2d     8c60.4f32.5d42   
nve1      20.254.3.1       Up    CP        1w2d     843d.c69f.7847   
nve1      20.254.3.2       Up    CP        1w2d     707d.b9b8.4155   
nve1      20.254.3.20      Up    CP        1w2d     380e.4d12.2d6b   

N7K-Leaf# show interface bdi 3018
Bdi3018 is up, line protocol is up, autostate enabled
<snip>

It is possible we are missing one or two lines of configuration somewhere -- Happy to help.

- Andrea, CCIE #56739 R&S

cmeyer3151 · ‎04-06-2020

I found the cause of this a long time ago. Having Feature VPC enabled on the 7K core switches (Spines) without VPC properly configured caused the L3VNI and DB to enter the down state due to the lo0 interface being down.

Andrii Oliinyk · ‎08-15-2020

Greetings

i've similar problem on the different platform. obviously for BDI to go in UP state bridge-domain must have at list 1 intf/instance in UP state. for this i've created int loX to make it member of target BDI. but int BDI doesnt accept LoX as a member. I wonder how Loopback was involved in your config to bring BDI UP.