I have a network with multiple (5) ISR4321s in an ELAN. I have set up service routing on all of these routers for the purposes of running different topologies and stacking different networks on the equipment. For example, some of the service instances belong to a flat layer 2 network that runs OSPF for distribution across the ELAN. Some of the services each have their own Vlan on the ELAN and translate tags from ELAN Vlans to a common internal Vlan for the management network. Some of these are e-lines from one campus to my internet drain for the purpose of giving that campus a dedicated internet access on my network.
I am having what I think to be a really odd issue with the bridge domains for my management network. All of these are in a VRF called "Management", and each BDI is running VRRP between two of my routers. On my campus side, for every campus, all "Management" traffic lives on Vlan 999. On my distribution side, each campus has a dedicated Vlan 900+(headend number). Each dedicated management Vlan has its own subnet as well. Here is an example topology. Color code is: light blue - physical link + L2 information; pink - distribution routers; purple - core/distributing routers running VRRP and housing all service instances; green - access node; black - management hosts.
! CR0 ELAN facing interface interface GigabitEthernet0/0/1 description "Provider edge" service instance 900 ethernet description "PE-SITE0-MGMT-NET" encapsulation dot1Q 900 rewrite ingress tag pop 1 symmetric bridge-domain 900 service instance 901 ethernet description "PE-SITE1-MGMT-NET" encapsulation dot1Q 901 rewrite ingress tag pop 1 symmetric bridge-domain 901 service instance 902 ethernet description "PE-SITE2-MGMT-NET encapsulation dot1Q 902 rewrite ingress tag pop 1 symmetric bridge-domain 902 ! interface BDI900 description "Router-SITE0-MGMT-NET" vrf forwarding Management ip address 10.0.99.2 255.255.255.0 vrrp 90 description "SITE0-MGMT-NET" vrrp 90 timers advertise 30 vrrp 90 priority 254 vrrp 90 ip 10.0.99.1 ! interface BDI901 description "Router-SITE1-MGMT-NET" vrf forwarding Management ip address 10.1.99.2 255.255.255.0 vrrp 91 description "SITE1-MGMT-NET" vrrp 91 timers advertise 30 vrrp 91 priority 254 vrrp 91 ip 10.1.99.1 ! interface BDI902 description "Router-SITE1-MGMT-NET" vrf forwarding Management ip address 10.2.99.2 255.255.255.0 vrrp 92 description "SITE2-MGMT-NET" vrrp 92 timers advertise 30 vrrp 92 priority 254 vrrp 92 ip 10.2.99.1 !Here is the relevant config from CR1:
! CR1 ELAN facing interface interface GigabitEthernet0/0/1 description "Provider edge" service instance 900 ethernet description "PE-SITE0-MGMT-NET" encapsulation dot1Q 900 rewrite ingress tag pop 1 symmetric bridge-domain 900 service instance 901 ethernet description "PE-SITE1-MGMT-NET" encapsulation dot1Q 901 rewrite ingress tag pop 1 symmetric bridge-domain 901 service instance 902 ethernet description "PE-SITE2-MGMT-NET encapsulation dot1Q 902 rewrite ingress tag pop 1 symmetric bridge-domain 902 ! interface BDI900 description "Router-SITE0-MGMT-NET" vrf forwarding Management ip address 10.0.99.3 255.255.255.0 vrrp 90 description "SITE0-MGMT-NET" vrrp 90 timers learn vrrp 90 priority 99 vrrp 90 ip 10.0.99.1 ! interface BDI901 description "Router-SITE1-MGMT-NET" vrf forwarding Management ip address 10.1.99.3 255.255.255.0 vrrp 91 description "SITE1-MGMT-NET" vrrp 91 timers learn vrrp 91 priority 99 vrrp 91 ip 10.1.99.1 ! interface BDI902 description "Router-SITE1-MGMT-NET" vrf forwarding Management ip address 10.2.99.3 255.255.255.0 vrrp 92 description "SITE2-MGMT-NET" vrrp 92 timers learn vrrp 92 priority 99 vrrp 92 ip 10.2.99.1 !Here is the relevant config from a distribution router, which can be applied with dot1Q tag and subnet changes like a cookie cutter:
! DR0 ELAN facing interface interface GigabitEthernet0/0/1 description "Provider edge" service instance 900 ethernet description "PE-SITE0-MGMT-NET" encapsulation dot1Q 900 rewrite ingress tag pop 1 symmetric bridge-domain 999 ! ! DR0 LAN facing interface interface GigabitEthernet0/0/0 description "Customer edge" service instance 999 ethernet description "CE-Management" encapsulation dot1Q 999 rewrite ingress tag pop 1 symmetric bridge-domain 999 ! interface BDI999 description "Router-Management" vrf forwarding Management ip address 10.0.99.4 255.255.255.0 ! ip route vrf Management 0.0.0.0 0.0.0.0 10.0.99.1Here's the output of `show bridge-domain 999` on DR0:
DR0#show bridge-domain 999
Bridge-domain 999 (3 ports in all)
State: UP Mac learning: Enabled
Aging-Timer: 300 second(s)
BDI999 (up)
GigabitEthernet0/0/0 service instance 999
GigabitEthernet0/0/1 service instance 900
AED MAC address Policy Tag Age Pseudoport
- E8EB.34D4.E3A3 to_bdi static 0 BDI999 After trying to ping a host on a different subnet with `ping vrf Management 10.1.99.4 source 10.0.99.4`, this is the ARP table:DR0#show arp vrf Management BDI999 Protocol Address Age (min) Hardware Addr Type Interface Internet 10.0.99.4 - e8eb.34d4.e3a3 ARPA BDI999 Internet 10.0.99.1 23 0000.5e00.0167 ARPA BDI999So, why can't hosts ping across subnets within the same VRF with different BDIs? I tried to confirm traffic direction but cannot take a packet capture with the embedded packet capture because it does not support BDI interfaces.
