BDI (Bridge Domain Interface) on ASR 1001-X with single and double tagged packets

[selvakumar.vec]

Dear Guys,

As per the rough diagram attached, our ASR 1001-X router is getting both single and double tagged frames from the Switch S2. We are planning to provide multiple services over a single vlan. For that, our access provider is giving double tagged VLANs as per the rough diagram. They are giving via Aggregation CPE (Huawei Switch) as trunk link with double tagged vlans. And they will be giving multiple outer vlans (one for Each Exchange Eg. 1570,1571) and all outer vlans will be having 4 inner vlans (11,12,13,14) to give multiple services.

Instead of creating subinterface for each inner and outer vlan combinations, we want to group those with the inner vlans for the ease of management.

For that, we created EVCs for each inner and outer vlans and combined the inner vlans using bridge domain and BDI.

below are the sample configs.

int g0/0/1

service instance 12 ethernet
encapsulation dot1q 1570 second-dot1q 12
rewrite ingress tag pop 2 symmetric
bridge-domain 12
!
service instance 13 ethernet
encapsulation dot1q 1570 second-dot1q 13
rewrite ingress tag pop 2 symmetric
bridge-domain 13
!
service instance 14 ethernet
encapsulation dot1q 1570 second-dot1q 14
rewrite ingress tag pop 2 symmetric
bridge-domain 14
!
service instance 22 ethernet
encapsulation dot1q 1571 second-dot1q 12
rewrite ingress tag pop 2 symmetric
bridge-domain 12
!
service instance 23 ethernet
encapsulation dot1q 1571 second-dot1q 13
rewrite ingress tag pop 2 symmetric
bridge-domain 13
!
service instance 24 ethernet
encapsulation dot1q 1571 second-dot1q 14
rewrite ingress tag pop 2 symmetric
bridge-domain 14

We created BDI also as BDI12,BDI13 and BDI14. 

int BD12

ip address 10.12.0.1 255.255.0.0

similarly for BDI13 & BDI14.

In the above setup, router is getting the ARP entries from the BDI12 interface for all the devices  on the specified VLANs without any issue. But we cannot able to ping the device IPs even from the bdi12 interface.

For example, If device IP is 10.12.0.50, router is getting the MAC address for this IP in the ARP, but ping is not working.

Is anybody faced this issue?

Kindly advise me to resolve this.

Note: All the connections shown in the network between switches and router are trunk links.

Thanks,

Selva

[selvakumar.vec]

Dear Guys @Richard Burts , @Giuseppe Larosa , @Joseph W. Doherty , @Jon Marshall , @Georg Pauwen 

Could you help and advise me to resolve this issue.

I tried debug arp and found that, arp requests and arp replies happening continuously for every second and ARP also updated. But I cannot able to ping the IP 10.12.0.50.

please find the below debug arp samples.

082028: Mar 25 13:17:45.797: IP ARP: rcvd req src 10.12.0.50 6cc2.1763.9452, dst 10.12.0.1 BDI12
082029: Mar 25 13:17:45.798: IP ARP: sent rep src 10.12.0.1 6cb2.aeb9.96bf, dst 10.12.0.50 6cc2.1763.9452 BDI12
082030: Mar 25 13:17:45.798: IP ARP: rcvd req src 10.12.0.50 6cc2.1763.9452, dst 10.12.0.1 BDI12
082031: Mar 25 13:17:45.798: IP ARP: sent rep src 10.12.0.1 6cb2.aeb9.96bf, dst 10.12.0.50 6cc2.1763.9452 BDI12
082032: Mar 25 13:17:45.798: IP ARP: rcvd req src 10.12.0.50 6cc2.1763.9452, dst 10.12.0.1 BDI12
082033: Mar 25 13:17:45.798: IP ARP: sent rep src 10.12.0.1 6cb2.aeb9.96bf, dst 10.12.0.50 6cc2.1763.9452 BDI12
082034: Mar 25 13:17:45.798: IP ARP: rcvd req src 10.12.0.50 6cc2.1763.9452, dst 10.12.0.1 BDI12
082035: Mar 25 13:17:45.798: IP ARP: sent rep src 10.12.0.1 6cb2.aeb9.96bf, dst 10.12.0.50 6cc2.1763.9452 BDI12

082045: Mar 25 13:17:46.903: IP ARP: rcvd req src 10.12.0.50 6cc2.1763.9452, dst 10.12.0.1 BDI12
082046: Mar 25 13:17:46.903: IP ARP: sent rep src 10.12.0.1 6cb2.aeb9.96bf, dst 10.12.0.50 6cc2.1763.9452 BDI12
082047: Mar 25 13:17:46.903: IP ARP: rcvd req src 10.12.0.50 6cc2.1763.9452, dst 10.12.0.1 BDI12
082048: Mar 25 13:17:46.903: IP ARP: sent rep src 10.12.0.1 6cb2.aeb9.96bf, dst 10.12.0.50 6cc2.1763.9452 BDI12
082049: Mar 25 13:17:46.903: IP ARP: rcvd req src 10.12.0.50 6cc2.1763.9452, dst 10.12.0.1 BDI12
082050: Mar 25 13:17:46.903: IP ARP: sent rep src 10.12.0.1 6cb2.aeb9.96bf, dst 10.12.0.50 6cc2.1763.9452 BDI12
082051: Mar 25 13:17:46.903: IP ARP: rcvd req src 10.12.0.50 6cc2.1763.9452, dst 10.12.0.1 BDI12
082052: Mar 25 13:17:46.903: IP ARP: sent rep src 10.12.0.1 6cb2.aeb9.96bf, dst 10.12.0.50 6cc2.1763.9452 BDI12

and sh ip arp bdi12 output as below

Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.12.0.1 - 6cb2.aeb9.96bf ARPA BDI12
Internet 10.12.0.50 0 6cc2.1763.9452 ARPA BDI12

Kindly help me to resolve this,

Thanks in advance,

Selva

[Richard Burts]

Selva

The output you posted shows that communication is successful at layer 2. That device is requesting your mac address and you are responding, and you have learned the mac address for that device. If ping is not working then there must be some layer 3 issue. My first thought is whether it is possible that the device has a firewall or some security policy that is not allowing ping. Is there some other way that you could try to communicate with it?

[selvakumar.vec]

Thanks @Richard Burts  & @Giuseppe Larosa for your comments.

There is no ACL configured in BDI interface as well the firewall is disabled in PC for the testing.

My doubt is why it is getting arp request and sending arp request for every second as per my debug. In a normal operation, it should not happen right.

I have not done anything for this. But sometimes I can able to ping without any issue. And now there is no such continuous arp request and reply.

Sometimes, I am getting ping drops as below.

Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125

The above ping taken from the PC connected in the network.

At the same time, when pinging from the bdi12 interface also, I am getting the drops as below

ping 10.12.0.50 so bdi12 repeat 100

Type escape sequence to abort.
Sending 1000, 100-byte ICMP Echos to 10.12.0.50, timeout is 2 seconds:
Packet sent with a source address of 10.12.0.1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!...........................
......!.......!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 96 percent (960/1000), round-trip min/avg/max = 1/1/7 ms

It is not stable at all.

What do you think, am I missing anything here. 

Also sometimes when I am clearing the arp entries, the ping will work.

@Giuseppe Larosa , 

Regarding the VLANs, Actually fiber operator is giving one outer vlan for each exchange which will have 4 inner vlans for different services. Thats why we bridged same inner vlans in the single bridge domain to provide the service over inner vlans.

Also I tried adding only one VLAN rather multiple VLANs in the bridge domain, even that time also I faced all of the issues as observed above.

Which way we can resolve this issue.

Could you please help on this.

Thanks,

Selva

[Giuseppe Larosa]

Hello @selvakumar.vec ,

the following is interesting taken from your last post in this thread

Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125
Request timed out.
Reply from 10.12.0.50: bytes=32 time=1ms TTL=125

I would say that we see that :

a) likely ICMP traffic is "process switched" because it looks like to perform round robin over the two EVCs

b) 50 % of answers and 50% of losses because you have two EVCs outer vlan 1700 inner vlan 12 and outer vlan 1701 inner vlan 12.

Given your network scenario you should test it using two PCs and avoiding to use the router itself as an endpoint for note a) above.

with two Pcs with firewall disabled you should see a behaviour like:

100% success or 0 % success depending on CEF load balacing.

The connectivity issue is likely caused when the ICMP reply arrives on a different EVC. This is the reason why I have noted that your network design does not look like correct as it creates this ambiguity in my previous post.

Last using two PCs test using a setup like :

PC1 -VL12 --- SW1 ---  VL1700,VL12----   EVC1--   ASR1000 ---- EVC2 --- VL1701,VL12 --- SW2 -- VL12-- PC2

this should work the problem is that EVC1 and EVC2 should not be defined on the same physical interface in my opinion to avoid to create ambiguity.

Hope to help

Giuseppe

[selvakumar.vec]

Dear @Giuseppe Larosa ,

I agree that we can bridge the EVCs created in different physical interfaces.

But our objective is to bridge and terminate all of the same inner VLANs as a single interface in our Router using BDI  for the ease of management and we are not passing to any other device.

My only concern is, it is getting the ARP entry but communication is not happening properly.

why you are not recommending to bridge multiple VLANs in the same physical interface. Is there any reason.

As per below guidelines, We can bridge multiple VLANs in the same physical interface.

"You can connect multiple EFPs to the same bridge domain on the same physical interface, and each EFP can have its own matching criteria and rewrite operation. An incoming frame is matched against EFP matching criteria on the interface, learned on the matching EFP, and forwarded to one or more EFPs in the bridge domain. If there are no matching EFPs, the frame is dropped."

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/cether/configuration/xe-3s/ce-xe-3s-book/ce-ether-vc-infra-xe.html

Am I missing anything, why the communication is not happening properly with BDI.

I also checked in the PC associated with one of the EFPs, it is not getting the arp entries of other devices in the same bridge domain except for BDI interface.

Kindly help me to troubleshoot this.

Thanks for your valuable discussion.

Selva

[Giuseppe Larosa]

Hello @selvakumar.vec ,

you are using 802.1Q double tagged encapsulations on your EVC definition.

Remember that this kind of encapsulation has been introduced for QinQ tunneling with the following meaning:

the outer Vlan tag indicates a customer X

the inner vlan tag indicates a Vlan of customer X

In my opinion your configuration tries to do the opposite you would like to communicate with a device despite the different outer Vlan with the same inner vlan

the debug ARP does not provide Vlan tagging information

You are using EVCs with the following properties:

outer vlan 1700 inner vlan 12 --->  BDI 12

outer vlan 1701 inner vlan 12 ---> BDI 12

but this is not the way to use double Vlan tagging in my experience.

You can create a BDI for outer Vlan 1700 and have it to talk using single tagged frames with someone in vlan 1700.

I suggest you the following:

disable the EVC with outer vlan 1701 inner vlan 12 and repeat your tests

in this way it should work

in a normal network without vlan tags I would think of a personal firewall installed on the PC connected to the switch as suggested by @Richard Burts .

This might be the case for you too, but I wanted to advise that your configuration is not best practice.

Hope to help

Giuseppe