ASR9000/XR Flexible VLAN matching, EVC, VLAN-Tag rewriting, IRB/BVI and defining L2 services

[xthuijs]

Introduction

In this article we'll be discussing the EVC (Ethernet Virtual Circuit) infrastructure that the ASR9000 and IOS-XR use to define L2 services.

You'll find in this article how packets coming in on an interface are being matched against an EFP (Ethernet Flow Point), how to manipulate the vlan tags and use them in XCONNECT (VPWS) or Bridging (eg VPLS) scenarios with the optional L3 endpoint to provide IRB (Integrated Route Bridging)

Core Issue

The way that IOS-XR handles EVC's is a bit different then the way that IOS handles it when for instance using the 7600.

When starting with IOS-XR and the ASR9000 there are a few things that you will want to be aware of when defining L2 services.

A couple of the key differences are:

- XR does have the concept of a trunk interface

- therefore XR cannot do vlan pruning

- XR does not strip vlan tags by default.

- XR does not have the concept of an "interface VLAN" a.k.a. SVI's (Switch Virtual Interface), instead it uses a BVI (Bridge Virtual Interface) that can be inserted into the bridge-domain.

Flexible VLAN matching

     How the ASR9000 matches traffic to an EFP

TAG rewriting

     how to modify the vlan headers and how that works

Defining L2 services

     and a comparison with IOS

Providing L3 services in L2VPN

     using IRB/BVI interfaces

Resolution

In order to define an L2 service, we need to match traffic to a particular interface. In IOS-XR and the ASR9000 we use the Ethernet Flow Point (EFP) to match this traffic. An EFP is effectively a subinterface of a physical interface with the keyword "l2transport" attached to it. This l2transport defines that we are going to use this (sub) interface for L2 Services. We can match L2 and L3 interfaces on a single physical port:

Flexible VLAN matching

When traffic is coming in on a port, we use the TCAM to find out which particular subinterface this traffic matches on. With that in mind there are a couple of rules as to how traffic is matched.

An EFP is defined as follows:

RP/0/RSP0/CPU0:asr(config)#int gig 0/0/0/4.100 l2transport

RP/0/RSP0/CPU0:asr(config-subif)#encapsulation ?

  default  Packets unmatched by other service instances

If a particular packet is not matching any other specific EFP on this physical port, this "Default" will capture all unmatched traffic.

  dot1ad    IEEE 802.1ad VLAN-tagged packets

  dot1q     IEEE 802.1Q VLAN-tagged packets

  untagged Packets with no explicit VLAN tag

If there is no vlan tag on the packet, the "untagged" EFP will capture this traffic, this is effectively plain ethernet and useful for instance to capture

BPDU's for instance.

When we are going to match on say dot1q encapsulated traffic we have a variaty of how we can match vlan tagged traffic (see also foot note below in the "Related Information" section on the ethertypes used).

RP/0/RSP1/CPU0:A9K-BOTTOM(config-subif)#encapsulation dot1q ?
  <1-4094>         Start of VLAN range
  <1-4094>         Single VLAN id
  any              Match any VLAN id
  priority-tagged  IEEE 802.1ad priority-tagged packets

At the first level of dot1q classification we can select a vlan, vlan-range or any. These are obvious. The option "Priority tagged" allows us to capture vlan encapped traffic that is with a vlan id of 0.

RP/0/RSP1/CPU0:A9K-BOTTOM(config-subif)#encapsulation dot1q 100 ?
  comma         comma
  exact         Do not allow further inner tags
  ingress       Perform MAC-based matching
  second-dot1q IEEE 802.1Q VLAN-tagged packets

Here is an important concept that is to be highlighted. You see the "word" exact" here. What that means is, in the absence of the keyword exact, if the outter vlan header is "100" in this example, this EFP is matched. so that means that also qinq frames that are of the 100 outter and 200 inner kind (if there is no specific EFP for the qiq combo 100/200 available) will match this EFP.

Just a few examples:

encapsulation dot1q 100: will match any number of vlan headers as long as the outter vlan id is 100

encapsulation dot1q 100 second any: will match any qiq frame where the outter vlan is 100

encapsulation dot1q 100 second 200: will match vlan tagged packets whereby the outter is 100, the inner is 200 and also a potential vlan combo of 100/200/300

encapsulation dot1q 100 second 200 exact: will match vlan tagged packets whereby the outter is 100, the inner is 200 and no other vlan tags are on the packet then these 2 specified.

Normally "longest match" will win, or better put, the most specific match will win.

ASR9000 doesn't support best match for VLAN ranges, but we do support best match if the "any" keyword is used. 

So the configuration :

EFP 1 VLAN: S-VLAN=1000, C-VLAN_range=1-4095

EFP 2 VLAN= S-VLAN=1000, C-VLAN=2000

isn't allowed because the more specific C-vlan is part of the range. The parser will reject this config upon commit.

The following options A and B, achieving the same, are allowed :

A)

EFP 1) VLAN: S-VLAN=1000, C-VLAN_range=1-1999,2001-4095

EFP 2) VLAN= S-VLAN=1000, C-VLAN=2000

B)

EFP 1) VLAN: S-VLAN=1000, C-VLAN=any

EFP 2) VLAN= S-VLAN=1000, C-VLAN=2000,

VLAN  TAG rewriting

The ASR9000/XR is capable of doing many translations on vlan to a packet.

The behavior is always symmetric (though this keyword is to be provided as part of that config command). The symmetry means that if we pop a tag on ingress, we push it back on egress.

the following rewrites are possible:

RP/0/RSP0/CPU0:A9K(config)#int gig 0/0/0/4.100 l2transport

RP/0/RSP0/CPU0:A9K(config-subif)#rewrite ingress tag ?
  pop        Remove one or more tags
  push       Push one or more tags
  translate  Replace tags with other tags

RP/0/RSP0/CPU0:A9K(config-subif)#rewrite ingress tag pop ?
  1  Remove outer tag only
  2  Remove two outermost tags

RP/0/RSP0/CPU0:A9K(config-subif)#rewrite ingress tag push ?
  dot1ad  Push a Dot1ad tag (ethertype 88a8)
  dot1q   Push a Dot1Q tag (ethertype 8100 by default, or 9100/9200 is specied on the main interface for the outter most tag)

RP/0/RSP0/CPU0:A9K(config-subif)#rewrite ingress tag push dot1q 100 ?
  second-dot1q  Push another Dot1Q tag
  symmetric     All rewrites must be symmetric

RP/0/RSP0/CPU0:A9K(config-subif)#rewrite ingress tag translate ?
  1-to-1  Replace the outermost tag with another tag
  1-to-2  Replace the outermost tag with two tags
  2-to-1  Replace the outermost two tags with one tag
  2-to-2  Replace the outermost two tags with two other tags

If you want to make a cross connect or bridge between two EFP's where one EFP is vlan 100 and the other EFP is vlan 200,

you need to make sure you pop the tags so that the vlan 100 is removed from the packet so it, by means of symmetry, will get vlan 200 on the egress

of the other EFP.

L2VPN configuration examples and comparison to 7600

The following picture highlights how to create L2 services on the ASR9000.

Using IRB/BVI

To provide L3 services in a bridge-domain, you can add a routed interface to the bridge domain.

What is important here is that the BVI is not vlan tagged. So in order for the EFP's to talk to the BVI, we need to pop ALL Tags on ingress!!

This means that frames with more then 2 tags cannot be natively using the BVI, unless we do some workarounds such as loopback cables to

pop more tags.

interface BVI5
ipv4 address 31.1.1.1 255.255.255.0
mac-address 0.4343.3434

! when creating the bvi verify the show int bvi to see if the mac address is correctly programmed, these macs are coming from the backplane mac table, ! if we ran out because of so many bundle interfaces etc, you may need to provide a mac manually.

interface TenGigE0/7/0/3.100 l2transport

encapsulation dot1q 100

rewrite ingress tag pop 1 symmetric

l2vpn

bridge group testing
  bridge-domain testing
   interface TenGigE0/7/0/3.100
   !
   interface GigabitEthernet0/1/0/14
   !
   routed interface BVI5
  !

Related Information

XR MTU calculations

Dot1q, Dot1ad and configuring "dot1q tunneling ethertype on the physical interface:

As per IEEE documentation, Ethertype 8100 is for 802.1q and Ethertype 88A8 is for 802.1ad.

IEEE also calls any kind of double tagging as QinQ, including dot1ad. That means that under IEEE's point of view, dot1ad = qinq.

Cisco, on the other hand, calls qinq a double dot1q, where both Ethertypes (inner and outer) are 8100.

Ethertype 9100 and 9200 were used as an option to differentiate the inner Ethertpe form the outer Ethertype,

but they are obsolete now and the standard became dot1ad (with Ethertype 88A8).

In order to configure ethertype 9100 and 9200 you have to explicitly issue the command dot1q tunneling ethertype 0x9100 or

dot1q tunneling ethertype 0x9200 under the physical interface you are configuring.

Summarizing:

encapsulation dot1q second dot1q                                     -> inner and outer ethertype are 8100

encapsulation dot1ad second dot1q                                   -> inner 8100, outer 88A8

encapsulation dot1q second dot1q (with ethertype 9100)      -> inner 8100, outer 9100

encapsulation dot1q second dot1q (with ethertype 9200)      -> inner 8100, outer 9200

Xander Thuijs, CCIE #6775

Sr. Tech Lead ASR9000

[xthuijs]

hi !

you can add them via the l2vpn bridge domain config like this:

l2vpn

bridge group myvlans

bridge-domain vlan100

int tenGigE0/7/0/3.100

routed-interface bvi200

bridge-domain vlan200

.<efp>

.<routed-interface>

cheers!

xander

ps. bridge group is just a config hierarchy, bridgedomain is the instantiation of the mac learning entity.

[m-stawiski]

Hi

Sorry still confused a litte bit so my whole bridge-group should looks like ?

l2vpn

bridge group myvlans

bridge-domain vlan100

int tenGigE0/7/0/3.100

routed-interface bvi200

bridge-domain vlan200

routed-interface bvi201

bridge-domain vlan201

routed-interface bvi202

bridge-domain vlan202

how in this case for example bvi202 will be know that it is assinged to tenGigE0/7/0/3.100 client vlan 202 ?

unless i need to create interfaces like bellow

interface TenGigE0/7/0/3.100200 l2transport
encapsulation dot1q 100 second-dot1q 200
rewrite ingress tag pop 1 symmetric

interface TenGigE0/7/0/3.100201 l2transport
encapsulation dot1q 100 second-dot1q 201
rewrite ingress tag pop 1 symmetric

interface TenGigE0/7/0/3.100202 l2transport
encapsulation dot1q 100 second-dot1q 202
rewrite ingress tag pop 1 symmetric

and the use those with bridge domains like bellow

l2vpn

bridge group myvlans

bridge-domain vlan200

   interface TenGigE0/7/0/3.100200

   routed-interface bvi200

bridge-domain vlan201

   interface TenGigE0/7/0/3.100201

   routed-interface bvi201

bridge-domain vlan202

   interface TenGigE0/7/0/3.100201

   routed-interface bvi202

???

[smailmilak]

Hi,

you got it. Just put the interface and BVI under same BD and you are all set.

btw. BVI does not care about VLAN tag. 

[thomas.hager]

Hi Alexander!

First of all: great explanation from you - thanks a lot! These are very useful!!!

I am currently working on the migration of our BBA/BNG-Router from C3925 to ASR1001-X and I have a strange behavior concerning EVC-framework or rewrite action on ASR9k in XR 5.3.3

Simplified scenario:

As you can see in the picture, our customers are using pppoe-dialer for dialup into our new ASR1001-X BBA-Router. From the left we are getting QinQ-Frames from the DSLAMs into our ASR920: S-Tag is DSLAM-Tag, C-Tag is for each customer on this DSLAM.

These double-tagged frames are sent into a PW unmodified (via VPLS, auto-discovery, LDP-based) and sent to the ASR9k. The 9k is just doing L2-stuff, meaning bridging between VFI and sub-interfaces to the new ASR1k PPPoE-Router. PW is working fine and all frames arrive correctly at 9k (verification with wireshark).

Because of design purposes we need to do VLAN-translation of the outermost tag and send the frames to the 1k. In our case we are translating S-Tag 10 –to-> S-TAG 1910, S-Tag 11 –to-> S-Tag 1911,…. and so on.

 Here’s the 9K and 1K config for more clearness:

 9K:

interface Te0/0/0/1.1910 l2transport

 encapsulation dot1q 1910 second-dot1q any

 rewrite ingress tag translate 1-to-1 dot1q 10 symmetric

interface Te0/0/0/1.1911 l2transport

 encapsulation dot1q 1911 second-dot1q any

 rewrite ingress tag translate 1-to-1 dot1q 11 symmetric

l2vpn

 bridge group BRIDGE-GROUP1

  bridge-domain BD_1000

   mtu 9180

   interface Te0/0/0/1.1910

   interface Te0/0/0/1.1911

   !

   vfi VFI100

    vpn-id 20000

    autodiscovery bgp

     rd 20000:1000

     route-target import 20000:1000

     route-target export 20000:1000

     signaling-protocol ldp

     !

     control-word

1K:

interface Te0/0/0.1910

 encapsulation dot1Q 1910 second-dot1q any

 pppoe enable group pppoe_www

interface Te0/0/0.1911

 encapsulation dot1Q 1911 second-dot1q any

 pppoe enable group pppoe_www

Problem-description:

If the clients are using dialer to dial in via pppoe, the PADI frames (broadcast) from ASR9k are sent through BOTH sub-interfaces to ASR1k! After frames arrive at ASR1k, the 1k sends PADO to one of his sub-interfaces; if it’s the correct sub-if everything is fine; if the ASR1K sends it back the wrong sub-if, the frame does not go back to customer-router (timeout, resend etc.)

This is just an example; in productive we have 100s of sub-interfaces and the ASR9k is flooding the PADI-broadcast to ALL sub-interfaces!

ASR1k debug of pppoe events:

Aug 17 09:29:49.577: PPPoE 0: I PADI  R:00d7.8f03.1c81 L:ffff.ffff.ffff 1910 Te0/0/0.1910

Aug 17 09:29:49.577: PPPoE 0: I PADI  R:00d7.8f03.1c81 L:ffff.ffff.ffff 1913 Te0/0/0.1913

I also created some wireshark captures where I could see the same behavior:

I searched the web and found Cisco docs where this looks as supported feature (1-to-1 translation of the outermost tag of double tagged frames)

Tries:

If I change the config not to translate 1-to-1 everything works fine and the frames are sent through the correct sub-interfaces.

If I change the second-dot1q from “any” to “1-3000” there is still the same issue; frames are flooded on both sub-interfaces.

Do you have any idea on this?

Thank you in advance!

Thomas

[xthuijs]

hi thomas!! thank you :) aha I see what the problem is. because you are using a bridge domain, you basically default to mac learning forwarding basics. this means that a broadcast which the PADI is will get out of both EFP's. Each EFP will translate it according to its configuration correctly and the BNG will get 2 PADI's now. it will send 2 offers for each vlan combo. 

because of the design, you probably better of carrying each S tag on a separate PW and use an xconnect to link the PW to the EFP.

this way the BNG will only see one discovery and will reply back where the a9k EFP will xlate it as desired and xcon it into the PW directly.

difference between xcon and bridgedomain really is mac learning. a BD is multipoint with a mac relearning resource. xcon is basically whatever comes in goes out the other leg.

or alternatively, if you like to continue to use the auto discovery, you can create 2 bridge domains, and use one EFP per BD and one auto discovered PW per BD/vfi.

because we still use mac learning here, you are limited to 2M mac's = subs. system wide.

cheers

xander

[stefano.toffolo]

Hi Xander,

in the process of deploying BNG, I decided to push Q-in-Q as close as possible to the access switch, where subscriber are actually connected. This allows for the same C-VLAN  IDs on every access switch and requires a unique S-VLAN for each access switch on the aggregation switch.

This works fine for the access interface on ASR9001 doing BNG, since I just match ambiguously on the S-VLAN ID range:

encapsulation ambiguous dot1q 1101-1900 second-dot1q any

where the S-VLAN IDs 1101-1900 indentify each access switch.

Unfortunately, on the 3850 I can't do selective Q-in-Q so that also the C-VLAN ID for the management of the access switches will be double-tagged to the same range. This forces me to take ist out from every S-VLAN on the ASR9001 (every access switch, since matching ambiguously doens't work on the outer tag) like this:

interface Bundle-Ether1.1101 l2transport
 encapsulation dot1q 1101 second-dot1q 133
 rewrite ingress tag pop 2 symmetric
!
interface Bundle-Ether1.1102 l2transport
 encapsulation dot1q 1102 second-dot1q 133
 rewrite ingress tag pop 2 symmetric
!

.

and put everything into a BD.

Since I have hundreds of those access switches, do you see a better way to reduce the amount of sub-interfaces for management while keeping the advantage of having an easy konfiguration on every access switch?

Thanks in advance!

Cheers,

Stefano

[marco@lamehost]

Hello Xander,

Very quick question.

Is it possible to configure L3 ACLs on EVCs on ASR9k?
It is something that you can do on ASR1k, but can't find a way on XR.

On IOS-XE it looks like

interface GigabitEthernet0/0
 service instance 10 ethernet
  encapsulation dot1q 10
  rewrite ingress tag pop 1 symmetric
  bridge-domain 1 split-horizon group 1
  ! *** IP access-list follows ***
  ip access-group CUST_A in

Thanks you

[Anuj Singhi]

Hi Xander,

Great Article and I am pretty sure, it must still be helping a lot of engineers.

I have a question regarding VPLS setup in ASR9K which I have seen in my client environment. They have two ASR over two sites over which thousand of vlans are getting stretched and correspondingly thousand of subinterface's have been configured over the BE interfaces.

What I found strange was they have created bridge domains for only limited number of sub-interfaces, does it mean they if all other sub-interfaces on that particular BE is getting tunneled through one BD only?

Also I still don't get the concept of routed bridge interfaces, why do we need BVI's over stretched l2 fabric when you can have it through normal MPLS circuit (I know it is quite naive question).

Regards,

Anuj

[Deniz AYDIN]

Hi Xander, 

I am trying to understand tag-rewrite ingress vlan command under pw-class. Does it mean to add-remove vlan towards the pseudowire like the inteface command rewrite ingress tag pop 1 symmetric?

pw-class test
encapsulation mpls
tag-rewrite ingress vlan 100

[Jhonny Estrella]

Hi Xander,

I have the issue in revision of the vlan:

how can I verify the vlans in IOS XR related to an interface or ARP or MAC,   I only achieved was to count ports, IDs, etc. but to verify services I need to relate to something, someone saw that case?

IOS-XR#show l2vpn forwarding bridge-domain location 0/3/CPU0
Wed Apr 24 10:16:21.660 GMT
Bridge MAC
Bridge-Domain Name ID Ports HW addr SW addr Flooding Learning State
-------------------------------- ------ ----- ------- ------- -------- -------- ---------
RC:VLAN35 0 1 1 1 Enabled Enabled UP
RC:VLAN45 1 1 1 1 Enabled Enabled UP
RC:VLAN50 2 1 0 0 Enabled Enabled UP
RC:VLAN51 3 2 4 2 Enabled Enabled UP

I need other command for relation it with interface or ARP o MAC. thanks ..

[smailmilak]

Hi Xander,

One question about this sentence "The behavior is always symmetric (though this keyword is to be provided as part of that config command). The symmetry means that if we pop a tag on ingress, we push it back on egress"

With that you mean that the tag will be pushed back on the SAME interface, correct?
To the other side of the PW no VLAN tag will be transported, meaning that the other side of the PW can be the same VLAN or another, depending on our configuration on that particular interface to the customer's equipment?