ASR-1000 bridge untagged interface with VLAN tagged subinterface

[firestorm-v1]

I'm at a bit of an impasse with trying to learn the ASR router and could use a kick in the right direction.  I'm using an ASR-1001 as an alternative to using AT&T's CPE using a GPON ONU-on-a-stick which works great.  The problem comes in that I can't access the GPON stick's management UI.

The GPON stick has two logical interfaces, one is an untagged interface meant for managing the stick, the other is a VLAN tagged interface that provides the upstream Internet connection.  The stick is inserted into Gi0/0/0 and the VLAN subinterface is attached to the VRF ATT_PUBLIC_ROUTING and pulls a DHCP address.  

The inward facing link is Gi0/0/1 which has two VLAN subinterfaces, Gi0/0/1.50 (management) and Gi0/0/1.900 (ATT Public Subnet).  Gi0/0/1.900 is connected to the ATT_PUBLIC_ROUTING VRF and has the default gateway IP for the public subnet and routes as expected.

The problem is that I can't figure out how to connect Gi0/0/1.50 and Gi0/0/0 (untagged) together so I can access the GPON stick's management UI.  I've tried to search on how to accomplish this and came up with a lot of information on how to set up the ASR1001-X, but very little on the ASR-1000.   Research seems to point to using a service instance that's untagged on Gi0/0/0 and tagged on Gi0/0/1 with the same broadcast domain ID, but when I set this up, I get no connection to the stick's management UI and I get an error that pops up on the console with fair regularity (maybe about once a minute?).  This happens anytime a service instance is set up on Gi0/0/1.

IOSXE_INFRA-3-PUNT_SVC_TYPE_UNKNOWN: Punted packet with unknown service type 73 (STP BPDU's)

On the switch that the Gi0/0/1 interface is connected, I've tried several incantations of spanning-tree portfast, but the error message and the lack of connectivity persist.

The switch (Cisco 2960-X) port is configured as such:

interface GigabitEthernet1/0/40
description ASR-1000-Gi0/0/1
switchport mode trunk
switchport trunk allowed vlan 50,900   (both VLANs are tagged)
spanning-tree portfast edge trunk

Based on what I'm seeing, the service instance is the correct way to do this, but I've only seen untagged to untagged or tagged to tagged examples, and it would seem that the ASR-1000 doesn't like rewrite (or specifically the "symmetrical" keyword).

I've since removed the service instance definitions from the ASR just to get things back to a "known working-ish" setup, but I can recreate the issue and post the full config (minus a few redactions) if that would help any.

Any ideas on how I can get this fixed up?  Thank you in advance.

[Richard Burts]

I am sure that there are things in your situation that I do not know, but it seems to me, based on your description, that you might want to configure the trunk to treat vlan 50 as the native vlan, which means that its traffic would not be tagged.

[firestorm-v1]

Hey Richard!  Thank you for your suggestion.  After reading my post, I realized I did a pretty sub-standard job of posting and probably dropped some important details.  Thank you for trying to decipher my mess and give a good reply anyway.

After I made the initial post, I kept stabbing at the ASR and finally figured out what was needed to get the thing to shut up about BPDUs.  I also considered reworking the ASR and asked myself the same question (about putting VLAN50 untagged on the downstream switch).

To fix the BPDU alert I initially posted about, I reconfigured the downstream switchport (Gi1/0/40) on the 2960X like so:

interface GigabitEthernet1/0/40
 description ASR1001-Gi0/0/1
 switchport trunk allowed vlan 50,900
 switchport trunk native vlan 50
 switchport mode trunk
 spanning-tree portfast edge trunk
 spanning-tree bpdufilter enable
end

This assigns VLAN50 as untagged and VLAN900 as tagged, and I can confirm that this works (VLAN900 has two access ports on this switch that are attached to my firewalls which are already working otherwise my wife would have left me during Thanksgiving, lol.)  This also stops the BPDU punt messages on the ASR.

On the ASR side of things, I still have the untagged GPON stick (10.0.5.10) interface on Gi0/0/0 which is unreachable from VLAN50.  Here's the almost full config of the ASR (Extra !'s removed and IP address redacted except where relevant) :

Current configuration : 2465 bytes
!
! Last configuration change at 21:19:56 UTC Mon Jan 1 2001
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
!
hostname Router
!
boot-start-marker
boot-end-marker
!
vrf definition ATT_PUBLIC_ROUTING
 !
 address-family ipv4
 exit-address-family
!
vrf definition INTERNAL_MGMT
 !
 address-family ipv4
 exit-address-family
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
no aaa new-model
!
subscriber templating
multilink bundle-name authenticated
!
license udi pid ASR1001 sn JAE190702CV
!
redundancy
 mode none
bridge-domain 50
!
bridge irb
!
interface GigabitEthernet0/0/0
 description GPON_ONT_STICK
 mac-address AAAA.BBBB.CCCC
 no ip address
 ip access-group GPON_MGMT_ALLOW in
 negotiation auto
 cdp enable
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface GigabitEthernet0/0/0.242
 encapsulation dot1Q 242
 vrf forwarding ATT_PUBLIC_ROUTING
 ip address dhcp client-id GigabitEthernet0/0/0
!
interface GigabitEthernet0/0/1
 description INTERNAL_VLANS
 no ip address
 negotiation auto
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface GigabitEthernet0/0/1.900
 description ATT_PUBLIC_VLAN
 encapsulation dot1Q 900
 vrf forwarding ATT_PUBLIC_ROUTING
 ip address AAA.BBB.CCC.DDD 255.255.255.248
!
interface GigabitEthernet0/0/2
 no ip address
 shutdown
 negotiation auto
!
interface GigabitEthernet0/0/3
 no ip address
 shutdown
 negotiation auto
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 ip address WWW.XXX.YYY.ZZZ 255.255.255.0
 negotiation auto
!
interface BDI50
 ip address 10.0.5.9 255.255.255.0
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
ip tftp source-interface GigabitEthernet0
ip route vrf Mgmt-intf 0.0.0.0 0.0.0.0 10.0.0.1
!
ip access-list standard GPON_MGMT_ALLOW
 permit 10.0.0.0 0.0.0.255
 permit 10.1.0.0 0.0.0.255
 deny   any
ip access-list standard ROUTER_SECURE_VTYS
 permit 10.0.0.0 0.0.0.255
 permit 10.1.0.0 0.0.0.255
 deny   any
!
bridge 1 protocol ieee
!
control-plane
!
line con 0
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 access-class ROUTER_SECURE_VTYS in
 access-class ROUTER_SECURE_VTYS out
 login
line vty 5 15
 access-class ROUTER_SECURE_VTYS in
 access-class ROUTER_SECURE_VTYS out
 login
!
end

Through several howtos and example configs, I added the BDI (but to be honest I'm not fully sure I understand it).  I can ping the BDI IP address, but can't ping the GPON stick at 10.0.5.10.  I know the stick is configured correctly, if I plug it into a basic media converter, the GPON stick is pingable and the WebUI is reachable.

[Ramblin Tech]

If you are familiar with the SVI concept, a layer-3 interface on a Catalyst VLAN, then the BDI is analogous. In the EVC/EFP configuration model (as opposed to the archaic switchport model), a bridge-domain is the internal multi-point "VLAN" that bridges service-instances together. A bridge-domain interface (BDI) is then a layer-3 construct attached to the BD in the same way that an SVI is the layer-3 construct on a Catalyst VLAN.

Anyway... have you tried removing the GPON_MGMT_ALLOW ACL from the g0/0/0 to see if that is the issue? Appears to me that maybe that ACL might be denying the subnet in use by the GPON stick's management interface (10.0.5.0/24). If that turns out to be the case, you might edit it to permit the 10.0.5.0 and apply it to the BDI rather than the physical interface.

[firestorm-v1]

Hello Ramblin Tech!  I tried blowing away the ACL altogether on Gi0/0/0 and it had no effect.  I then added 10.0.5.0/24 to the ACL and added it to the BDI interface as recommended, but it didn't change anything with respect to the GPON stick access. Then I got to thinking about how to prove this works and I decided to test things deeper.

I added a bridge-domain to Gi0/0/2 using the syntaxes from Gi0/0/0 and Gi0/0/1 and then connected a laptop with a static IP configured for 10.0.5.x to it and lo and behold the GPON stick started pinging and responding to http.  I then went to the downstream switch and added an access port to VLAN50 and moved the laptop there and re-ran the tests.  I still had full connectivity to the GPON stick.

At this point, I can rule out the ASR config between Gi0/0/0 and the BDI as when the BDI was extended to Gi0/0/2, I had full access to the stick.  I can also rule out the trunk configuration to the switch as I was able to ping and access http on the stick there. 

To recap (because I hate those threads that end with "Fixed it!" with no details:

- ISSUE 1:  ASR logging shows errors about punted packets unknown service type 73
   Cause:  This is due to the switch sending BPDUs to the ASR.
   Resolution:  Enable "spanning-tree portfast edge trunk" and "spanning-tree bpdufilter enable"

- ISSUE 2:  Can't ping across the BDI interface to a host on another port.
   Determination:  Better testing would have revealed that the BDI was indeed working correctly.  For the next time, configured an unconfigured interface to join the existing BDI and test that access works across the device.  Once this is confirmed, test again using an access port on the downstream device.  If both tests pass, then the fault with access lies in another device (probably the firewalls).  

Final conclusion:  My princess is in another castle.  Take a look at the firewalls instead.

Thank you both @Ramblin Tech and @Richard Burts for your help!  I really appreciate your efforts, and now I at least know how BDIs work. 

[Richard Burts]

You are quite welcome. This has been an interesting discussion. I hope to see you continue to be active in the community.

[firestorm-v1]

Hello Richard!  Thank you for the reply, I admit I'll definitely be here more often as I work on more complex scenarios.  Thank you for the encouragement (and I'll try and make better initial posts).

Well, I'm back at it and I'm really not sure what's going on here.

• Using a Linux laptop attached to the ASR, I can ping/access the stick using a static 10.0.5.x  IP and a DHCP address from the firewalls pulled through the ASR (discover, offer, request, ack) from the firewalls.
• Using a Linux laptop attached to the 2960X access port for VLAN50, I can ping/access the stick using a static 10.0.5.x IP and a DHCP address.
• Using a Linux laptop attached to a Cisco 4500-X (core switch) and an access port for VLAN50, I cannot ping the stick with a DHCP or static IP address.  This may be due to the Linux laptop only has a 100mbit NIC and the 4500-X is only 1G/10G (but it pulls a successful IP, wtf?)
• Using a Windows laptop attached to the ASR, I can pull DHCP or use a static IP address, but I cannot ping the stick.
• Using a Windows laptop attached to the 2960X on the same access port as before, I can pull DHCP or use a static IP address, but I cannot ping the stick.
• Using a Windows laptop attached to the 4500-X on the same access port as before, I can pull DHCP or use a static IP address, but I cannot ping the stick.

Now for the real weirdness.   The trunk between the rack switch and the 4500 is identical on both ends, the firewalls can ping the VLAN50 DHCP address on the rack switch and the BDI interface on the ASR.  The BDI cannot ping the stick, even though a directly attached laptop can.  I'd say that this is a Layer 2 issue between the 4500 and the 2960-X but if that were the case, wouldn't DHCP fail on the 2960-X and the ASR.  The 2960-x VLAN50 interface successfully pulled DHCP through the 4500, the ASR also pulled a DHCP address from the firewalls for the BDI virtual interface. 

I have no idea what's going on, this is breaking my brain.

• Why can't the BDI ping the stick even though the Linux laptop can?
• Why can't the 2960X ping the stick even though we've established that the Linux laptop can pull DHCP and ping the stick through it?
• Why can't the 4500-X ping the stick when we've already established that the BDI on the ASR and the VLAN50 interface on on the 2960-X both can pull DHCP?
• Why doesn't the stick have an ARP anywhere but is reachable on the 2960-X's access port and the ASR's third interface?

[Richard Burts]

I agree that this is strange. I am especially puzzled by what appears to be different behavior of Linux laptop and Windows. Can you verify that the IP address, mask, and gateway used by both agree with each other (same subnet, same gateway)?

[firestorm-v1]

Hello Richard:

Yep, that's the weird part.  VLAN50 is 10.0.5.x /24 with a gateway of 10.0.5.1 and I use the same settings whether DHCP or static.   The only devices that provide DHCP on VLAN50 is the two firewalls.  I also managed to discover why my Linux laptop couldn't get an IP address on the VLAN50 test port on the 4500-X, it was due to the fact that the USB adapter was 100mbit.  While "sh int status" showed me it linked at 100mbit, unbeknownst to me at the time, it wasn't passing traffic at all.  Swapping it out for a gigabit USB adapter has fixed that issue and demonstrated I can pull DHCP with both laptops (the Windows laptop has a built-in gigabit NIC).

I decided to move the ASR from the rack switch to the 4500-X (since it is the core switch) and configured the switchport identically to the rack switch.  VALN900 continues to work as expected and I still have Internet access through the 4500/ASR/rack switch.  What's still weird is that even though I can pull DHCP or use a static IP address on the 4500-X test port, I still can't ping the stick!

I'm tempted to start a thread in the switching category as I don't think this is an issue with routing (or the ASR for that matter).  It's specifically to do with this 4500-X.

[MHM Cisco World]

interface receive untag config 
encap untag 

interface receive tag 
encap tag 
rewrite pop 1 symmetric

the rewrite will remove tag and send traffic to untag interface and  return traffic rewrite will add tag 

try above 

and note use service instance for both 

 MHM

[firestorm-v1]

Hello MHM Cisco World:

I'm not sure I understand.  On the ASR, the service instances are all untagged.  The 4500-X switchport that the ASR is connected to is set with a native VLAN 50 so it too should be untagged.  I can ping the BDI on the ASR from the 4500-X. 

The 4500's ports are configured as follows:

interface TenGigabitEthernet1/13
 description ASR-1000 - Gi0/0/1
 switchport trunk allowed vlan 50,900
 switchport trunk native vlan 50
 switchport mode trunk
 spanning-tree portfast edge trunk
 spanning-tree bpdufilter enable
!
interface TenGigabitEthernet1/14
 description VLAN 50 TEST PORT
 switchport access vlan 50
 switchport mode access
 spanning-tree portfast edge
!

The relevant configs on the ASR:

interface GigabitEthernet0/0/0
 description GPON_ONT_STICK
 mac-address xxyy.zzaa.bbcc
 no ip address
 negotiation auto
 cdp enable
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface GigabitEthernet0/0/1
 description INTERNAL_VLANS
 no ip address
 negotiation auto
 service instance 50 ethernet
  encapsulation untagged
  bridge-domain 50
 !
!
interface BDI50
 vrf forwarding INTERNAL_MGMT
 ip address dhcp
!

I tried having VLAN50 tagged on my rack switch and it didn't work (or I thought it didn't work) so I went back to basics and switched VLAN50 to untagged on the port where the ASR connects.  

[firestorm-v1]

After many many more labs and testing with a basic media converter (2.5g SFP to Copper Ethernet), it appears that the stick just drops the management interface once a successful link with the PON has been established.  While I'm the first to encounter this "error", the fact that I've been able to reproduce it reliably after several attempts does seem to support the theory.  While I'd rather be monitoring the GPON stick for fiber statistics, I'll have to settle on monitoring the interface's link state for a rudimentary "is there light on the fiber" status.

While I don't entirely get what I want (I want access to the stick for monitoring, and I want the stick to pass Internet traffic), the minimum has been reached and the stick is indeed passing Internet traffic to the ASR.  I'm going to call it "good enough" unless I get some kind of breakthrough.

Thank you all!

[Richard Burts]

Thanks for the update. It does sound like you have correctly identified the behavior. It is too bad that you have not achieved all that you wanted (and it sounds quite reasonable) but I agree that sometimes we just need to settle for good enough.

[firestorm-v1]

Hello Richard:

Well, I'm not really sure what to think, but after a quick change, absolutely everything is working now.  Based on this new setup, I'm thinking that the stick isn't completely compatible with Cisco (definitely not compatible on the 4500, only partially compatible with the ASR).  All I did was grab a simple media converter and a copper 1G SFP and put the stick in the media converter with a short jumper to connect it to the ASR.  The management interface pings and is reachable, the traffic VLAN is still routing.

It's so weird how this manifested, but with everything working as was originally intended, I'm not gonna touch it again (unless I have to).  I've already got the config backed up in case I need to reload the ASR, but for the time being all is good.

This has been a wild ride to say the least.  I've learned a whole lot on the way (one of the reasons why I even went down this path of replacing the ISP's CPE with the GPON stick) and I now know how ACLs and VRFs work on a basic level.  I'm still pretty new at the routing portion, but there's a first time for everything.

Thank you to everyone that contributed!  

[Richard Burts]

Thanks for the update. Glad to know that it is now working. I hope you continue to learn about networking and continue to participate in the community.