QinQ 4500X -> ME3600 and access(pop) multiple inner vlans

[johnelliot6]

Hi Everyone,


We have an AGG port(Standard trunk port) to a carrier on a 4500X - Port has multiple customer vlans for p-t-p eth services.

A service they have released will allow us to connect to azure/office 365 via QinQ(Carrier doing QinQ, not us) - i.e. We agree to an outer vlan tag with the carrier, and they create QinQ tunnel to azure/office 365...then multiple inner vlan tags are agreed to between us/azure for various services over this QinQ tunnel.

My question is this:

With our current setup (i.e. 4500X, standard dot1q trunk), we would just tag the outer vlan for the carrier to use for the QinQ tunnel to azure...this is fine, but for us to be able to "access" the inner vlans, Im hoping we can trunk this outer vlan to an ME3600, and then pop each inner vlan, and use them as needed.....Is this possible? ie will the "inner" tags be maintained going through the 4500X, and if so, if someone could point me in the direction of ME3600 docco that details how to pop the individual inner vlans, it would be greatly appreciated.

Eg....Ive tried the following, but Im not 100% if this would be the "correct" way to accomplish our goal (i.e. access the inner tag on the ME3600, and bring it up in a vlan int)

A test service was created by carrier, and MS (I cant be 100% certain that the Carrier/MS/Azure section is correct/complete.....they tell me it is :P)

Vlans are:

940 (outer)
941 (Inner)

Both vlans have been created on the ME, and only vlan 940 on the 4500X that connects to carrier:

ME3600 conf

interface GigabitEthernet0/24   <- Connects to 4500X
service instance 940 ethernet
  description description Inner_outer_tag_test_Outer_940_Inner_941
  encapsulation dot1q 940 second-dot1q 941
  rewrite ingress tag pop 2 symmetric
  bridge-domain 941

interface Vlan941
 description INNER_OUTER_TAG_TEST
 mtu 9100
 ip address xxx.xxx.xxx.xxx 255.255.255.252
 no ip proxy-arp


Im unable to ping remote end, nor am I seeing any dynamic Macs for bridge domain 941 - Is there any additional commands I can run to "see" if we are indeed receiving the Outer and Inner Tags on the ME? 

The only MAC I am learning on the 4500X is from the ME3600

#sh mac address-table dynamic vlan 940
Unicast Entries
 vlan     mac address     type        protocols               port
---------+---------------+--------+---------------------+-------------------------
 940      3462.882a.4640   dynamic ip,ipx,assigned,other TenGigabitEthernet1/1/3   

Thanks in advance.

[Philip D'Ath]

I don't think you can do this on a 4500-X at all, since I don't think it supports QinQ frames.  I could be wrong.

The ME3600 does however,  If you plug your carrier circuit into this, then you could use " encapsulation dot1q 940 second-dot1q 941".

I'm not sure on an ME3600, but on a router you can also do:

interface gigabit 0/1.941
encapsulation dot1q 940 second-dot1q 941
ip address ...

interface gigabit 0/1.942
encapsulation dot1q 940 second-dot1q 942
ip address ...

[johnelliot6]

Thanks for the reply,

4500X doesnt support QinQ frames?  Seriously?  Low end Cisco switches support QinQ (The old 3560's do?)

Or are you saying that the 4500X will "mangle" the frame, so we would lose the inner tag once it gets to the ME3600?

Cheers

[Philip D'Ath]

Cisco 3560's don't support QinQ.  Let me be specific, I mean double tagged frames - more specifically a frame with a VLAN tag immediately followed by another VLAN tag - these are not the same as standard Ethernet frames, so you have to have special hardware support.

Cisco 3560's, and to the best of my knowledge 4500-X, support only singe tagged frames.

My guess (and this really is a guess - never tested it) is that a Cisco 3560 and a 4500-X will see a double tagged frame as malformed and drop it.

I could be wrong.  I'm about 80% confident.

Double tagged frames is a common metro Ethernet service, where it is common to transport a customer's network inside of another tagged service provider frame.

You could ask your provider to de-tag the frame for you, so you only see the "customer" vlan tags.

[Philip D'Ath]

And the fact that you can not see any MACs for the remote end supports the idea that the 4500-X would be dropping the frames even more.

[johnelliot6]

Hi,

Just an update to this - Carrier was seeing traffic originating from us, but no return traffic from Azure/MS....investigating whether there end was configured correctly was taking a very long time, so we created another test service (Same outer vlan, new inner - 940(Outer), 942(Inner)...still going through the 4500X, then to ME3600 - Well, this one worked immediately with my original conf, so it seems the 4500X happily forwards the double tagged frame(i.e. doesnt drop it)...so working config is below, for anyone that comes across this scenario..

service instance 940 ethernet
  description description TEST_OUTER_940_Inner_942
  encapsulation dot1q 940 second-dot1q 942
  rewrite ingress tag pop 2 symmetric
  bridge-domain 942

interface Vlan942
 description TEST_OUTER_INNER_TAGs
 mtu 9100
 ip address 10.97.97.1 255.255.255.252
 no ip proxy-arp
end


#ping 10.97.97.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.97.97.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

#sh mac address-table dyn bridge-domain 942
          Mac Address Table
-------------------------------------------

BD      Mac Address       Type        Ports
----    -----------       --------    -----
 942    5087.89fd.d320    DYNAMIC     Gi0/24+Efp940 

[Giuseppe Larosa]

Hello John,

good to know that the original approach worked. You have been kind to provide an happy end to this thread. This makes it more useful for other people that may have the same issue in the future.

For this reason I have rated your post to signal a solution was found.

Best Regards

Giuseppe

[johnelliot6]

Wow....that is really really annoying if true....I was hoping the 4500X would simply pass the frame to the ME3600 "unchanged", but from what you are saying, this doesnt appear to be the case....Our problem is, that the "AGG" port connected to the 4500X has multiple other customer vlans, for simple p-t-p eth services...so moving it to the ME3600 is going to be difficult...aside from the fact that we have no 10Gb ports on the ME's, only on the 4500X's....this AGG is 10G....ahh, not what I wanted to hear, but thanks for the info.

[Philip D'Ath]

Ask the service provider to de-tag frames for you then - you just need to be careful that the inner VLANs are unique when compared to what you currently use.

Another option.  Would it be feasible to get a second smaller access and plug that directly into the metro Ethernet switch?

[Philip D'Ath]

Going completely sideways; I don't think the ASR903's are two expensive, and you can get them with 10Gbe ports and I'm pretty sure they can do QinQ.

[Philip D'Ath]

If you check out the below Wikipedia article, and scroll down to frame format, you can see how the extra VLAN tags push the packet out.  They are no longer standard Ethernet frames.

https://en.wikipedia.org/wiki/IEEE_802.1ad

[Giuseppe Larosa]

Hello Philip,

there is nothing that makes an 802.1Q in 802.1Q frame different from a normal single tagged frame except the added 4 bytes of overhead. The original poster can be facing an MTU issue with services.

The old way to access the inner Vlan tag in a double tagged frame is to have a tunnel dot1q port that pops the external Vlan tag and make available to a device downstream the inner vlan.

Looking at the configuration guide we see that C4500-X supports 802.1Q tunneling

see

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/XE3-6-0E/15-22E/configuration/guide/xe-360-config/tunnel.html

Hope to help

Giuseppe

[Philip D'Ath]

The frame does look different, because the extra tag is inserted in front of the Ethertype field in the layer 2 frame.  This is then used to find the frame length.  If the device doesn't know that the frame is double tagged it will look at the wrong place in the frame to determine the type of that frame, and consequently the frame length.  If it can't get the correct frame length how can it be expected to forward that frame.

But I do like the dot1q tunnel trick.  That sounds like it would work to me.