How can VLAN inside of QinQ interacts with "outside" world?

Konstantin Dunaev · ‎01-13-2011

Hello,

we have a simple QinQ topology:

PE-SW1-----------QinQ-------PE-SW2-----V652-----PE-UPLINK

| <-----QinQ edge ------------->| |

| <---Trunk allowed V652 -->| | <-- Trunk allowed V652

CE-SW1 CE-SW2

| <-------------V652--------------> |

HOST1 HOST2

PE-SW2 and CE-SW2 are connected with 2 physical interfaces, first one is QinQ edge port, the second is normal trunk, Vlan 652 is allowed on both ports. The HOST1 and HOST2 are placed in VLAN652 and can communicate with each other via QinQ link, HOST2 can communicate with PE-UPLINK via the Trunk between CE-SW2 and PE-SW2.

But HOST1 Can't communicate with PE-UPLINK.

What could be the problem here? Is it not allowed to use the same VLAN inside of QinQ and outside of?

rsimoni · ‎01-15-2011

Hi Kostantin,

I don't understand the topology.

Can you attach a proper drawing instead?

Riccardo

Konstantin Dunaev · ‎01-16-2011

Hello,

sorry, I've update my posting with the drawing.

rsimoni · ‎01-17-2011

Hi Konstantin,

I have an idea of what is happening but I would like to see the port configuration of Pe-Sw1 and Pe-sw2 first (all the interconnecting ports including the one to the uplink router).

Also, on ce-sw2 the port where host2 is connected to is an access port in vlan 652, isn't it?

Riccardo

Konstantin Dunaev · ‎01-18-2011

Hi Riccardo,

I've tried to reproduce the error in my test lab, and reviewed the topology, it was changed a little bit, I've uploaded the current version.

In the Topology you can see the Backbone switch - "PE-UPLINK" which is connected to both PE switches, where QinQ termination is taking the place.

PE-UPLINK has an interface in Vlna652 with IP 10.10.10.1 which should be used by HOST1 and HOST2 as default gateway, but only HOST2 can reach it. And HOST1 can reach HOST2 and other way around.

If I connect the additional device to the PE-UPLINK on the VLAN652, e.g. the router with IP 10.10.10.254 then both HOST'S can reach 10.10.10.254.

PE-UPLINK#         
interface FastEthernet0/31
 switchport trunk encapsulation dot1q
 switchport mode trunk
 no ip address

interface FastEthernet0/47
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 652
 switchport mode trunk
 no ip address

interface FastEthernet0/48
 switchport trunk encapsulation dot1q
 switchport mode trunk
 no ip address

interface Vlan652
 ip address 10.10.10.1 255.255.255.0
end

PE-SW1#
interface FastEthernet0/1
switchport access vlan 3298
switchport mode dot1q-tunnel
l2protocol-tunnel cdp
l2protocol-tunnel stp
l2protocol-tunnel vtp
no cdp enable

interface FastEthernet0/48
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 3298
switchport mode trunk
end

PE-SW2#
interface FastEthernet0/1
switchport access vlan 3298
switchport mode dot1q-tunnel
l2protocol-tunnel cdp
l2protocol-tunnel stp
l2protocol-tunnel vtp
no cdp enable
spanning-tree bpdufilter enable

interface FastEthernet0/3
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 652
switchport mode trunk

interface FastEthernet0/5
switchport trunk encapsulation dot1q
switchport mode trunk
end

CE-SW1#
interface FastEthernet0/1
 switchport mode trunk



CE-SW2#
interface FastEthernet0/1
 switchport mode trunk

interface FastEthernet0/2
  switchport trunk allowed vlan 1,652,1002-1005
 switchport mode trunk

As I understood the problem comes somewhere from MAC-table, but where exactly is not clear to me yet, it would be great if sombody could explain it to me.

rsimoni · ‎01-18-2011

Hi Konstantin,

you changed the topology to make things more complicated I guess

Anyway, the issue is indeed due to MAC address not populated in the vlan you expect.

The cause is that you cannot use the same vlan as a tunnel port and an access port (that was the initial case).

On the second topology you have something similar on PE-SW2 which get the same frames in 2 different vlans (652 and 3298). When a frame is received on Fa0/1 it will pop a second tag (3298 now) to it (mac learning will happen in vlan 3298); if it is received on Fa0/3 no extra tag will be popped and learning will happen in vlan 652.

So the answer is that what you are doing is not meant to work and configured.

If we want to know why you reach certain hosts and not others we need to check the mac table on all switches and see what is the used exit interface.

I can give you an help in that sense but I need the following for all switches.

First I need IP to MAC mapping for 10.10.10.1, 10.10.10.254 and the 2 hosts.

Then from all switches (including CE switches).

show mac-add add

we will then check where the given mac addresses are pointing to and we will understand things better.

Riccardo

Konstantin Dunaev · ‎01-18-2011

Hi Riccardo,

thank you for answer!

I've updated to topology because now it looks exactly like in real world, the problem comes first up because until now those "two" hosts didn't need the uplink.

ok now the ARP table:

Internet 10.10.10.1 0 000f.9059.0d00 ARPA Vlan652 - PE-UPLINK
Internet 10.10.10.142 0 0012.7f2b.3e80 ARPA Vlan652 - HOST1 (in my case it's CE-SW1)

Internet 10.10.10.140 15 0006.525e.0800 ARPA Vlan652 - HOST2 (in my case it's CE-SW2)
Internet 10.10.10.254 0 68ef.bd6d.198e ARPA Vlan652 - external router connected to PE-UPLINK switch

CE-SW1#sh mac-address-table

652 0006.525e.0800 DYNAMIC Fa0/1

652 0006.525e.0801 DYNAMIC Fa0/1

652 000f.9059.0d00 DYNAMIC Fa0/1

652 0012.7f10.a300 DYNAMIC Fa0/1

652 68ef.bd6d.198e DYNAMIC Fa0/1

CE-SW2#sh mac-address-table

652    000f.9059.0d00    DYNAMIC    Fa0/2
652    0012.7f10.a300    DYNAMIC    Fa0/2
652    0012.7f10.a303    DYNAMIC    Fa0/2
652    0012.7f2b.3e80    DYNAMIC    Fa0/1
652    68ef.bd6d.198e    DYNAMIC    Fa0/2

PE-SW1#sh mac-address-table dynamic

3298    0006.525e.0800    DYNAMIC     Fa0/48
3298    0006.525e.0801    DYNAMIC     Fa0/48
3298    000f.9059.0d00    DYNAMIC     Fa0/48
3298    0012.7f10.a300    DYNAMIC     Fa0/48
3298    0012.7f2b.3e80    DYNAMIC     Fa0/1
3298    0012.7f2b.3e81    DYNAMIC     Fa0/1
3298    68ef.bd6d.198e    DYNAMIC     Fa0/48

PE-SW2#sh mac-address-table dynamic 
 3298    0006.525e.0800    DYNAMIC     Fa0/1
3298    0006.525e.0801    DYNAMIC     Fa0/1
3298    000f.9059.0d00    DYNAMIC     Fa0/1
3298    000f.9059.0d1f    DYNAMIC     Fa0/5
3298    0012.7f10.a300    DYNAMIC     Fa0/1
3298    0012.7f2b.3e80    DYNAMIC     Fa0/5
3298    0012.7f2b.3e81    DYNAMIC     Fa0/5
3298    68ef.bd6d.198e    DYNAMIC     Fa0/1
 652    0006.525e.0800    DYNAMIC     Fa0/3
 652    000f.9059.0d00    DYNAMIC     Fa0/5
 652    000f.9059.0d1f    DYNAMIC     Fa0/5
 652    0012.7f2b.3e80    DYNAMIC     Fa0/3
 652    68ef.bd6d.198e    DYNAMIC     Fa0/5

PE-UPLINK#sh mac-address-table dynamic 
   652    0006.525e.0800    DYNAMIC     Fa0/31
 652    0012.7f10.a300    DYNAMIC     Fa0/31
 652    0012.7f2b.3e80    DYNAMIC     Fa0/31
 652    68ef.bd6d.198e    DYNAMIC     Fa0/47
3298    0006.525e.0800    DYNAMIC     Fa0/31
3298    0006.525e.0801    DYNAMIC     Fa0/31
3298    000e.83b1.9330    DYNAMIC     Fa0/48
3298    000f.9059.0d00    DYNAMIC     Fa0/31
3298    0012.7f10.a300    DYNAMIC     Fa0/31
3298    0012.7f2b.3e80    DYNAMIC     Fa0/48
3298    0012.7f2b.3e81    DYNAMIC     Fa0/48
3298    68ef.bd6d.198e    DYNAMIC     Fa0/31

rsimoni · ‎01-18-2011

Kostantin,

by following the MAC addresses hop by hop you should also have connectivity from host1 to pw-uplink address in vlan652

the flow is

host1 to pe-uplink vlan 652

host1 - ce-sw1 vlan 652 - pe-sw1 vlan3298 -pe-uplink vlan3298 - pe-sw2 vlan 3298 - ce-sw2 vlan652 - pe-sw2 vlan652 - pe-uplink vlan652

pe-uplink vlan652 to host1

pe-uplink vlan652 - pe-sw2 vlan652 - ce-sw2 vlan652 - pe-sw2 vlan 3298 - pe-uplink vlan3298 - pe-sw1 vlan3298 - ce-sw1 vlan 652 - host1

so for both directions the tables are correctly populated.

In my opinion Pe-uplink can be confused by the fact that multiple svi's share the same mac address.

Try to assign a different mac address to SVI 652 on pe-uplink (0123.0123.cafe for instance) and try again (after clearing arp and mac tables everywhere)

If it does not work I would also try to connect real devices as host1 and host2 as well and try again as sometimes switches can be confused when frames are destined to mac they own.

Riccardo

Konstantin Dunaev · ‎01-18-2011

Hi Riccardo, thank you for your answer, you've pushed me to examine more carefully the tables on my switches.

hmm, something strange is happening, i tried to get all MAC again but now on

the PE-UPLINKS I have only following:

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
   1    000e.83b1.9330    DYNAMIC     Fa0/48
   1    0012.7f10.a305    DYNAMIC     Fa0/31
652    0006.525e.0800    DYNAMIC     Fa0/31
652    0012.7f10.a300    DYNAMIC     Fa0/31
652    0012.7f2b.3e80    DYNAMIC     Fa0/31
652    68ef.bd6d.198e    DYNAMIC     Fa0/47
3298    0006.525e.0800    DYNAMIC     Fa0/31
3298    0006.525e.0801    DYNAMIC     Fa0/31
3298    000e.83b1.9330    DYNAMIC     Fa0/48
3298    0012.7f2b.3e80    DYNAMIC     Fa0/48
3298    0012.7f2b.3e81    DYNAMIC     Fa0/48
3298    68ef.bd6d.198e    DYNAMIC     Fa0/31

there is no entry for MAC "000f.9059.0d00" in dynamic table.

entry "3298 000f.9059.0d00 DYNAMIC Fa0/31" comes into PE-UPLINK's mac table only if I ping broadcast IP address 10.10.10.255 from the PE-UPLINK, and I think this entry is not correct because PE-UPLINK simply may not forward the frame with a MAC address for it self - "000f.9059.0d00" over other interface, F0/31 in that case.

I'm using catalyst 3550 and it seems they don't support manuall MAC addresses on a VLAN interface and I can't chage MAC address for it.

rsimoni · ‎01-18-2011

Hi Konstantin,

"3298 000f.9059.0d00 DYNAMIC Fa0/31" just expired on PE-Uplink after 300 seconds (or after another value if you changed defaul mac-address aging timer).

It is correct as this switch does not consider it as a local MAC as it is learned from a port. Anyway the issue for host2 should be something related to local mac addresses.. In real scenarios you will not face the problem though as the switches will not switch traffic destined to their MAC addresses.

Bottom line, as I wrote before, the issue arose because you are mixing up SP vlans with Customer vlans, or to say it better you use on a SP device a vlan ID as a tunnel vlan and as a customer vlan. This is not permitted by qinq standards and what you see is just a drawback of this.

You can find a way to hack this, but you risk to face some issue that no vendor will solve (issues arising from not supported configurations are normally junked by vendors).

Riccardo

Konstantin Dunaev · ‎01-18-2011

Hmm, if I move the "default gateway IP" out of PE-UPLINK to the next router, would it be QinQ conform? Probably not, because I still mixing the SP and Customer VLANs on PE-UPLINK and PE-SW2 switches.

But what be the right way to provaide all sites, connected by "QinQ" with a single uplink, e.g. Internet access? to build the second link for each site is not an option in our case.

rsimoni · ‎01-19-2011

Hi Konstantin,

You might make your PE-UPLINK an SP switch with a tunnel port towards the IP gateway. In other words Fa0/47 becomes a tunnel port for SP vlan 3298.

Then you need to make sure that the router sends tagged traffic (in vlan 652) or else you risk that untagged traffic won't be encapsulated (that is the same problem for native vlans).

You might try to configure a dot1q trunk on it or add another switch before it with a trunk towards PE-UPLINK.

By doing that you would comply with the QinQ specs and at the same time you would provide an internet access to all hosts for vlan 3298.

Riccardo

Konstantin Dunaev · ‎01-21-2011

Hi Riccardo,

making the port f0/47 on the PE-UPLINK switch as QinQ edge is not possible as this post is a single physical connection to the IP gateway router.

If I change it then all other services which goes via that ports will be cut off the IP gateway. PE-UPLINK is just a backbone switch which forwards the traffic between different access switches and provide access to IP gateway.

I read somewhere that there is a possibility to install "hardware loop" on PE-UPLINK switch - just a crossover cable which connects 2 ports on PE-UPLINK, one of them is QinQ edge port (mode dot1q-tunnel) and other is a normal Trunk. Thus PE-UPLINK have access to all customer VLANs which need to be forwarded outside of QinQ . I didn't tests it yet

rsimoni · ‎01-21-2011

Hi Konstantin,

Ok I see.

I did not read any documentation about that, however in my opinion you might indeed use a loopback cable on PE-UPLINK to bypass your issue.

After all QinQ logic is not that complicated, it still relies on MAC learning on the outmost vlan tag. So if you create a tunnel port in vlan 3298 terminated on a trunk allowing vlan 652 you might get your goal.

The approach is not scalable as you will consume 3 ports per customer but if this is just one time solution I don't see why it should not work.

Let me know how it goes.

Riccardo