LAN Issues with remote offices using Q n Q.

[Jose A]

Good Afternoon:

I was currently working with 1 main office with 1 remote office. On my main office I have a router on a stick connected to a switch.

This switch then is connected to remote office (1) using Q n Q provided by my ISP.

Everything was working as expected, until we yesterday added a new remote office (2), using Q n Q provided by the same ISP provider of remote office 1.

Now from the remote switch or router I cannot see devices on the same Vlan (Management). But from other Vlans they are reachable.

all the switches are 2960X

Management Vlan is 4.

Router ip: 10.10.4.1 (Default Gateway for all switches)

main switch ip: 10.10.4.11

ups on main office: 10.10.4.5

remote office 1 ip: 10.10.4.12

ups on remote office 1: 10.10.4.6

remote office 2 ip: 10.10.4.13

ups on remote office 2: 10.10.4.7 

Your help is more than appreciated.

[Francesco Molino]

Hi

Can you share please your configs in text files?

And can you share outputs of show spanning-tree, show arp and show mac ?

You said you have issues on management vlan but not others. Q-in-Q provided by your ISP is for which vlan?

[Jose A]

Hi Franceso, thank you for your time. 

Attached you will find the config files as requested.

hq-sw = switch located at main office

ro1-sw= switch located at remote office 1

ro2-sw= switch located at remote office 2

By now I´had only identified the problems with vlan 4 which i´m trying to keep at a layer 2 network. 

Thanks again.

Thanks again. 

[Francesco Molino]

Thanks I'll take a look later. Can you give more details about interface that are facing your Q-in-Q provider and which vlans are part of this Q-in-Q service

[Jose A]

Hi Franceso: 

The interfaces are configured as a trunk, with a not the default native (vlan 1). The vlans that I´m carrying throw the Q in Q service are for: 

vlan 3: CCTV Vlan

vlan 4: Management Vlan

vlan 6: Clock devices Vlan

vlan 7: IT Staff Vlan

vlan 9: Manager Vlan

vlan 10: Voice Vlan

and a data vlan for each specific remote office (In this case vlan 12 or 13).

Interfaces facing the Q in Q Services are: 

hq switch: g1/0/24 to ro1-sw: g1/0/24

hq switch: g1/0/22 to ro2-sw: g1/0/24

ro1 = Remote Office 1

ro2 = Remote Office 2

If you need any more information, please let me know. 

Thanks.

[Francesco Molino]

Please detail a bit your issue exactly what are you facing?

I'm saying because everything seems fine (Q-in-Q is done by your ISP) and I see mac-addresses in vlan 4 and others learned on all 3 switches.

Can you try pinging IP 10.10.4.7 from all 3 switches?

[Jose A]

Hi Francesco, thanks again for your time.

That’s the problem I cannot ping all the devices from all the locations. For example 10.10.4.7 can only be ping from the switch where the device is connected to (remote office 2).

I was able to replicate this issue by adding vlan 3 to a single port on every switch.

I then removed the vlan from one of the remote offices switches and the problem went away.

Hope this info help.

[Francesco Molino]

Ok thanks for the clarification.

Why do you have 2 interfaces from your HO office?

I mean, do you have a design how all sites are interconnected?

What your explaining looks like a stp issue.

When the vlan is on all sites, and you try to ping from HO to office 2 (if we take the IP of previous post), do see the icmp packet arriving on remote site? Can you run a Wireshark on all sites while trying to ping to see if we see all packets?

Do you know if isp is filtering stp, cdp on Q-in-Q tunnel?

[Jose A]

Hi Franceso below my comments.

Why do you have 2 interfaces from your HO office?

• The idea was to be prepared in case we needed a second router on the HQ office. I had shutdown the trunk port g1/0/27 that goes from router to hq switch.

I mean, do you have a design how all sites are interconnected?

• No at the moment.

What your explaining looks like a stp issue.

• Do you wish me to run any other command to review other outputs?

When the vlan is on all sites, and you try to ping from HO to office 2 (if we take the IP of previous post), do see the icmp packet arriving on remote site? Can you run a Wireshark on all sites while trying to ping to see if we see all packets?

• Yesterday while making the test with vlan 3 (from remote office 2) I ran a wireshark, I will attach in case you want to check it.

On vlan 3 I had three devices:

On hq = 10.10.3.254

Remote office 1: 10.10.3.254

Remote office 2: 10.10.3.2

Do you know if isp is filtering stp, cdp on Q-in-Q tunnel?

According to them they do not run any type of filtering on their circuit.

[Francesco Molino]

This means that g1/0/22 is now shutdown and all traffic is passing through g1/0/24 for all remote sites?

The Wireshark you took are on all switches from all location for vlan 3? If so yes i would to review them.

You said that all vlans are working except vlan 4, right?

If ISP was allowing stp, cdp, on remote sites you should have seen HO switch as root bridge and here this isn't the case.

Let's take vlan 4. Each site is connected using 1 port to isp. When vlan 4 is available on all sites, nothing works but when you remove it from 1 site, then everything is back up, is that right?

Can you modify spanning-tree priority for vlan 4 on HO as 4096? Then issue show spanning-tree vlan 4 on remote offices and share the output please.

If everything is connected with 1 link you shouldn't have any stp issue and based on outputs this is a case because interfaces facing Q-in-Q cloud are in forwarding state.

For simplicity, can you have a svi for vlan 4 on all switches and run debug ip icmp on all switches. Now try to ping from any site all other vlan 4 svi and finally paste the output of debug ip icmp.

[Jose A]

Francesco, Hello again, my comments below.

This means that g1/0/22 is now shutdown and all traffic is passing through g1/0/24 for all remote sites?

• I meant one of the trunk ports to the router was shutdown. If you check the config file for hq_sw I had two ports trunking to the switch (int g1/0/26 and 27).

           Right now only 26 is up.  

The Wireshark you took are on all switches from all location for vlan 3? If so yes i would to review them.

• I only was able to run wireshark from remote office 2

You said that all vlans are working except vlan 4, right?

• All vlans are working properly if they are kept only in 1 or two switches. As soon you add the vlan to a third switch the issues with that particular vlan starts.

If ISP was allowing stp, cdp, on remote sites you should have seen HO switch as root bridge and here this isn't the case.

• I guess so. But they insisted they are not blocking anything.

Let's take vlan 4. Each site is connected using 1 port to isp. When vlan 4 is available on all sites, nothing works but when you remove it from 1 site, then everything is back up, is that right?

• That’s right. It happens with vlan 4 right now because is the only vlan with atleast 1 port assigned on every switch. ( Remember I have some UPS´s connected on each site on vlan 4). 10.10.4.5 / 10.10.4.6 / 10.10.4.7

Can you modify spanning-tree priority for vlan 4 on HO as 4096? Then issue show spanning-tree vlan 4 on remote offices and share the output please.

• Output on HQ switch:

hq-sw(config)#spanning-tree vlan 4 priority 4096

hq-sw(config)#

hq-sw#show spanning-tree vlan 4

VLAN0004

  Spanning tree enabled protocol rstp

  Root ID    Priority    4100

             Address     009a.d227.7a80

             This bridge is the root

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    4100   (priority 4096 sys-id-ext 4)

             Address     009a.d227.7a80

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/16            Desg FWD 4         128.16   P2p Edge

Gi1/0/21            Desg FWD 19        128.21   P2p Edge

Gi1/0/22            Desg FWD 19        128.22   P2p

Gi1/0/24            Desg FWD 19        128.24   P2p

Gi1/0/26            Desg FWD 4         128.26   P2p

hq-sw#

Output on remote site 1 switch:

r01-sw#show spanning-tree vlan 4

VLAN0004

  Spanning tree enabled protocol rstp

  Root ID    Priority    32772

             Address     00c1.b1de.dc80

             This bridge is the root

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32772  (priority 32768 sys-id-ext 4)

             Address     00c1.b1de.dc80

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/13            Desg FWD 19        128.13   P2p Edge

Gi1/0/24            Desg FWD 19        128.24   P2p

r01-sw#

Output on remote office 2:

r02-sw#show spanning-tree vlan 4

VLAN0004

  Spanning tree enabled protocol rstp

  Root ID    Priority    32772

             Address     00c1.b1d7.9100

             This bridge is the root

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32772  (priority 32768 sys-id-ext 4)

             Address     00c1.b1d7.9100

             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/13            Desg FWD 19        128.13   P2p Edge

Gi1/0/24            Desg FWD 19        128.24   P2p

r02-sw#

*Note: on both remote sites the UPS are plugged on interface g1/0/13

If everything is connected with 1 link you shouldn't have any stp issue and based on outputs this is a case because interfaces facing Q-in-Q cloud are in forwarding state.

For simplicity, can you have a svi for vlan 4 on all switches and run debug ip icmp on all switches. Now try to ping from any site all other vlan 4 svi and finally paste the output of debug ip icmp.

• I believe I already have svi created on all the switches for vlan 4

hq-sw#sh ip interface brief | i Vlan

Vlan1                  unassigned      YES NVRAM  administratively down down

Vlan4                  10.10.4.11      YES NVRAM  up                    up

hq-sw#

ro1-sw#sh ip interface brief | i Vlan

Vlan1                  unassigned      YES NVRAM  administratively down down

Vlan4                  10.10.4.12      YES NVRAM  up                    up

ro1-sw#

ro2-sw#sh ip interface brief | i Vlan

Vlan1                  unassigned      YES NVRAM  administratively down down

Vlan4                  10.10.4.13      YES NVRAM  up                    up

ro2-sw#

Thanks one more time for all the help!

[Francesco Molino]

I will wait for your wireshark and try to reproduce it (I'll try to do it this weekend).

If you ping your svi vlan 4 and run the debug ip ICMP, what are the outputs?

[Jose A]

attached is the wireshark. Remember i simulated by assigning the vlan 3 to a port on all (3) switches. 

Thanks!

[Francesco Molino]

Hi

Your wireshark isn't for all sites and you still not attached the debug ip icmp and wireshark of all sites while pinging from HO both remote sites.

I'm not at the office right now but I've done a lab to show you on Cisco Virl.

This is the design I built (can you share yours to be sure we're on the same page). All switches named iosvl are ISP switches and your switches are HO, ro1 and ro2.

I copy pasted your port config and vlans and everything is working as expected. All vlans are reachable.

For dot1q, I used vlan 99. Here config output of ISP switches g0/1 facing your switches:

interface GigabitEthernet0/1
switchport access vlan 99
switchport mode dot1q-tunnel

And output of wireshark, you can see the 2 dot1q tags. You won't be able to do the same capture on your side because it has to be done on ISP switches:

At the same time, in this capture you can see that 10.10.7.1 (ho vla 7) can ping .2 and .3 (ro1 and ro2 vlan 7).

Your config looks like good and maybe it's an issue with your ISP but we need to get all infos before saying that (that's why I'm asking for debug and wireshark on all site).

Do you know which vlan your ISP is using for q-in-q?

[Jose A]

Hi Franceso:

Thanks again for your interest on this strange case.

Here you will find the how the network equipment’s are connected.

Regarding the question of the vlan of the ISP, they are using two vlans. 1 for each remote office.

HQ to Remote Office 1 = vlan 3685

HQ to Remote Office 2 = vlan 31xx (I don’t remember)

In theory all of these vlans should be transparent to me right?

Attached you will find the outputs from the debug ip icmp.