Q-in-Q problems

[igor.hamzic]

Hi all. My service provider has just brought up a q-in-q link between my new and old locations because I need to expand our current VLANs to the new location. One one side of the link I have a 6500 switch with a trunk interface to the new loaction and on the other side I have a stacked 3750 switch with a trunk interface to the old location. I can see VLANs on the new locaion, ping servers in different VLANs on the old location, go to internet etc..

The problem occurs when I try to access some of the resources in different VLANs on the old location. For example I try to use a program that most users use for creating a newspaper which connects to the server in the old location and midway through loading it just stops or I try to access my link monitoring software and the web page doesn't open or I try to access my ACS and the web page just keeps loading. On the other hand mail connects without a glitch to the server for example.

I have used wireshark on the problematic connections to the servers and I can see that there are some segments missing and that either my PC or the server on the other side closes the connection after a few missed segments.

My provider has checked the link and they can't see any problems. Does anybody have any clue what could be happening with the link?

[igor.hamzic]

Today we have tested the link further and narrowed the problem even more. The problem exists only with TCP and when TCP traffic exeeds speed of 14449 bytes/sec. When traffic going over the link exeeds this speed the TCP communication falls apart and you don't see any regular TCP communication except when the connection closes with the usual sequence or the connection is reset. When this happens we can see servers form the old location trying to send ACK to the new location but on the new loaction we can't see those ACK getting over the link.

With UDP and ICMP protocols we have no problems when traffic goes over the link. In fact with UDP we have managed to use almost the entire 100 Mbit link between locations.

It seems that while the TCP traffic is small and doesn't use a lot of bandwidth everything is fine but when it hits the magic limit TCP falls apart and we can't see any TCP communication between the sites. My coworker has also suggested that the problem might be with window size.

Does anybody know if anything on Cisco equipment(6500 and 3750 switches) could cause something like this or should I go the provider with this information?

[gtosh3500]

You might check the MTU capabilities on the link. A lower MTU on the provider link would result in larger packets being dropped, which would explain why some applications such as mail with relatively smaller packets, work and others do not. I've run into this problem myself a few times, especially with Metro E over SONET.

You might try pinging across the link and increase the size of the ping packet to see if this is your issue. If it turns out this is the issue you can use the ip tcp adjust-mss to control the TCP windowing.

Sent from Cisco Technical Support iPad App

[igor.hamzic]

The problem is solved. Whatever the problem was it was on the providers side. They didn't disclose what the problem was exactly and honestly I don't care because the link is finally working. Thanks for the help none the less.