remote-span session blocks the traffic on the 3550/3560 switch

Konstantin Dunaev · ‎07-19-2011

Hello community!

I have some problem with a remote-span session on cisco 3550/3560.

Assuming we have 4 switches which are connected in a straight line like:

c3560-2 <--> c3560-1(gi0/28) <------------>(gi0/1) c3550-1<-->c3550-2

c3560-1 has a c7200-1 connected to the port Gi0/1

and

c3550-2 has a c7200-2 connected to the port Fa0/1

All links are Trunks.

I'd like to monitor the traffic between both c7200 (we see some strange packet loss and I can't identify the source of that problem).

c3560-2 has a remote-span session with a source Vlan555 (type remote-span) and destination port, let say gi0/24, Vlan 555 is allowed over all trunks.

If I configure the remote-span session on c3560-1 with source port Gi0/1 and destination Vlan555, then it's all perfect - I can see and monitor the traffic which c7200-1 sends and recieves.

but if I configure the remote-span on the switch c3550-1 with source on its uplink port Gi0/1 and destination Vlan555 then I loose the connection to the c3550-1 completly .

Does it mean that one may not configure the remote-span session with a source port where is allowed the remote-span Vlan ? I didn't see such a requirements anywhere in Docu.

Or is there othe problem?

Thank you!

andtoth · ‎07-20-2011

Hi Konstantin,

The limitation might be caused by the following:

The destination port cannot be a source port; a source port cannot be a destination port.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3550/software/release/12.2_44_se/configuration/guide/swspan.html

Additionally, you cannot have an RSPAN source session and an RSPAN destination session using the same rspan-vlan, on the same switch.

Did you try using the filter vlan for the SPAN configuration as mentioned in the above documentation to monitor specific vlans only?

Best regards,

Andras

Konstantin Dunaev · ‎07-21-2011

Hello Andras,

regarding RSPAN source/destination port, I understand this limitation so that the RSPAN session shouldn't use the port as destination or source, which is configured in other RSPAN session as source or destination, not so that I can't use an interface as source for RSPAN where simply the RSPAN Vlan is allowed.

In my topology I don't use RSPAN source/destination port on the same switch.

I tried the "rspan filter vlan" with a result that I loose the connection over this VLAN which I use in filter.

andtoth · ‎07-21-2011

Hi Konstantin,

Are you using a separate vlan for the RSPAN? Are you trying to access the 3550 switch on another vlan?

Best regards,

Andras

Konstantin Dunaev · ‎07-21-2011

yes sure, I'm using the separate vln for RSPAN session, ans that VLAN has a special type - "remote-span"

andtoth · ‎07-21-2011

Hi Konstantin,

I would suggest to check the switch from console connection when this is happening. Check STP states, Gi0/1 status, look at the logs, try pinging a neighboring device, etc.

Best regards,

Andras

Konstantin Dunaev · ‎08-23-2011

Hello again,

sorry for not been answering so long.

I tried to reproduce to topolgy in a test lab and of course didn't see any problems, but in real network is something very strange happens.

Away from the first problem I described, I did now the following see the picture down there.

"RSPAN destination" session is confgiured on sw-01-02, sorce VLAN 555, destination port Gi0/18

"RSPAN source" session is on sw-10-01- source int F0/48, destination VLna555 (reflector port is one of unused ports)

before I start the RSPAN source session on sw-10-01 I can reach IP 10.93.1.1 (R-10-01) without problem, but after activation the RSPAN I can see that only 50% packet are coming through and on the managemnet station in tcpdump I see only ICMP replies (those 50%) but no ICMP requests.

I've tried all possible RSPAN option whithout success. One thing to notice, on all switches there is a command "vlan dot1q tag native", in case it's important.

I'm lost and can't find any reason.

Peter Paluch · ‎08-23-2011

Hello Konstantin,

Sorry to point to towards two other threads without providing a full explanation, but do you believe your issue is similar to any of these two which have already been discussed here?

https://supportforums.cisco.com/message/3198475#3198475

https://supportforums.cisco.com/message/3202442#3202442

Best regards,

Peter

andtoth · ‎08-23-2011

It can indeed be that you are hitting the following defect:

CSCtj76880 RSPAN causes port to go into BKN state due to PVID inconsistency

This has been fixed in 12.2(55)SE and 12.2(58)SE.

Best regards,

Andras

Konstantin Dunaev · ‎08-23-2011

do you mean update could be the solution? but our problems don't fit 100% to those descrided in that bug.

andtoth · ‎08-23-2011

I would recommend to upgrade to the latest version on the 3560 switches. I see that you are already running the latest on 3550.

Andras

Konstantin Dunaev · ‎08-23-2011

but if I correctly understand, the source of problem is c3550, because I lost the connection to a vlan on c3550 where I star the RSPAN source session.

C3560 is only destination RSPAN and it works well if I start source RSPAN session on the neighbour c3560 - SW-01-01, then I can see and sniff all the traffic without lossing the connection.

Konstantin Dunaev · ‎08-25-2011

it's more funny!

if I disable the RSPAN VLAN 555 on the trun interface of sw-10-01 (where source interface for RSPAN session is configured) then I don't see of course any traffic on the sniffer but no packet loss!

I activated the RSPAN VLAN 55 on the trunk again, and run the "debug ip packets" on the router R-10-01 (which I'm testing with ping) and I see the following:

1. the ping from the router R-01-01 to 10.93.1.1 says "timeout"

2. sniffer host shows nothing, sometimes come the ARP's request from that VLAN 3326, but no ICMP packets.

3. the VLAN3326 interfac on SW-10-01 with an IP 10.93.1.12 can be reached without problem

4. the debug on the destination router R-10-01 says that it gets ALL packets and sends ALL answers!

it means that at least forward direction is working, but the backward direction has a problems, but why I don't see any ICMP request on the sniffer host?

It seems that the answer from R-10-01 doesn't go into the correct vlan 3326 but somewhere else. Sometimes I can see on the sniffer host the ICMP answer from the R-10-01, but I think it comes only when MAC or ARP table doesn't have the entry and the packet will be flooded over all ports.

It's very very strange, I'd be appriciate if somebody could at least give an idea what can be wronfg there.

Konstantin Dunaev · ‎08-23-2011

I'd say the initial problem, when I lost the connection complete could be somehow related to that bug, but the versions I use on the switchs are different what the bug and topic are discussed

c3550 (SW-10-01) Version 12.2(44)SE6

c3560 (SW-01-01, SW-01-02) 12.2(44)SE

and why I get 50% packet loss in a second scenario? if the STP blocks the port I would lost 100% of packets.

One another thing is the "cloud" which conncets both sites - is a Provider network with QinQ tag, but I don't think it produce such a problem.

Konstantin Dunaev · ‎09-06-2011

Hi Peter!

I've just stated a new thread https://supportforums.cisco.com/thread/2102954

it seems to be a known problem - QinQ doesn't really work with RSPAN session.

It looks like that the "remote-span" VLAN (555 in our case) makes the PE switch think that there is a loop in a topology because "remote-span" vlan doesn't use STP.