SPAN Destination port packet drop

[ciscoannie]

Hi,

I configured a monitor session on our network and mirror three source ports bidirection traffic to a single destination. But the destination port is getting the serious packet drop even the traffic load is far away from the port bandwidth capability.

Anyone has the similar experience? Why it happens and how to resolve it.

Thanks in advance.

[nkarpysh]

Hello,

Traffic load is always an average of rate for particular period. If traffic on the ports is bursty you can  expect momebtary spikes up to the line rate especially when you SPAN multiple sources to single port. Those spikes cause the output drops on egress.

Would be more clear if you can attach following commands:

show int gix/y (for all ports in question)

show counter int gix/y

sh int gix/y counter err

Nik

[ciscoannie]

Thanks for reply. I am guessing it is because of burst traffic but has no evidence. 

Sorry, first time to post here. CTRL+V does not work. Do not know how to attach the command output.

BTW, I did not see any error on the interface.  

[nkarpysh]

Hey,

During reply there is a link "advance editor" in the top right corner of reply window. Click on it - it will give you a chance to attach file.

Hope this helps.

Nik

[ciscoannie]

Thanks for help.

The command output has been attached below.

[nkarpysh]

Hi,

Alld drops are outdiscards:

     OutDiscards = 1478491

Those drops occur when there is no output buffer available for traffic coming to egress port. So buffer already full with other packets as those coming to qucikly - possibly from multiple source to the single egress port. This can be momentary burst, as you rate already showing 200+MB which is average so we can get up to 1 GB easily. Solution can be appplyting QoS and prioritizing the traffic. Also QoS can help to tune the buffers and take additional ones from common pull.

Nik

[ciscoannie]

We have two span ports on different switches. The other one 5 mins average is 5M, but still get the packet drop.

This burst traffic has very little chance to get 1000M interface capability. 

I checked the counters on this interface and all drops are on OutDiscards. Is there any command I can monitor the real time traffic on the interface?

Thanks.

[nkarpysh]

Hey,

There is no in-built tool to check real time interface speed. I no there are some external hardware like taps which you can put in the middle of the link and check the real time speed. On Cisco you can only configure interface to calculate average rate for 30 seconds which is minimum. That is configured on interface bases with command "load-interval 30".

Also make sure the other side is not sending yoou Pause frames causing your ports to slow down.

Nik

[ciscoannie]

I will try to troubleshoot this further by removing some source port.

Do you have any experience on VACL? I went through the document and it sounds like a better idea than SPAN?

[nkarpysh]

VACL capture has one advantage to SPAN. It rarely capture several copies of a single [acket. WHile if you configure SPAN on RX and TX on [particular VLANs (no just single ports) - you will copy many packets twice. On incmong ports and outgoing ports. Imagine that broadcast is going in that VLAN - SPAN will capture that on ingress port and each single copy of that broadcast packet on each outgoing port.

VACL captures will capture only single packet of it.

So possible worth trying that.

Nik