qos3Outlost counters incrementing

[ainhoa.garcia]

Hi,

I have one problem with some ports of one Cisco 6509 with the following IOS version: c6sup22-pk2sv-mz.121-26.E6.bin.

We look through the oversubscription counters once peer week, and in the last weeks we have seen a noticeable increment on the ports that have SPAM enable. We have prove to remove the SPAM but the counters are still growing.

We have also notice, that on some of the ports that are "disabled" there is oversubscription, I mean, the value of the counters of qos3Outlost have a non-zero value.

Please if you know something about this help me. Don`t hesitate to ask for more information or details you need ok?

Thanks,

Ainhoa

[Shashank Singh]

HI Ainhoa,

Incremeting qos3Outlost counters indicate packet drops due to oversubscription on the line card in question.

Since you are getting into known oversubscribtion issue with the concerned blade, it will be a good idea to move some of the traffic from this line card to some other line card on the chassis.

Another option is to configure flowcontrol under the interface but it will only work if the server connected to it also supports flowcontrol.

Backgroud info on FlowControl is available at:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/catos/7.x/configuration/guide/ether.html#wp1020960

I would suggest that you clear counters and monitor these counters for sometime to find out which interfaces have the counters incrementing. Move the traffic on that interface to some other blade on the chassis.

This information is documented at

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801751d7.shtml

Hope this helps,

Shashank

Please rate if you found the content useful

[ainhoa.garcia]

Hi Shashank,

Thank you for the response. I have some questions about it. You suggest to enable flow-control on the ports that have them qos3Outlost counters incrementing. What I wanted to know if this would affect to the service provided by our customer or not.

On the other hand, I would also like to tell you that we have the interfaces indentified and, like I said, there are the ones that have the SPAM enable. We tried to disable it and catch the counters value again, and it kept incrementing.

This only happend on the last octect of the 6509. I don`t know if there is any bug that explains this situation, that with SPAM enable the counters increment...or something.

I would be grateful if you could answer me these question.

Thanks,

Ainhoa

[Shashank Singh]

Hi Ainhoa,

The fact that drop counters are incrementing for certain ports confirms that these ports are receiving more data than what they can handle (possibly due to oversubscription).

Enabling Flow-control may help to delay excess traffic and hence stop the drops to a certain extent. As flow control is a per port configuration, I would suggest to implement it only on the ports having the drops and enabling it will not impact traffic on any other interface on the switch.

Cheers,

Shashank

P.S. please rate the helpful post.

[ainhoa.garcia]

Hi Shashank,

As far as I know, there is no too much trafficc through these ports so I don`t know if enabling the flow-control is going to help but I am going to ask our customer to try it, at least on one of the interfaces experiencing the issue.

Thanks,

Ainhoa

P.S. I have rate your responses

[mgalazka]

Which model line card is it experiencing the issue?  Where I've seen qos3Outlost cause problems is on cards that use HOL blocking like a 6148.  Most of the content in this post is focused around HOL blocking.

Also, are you tracking the rate of increase for qos3Outlost -- do you know when it is increasing and by how much?  Knowing the timing and rate of lost packets can help you determine the cause.  For instance, I have seen this in the past happening on weekend evenings, coinciding nicely with weekly backups.

When HOL blocking is occurring, it is not that any one port is receiving too much traffic, but the ASIC handling the ports is overloaded.  Aside from flow control, there are a couple other things you can try.

1. Issue "show inter capabilities module X | include ASIC" (where X is module with qos3Outlost counter at non-zero value).  This will give you the ports sharing the same ASIC.

2. Are there a mix of 10/100Mbit and 1Gbit ports in that port group?  Try separating these out and see if this alleviates any of the HOL-blocking performance issues (as it may be related to burst traffic that needs to be buffered by the ASIC).  This may also help determine what ports are causing the HOL blocking, so they can be moved to a card that does not used shared buffers.

3. The interfaces have small buffers compared to the shared ASIC buffer, but another alternative is to disable the shared buffer on the ASIC forcing traffic to use the interface buffers.  While I would not recommend this in the long term - what it will do is make it readily obvious which port is causing the HOL-blocking issues, as its own small interface buffer will be dropping traffic like crazy.  As such, expect that the "problem port" will see network performance drop significantly. You can then move this to an open port on a 6748 linecard or another non-blocking model.  Once the problem ports are removed from the linecard, you should be able to enable the shared buffer again, and hopefully you will not have any more issues with HOL blocking.

Router(config)# interface gigabitethernet

Router(config-if)# hol-blocking disable

4. Of course the long-term solution is to replace your modules with less oversubscribed cards, especially if this is in a data center or the like.

Most of this info is in the doc linked to previously: http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801751d7.shtml

Hope this helps,

Matt

[ainhoa.garcia]

Hi Matt,

Our customers line card is a WS-X6548-GE-TX, so I think it would be possible what you are saying. Besides, I have found one Cisco bug that says the following:

When a WS-X6548-GE-TX-CR or WS-X6148-GE-TX port is configured as a SPAN
destination and the aggregate bandwidth of the spanned traffic exceeds the
SPAN destination port's capacity, user can see performance issues on other
ports within the same group.

A group of ports is defined as a group of 8-ports (ie 1-8, 9-16,17-24, etc..)


Workaround:

Disable destination SPAN to ports of above module (or) put the SPAN destination
port in a group where there is nothing connected.

I have asked our customer to try to put these ports alone to see what happend. I would tell you something when I have the results.

Thanks for the information.

Ainhoa

[ainhoa.garcia]

Hi there everyone,

Matt, I have told our customer to try the hol-blocking, but in their model there is no option to enable/disable it. I am looking througth the web ti see if there is a similar command for it but I have not found nothing. Did you know if there is another way to configure it?Our customer`s is a cisco WS-C6509 (R7000) processor (revision 3.0).

If you know another way to solve this issue please don`t hesitate to tell me ok? I would be grateful if you do so.

Thanks,

Ainhoa

[ainhoa.garcia]

Hello!!

Anyone knows another feature that has the same functionality that "hol-blocking" for a Cisco 6509 (R7000) processor (revision 3.0)??

Every help provided would be very hepful!

Thanks,

Ainhoa