SUP-2T VSS ; ARP drops & throttles on input

[andy!doesnt!like!uucp]

Dear All

for couple of days i've been fighting with ARP input drops & throttles on interfaces of the customer VSS (SUP-2T+69XX&68XX LCs).

I was able to avoid of any throttles on the interfaces (with hold-queue in incrementing) of the system, which imho should effectively remove any ARP drops on input (indicated by the drops in ARP section of the "sho ip traffic"output).

I was surprised with the result: even with no drops&throttles indicated on the all interfaces system still shows drops incrementing in the ARP section of "show ip traffic"... Could anybody enlighten on any gaps in  the logic?

[brselzer]

Hello,

Is your CPU high? Are you seeing a lot of arps come into your device?

You could be hitting CoPP or a hardware rate limiter if you are getting overloaded with traffic. I would check the following commands:

show policy-map control-plane  <----Look for anything that has exceeded increasing

show fabric utilization detail

show fabric drop

show fabric error

show fabric channel-counter

show proc cpu hist

Feel free to post any of the outputs above that you are comfortable posting and I can take a look at them with you. 

Hope that helps!

[andy!doesnt!like!uucp]

Hello Bradley

Acoording to your advices I've reverified control-place p/m and found some drops on the class below, but class itself has no definition :|
class-map: class-copp-icmp-redirect-unreachable (match-any)
exceeded 12680349 packets action: drop
aggregate-forward 69 pps exceed 5 pps
exceeded 10287152 packets action: drop
aggregate-forward 95 pps exceed 23 pps
exceeded 0 packets action: drop
aggregate-forward 0 pps exceed 0 pps
exceeded 24752412 packets action: drop
aggregate-forward 53 pps exceed 21 pps
exceeded 11361275 packets action: drop
aggregate-forward 83 pps exceed 11 pps
exceeded 0 packets action: drop
aggregate-forward 0 pps exceed 0 pps
Class-map: class-default (match-any)
5 minute offered rate 1780000 bps, drop rate 0000 bps

VSS-Core-Hol#sho class-map class-copp-icmp-redirect-unreachable
Class Map match-any class-copp-icmp-redirect-unreachable (id 1)
Match none

So i wonder what is dropped undeer this section? :)
Fabric utilization seems to be Ok:
VSS-Core-Hol#sho fabric utilization detail
Fabric utilization: Ingress Egress
Module Chanl Speed rate peak rate peak
1 0 40G 0% 34% @23:31 16Feb17 1% 34% @23:31 16Feb17
1 1 40G 0% 34% @23:31 16Feb17 0% 34% @23:31 16Feb17
2 0 20G 2% 34% @23:31 16Feb17 0% 47% @03:36 08May16
5 0 20G 0% 34% @23:31 16Feb17 0% 47% @03:36 08May16
5 1 20G 0% 34% @23:31 16Feb17 0% 34% @23:31 16Feb17
the same for the rest of fabric commands but drops & errors:

VSS-Core-Hol#sho fabric drop
Polling interval for drop counters and timestamp is 15 in seconds

Packets dropped by fabric for different queues:
slot channel Low-Q-drops High-Q-drops
1 0 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
1 1 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
2 0 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
5 0 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
5 1 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13

VSS-Core-Hol#sho fabric error
Module errors:
slot channel crc hbeat sync DDR sync
1 0 0 0 0 0
1 1 0 0 0 0
2 0 0 0 0 0
5 0 0 0 0 0
5 1 0 0 0 0

Fabric errors:
slot channel sync buffer timeout
1 0 0 0 0
1 1 0 0 0
2 0 0 0 0
5 0 0 0 0
5 1 0 0 0

How i can correlate it out with ARP drops indicated in "sho ip traffic | sec ARP"?

[andy!doesnt!like!uucp]

answer on empty body of default classes for the default policy-map is found:

CoPP is enabled by default on Catalyst 6500 / SUP2T and Catalyst 6880 switches and is based on a preconfigured template. Some class-map configurations do not have corresponding match statements due to the fact that they capture traffic not on the MAC/IP Access Control List (ACL), but rather on internal exceptions that are signalled by the forwarding engine when traffic is received by the switch and a forwarding decision taken.

[andy!doesnt!like!uucp]

Hello Bradley

i've also found some ARP-relevant commands under platform:
platform qos protocol arp police|pass-through
sho platform qos arp

i've read about it in the https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/15-0SY/configuration/guide/15_0_sy_swcg/denial_of_service.html

but i'm still not clear on the dependencies & effects. Could u pls help me to understand?

[brselzer]

Hello,

Sorry for the delay. The fabric drops you see are old and from last year or farther back so they are probably unrelated, you can clear them with the command:

clear fabric drop 

clear fabric peak-utilization

Then you can monitor to see if they go up again. 

You are correct, CoPP has classes for things that don't match an ACL very well. For example unreachable, if we receive a packet that we don't have a route for or is TTL 1 when we get it, we will drop the packet and send a message back to the source saying the destination is unreachable. However, in order to generate that message back, we would have to punt the packet to the CPU. To keep the CPU from getting overloaded if there are a lot of these packets, we limit them with CoPP. You can usually tell by the name what kind of issue it is. Is your drops increasing in the class-copp-icmp-redirect-unreachable" section? If so, that would indicate you are hitting the above issue or a redirect. Neither of which should be arps getting dropped.

Do you have arp policing turned on? If you run "show run all | inc platform qos protocol" do you have an entry for arp?

Hope that helps!

[andy!doesnt!like!uucp]

Hi Brad
i've cleared fabric events at 1st iteration over it. Now fabric doesnt signal anything wrong. But i still have ARP drops incrementing.
copp-arp was deleted by someone from the defaulp policy-map far in the past. So i'm pretty sure copp has nothing to do with ARP drops.
Also platform qos arp seems to be relevant to the interfaces where qos is configured with policy-map(s).
I'm still not sure ~output like below, but i tend to think it has nothing to do with ARP drops as well:
QoS Summary [ARP]: (* - shared aggregates, Mod - switch module, Sid - Switch Id, E - service instance)
(^ - class-copp keyword)

Int Sid Mod Dir Class-map DSCP Agg Trust Fl AgForward AgPoliced
Id Id
-------------------------------------------------------------------------------
CPP 1 1 In ^icmp-redi 0 3 dscp 0 56123964 14652308
CPP 1 2 In ^icmp-redi 0 3 dscp 0 73093082 11908970
CPP 2 1 In ^icmp-redi 0 3 dscp 0 82242077 27126055
CPP 2 2 In ^icmp-redi 0 3 dscp 0 74736539 12815736
Vl302 2 5 Out class-defa 0 18 -- 0 12374886 0

All 1 1 - Default 0 0* No 0 2755818736736897 0
All 1 2 - Default 0 0* No 0 484232432191698 0
All 1 5 - Default 0 0* No 0 2365683260 0
All 2 1 - Default 0 0* No 0 8630268364384599 0
All 2 2 - Default 0 0* No 0 4078169118733645 0
All 2 5 - Default 0 0* No 0 10262885026671 0

[andy!doesnt!like!uucp]

& BTW output of "show run all | inc platform qos protocol"is empty

[brselzer]

What command did you run to get that output? Also, can you post the ARP section of show ip traffic? Thanks!

[andy!doesnt!like!uucp]

sho platform qos arp | ex 0 0 0
#sho ip traffic | sec ARP
ARP statistics:
Rcvd: 132550654 requests, 44956893 replies, 9159 reverse, 0 other
Sent: 7717897 requests, 12413429 replies (1252377 proxy), 0 reverse
Drop due to input queue full: 5215

[brselzer]

That is a very low number of drops compared to the number through the switch.  How fast does it go up? If you were to clear the counters, how long would it be until you got a drop?

clear ip traffic

[andy!doesnt!like!uucp]

Hi Brad

it increments with portions of 2K per 1-3 minuts.

Currently i'm looking for detailed explanation of 'platform qos' command. unfortunatly documentation about it is quite poor. May be do u have some relevant boilerplate by any chance?

[brselzer]

Hello,

Just to confirm,

Drop due to input queue full: 5215 <----This number increases 2k every couple minutes?

If that is the case did you recently clear your counters before grabbing this output?

#sho ip traffic | sec ARP
ARP statistics:
Rcvd: 132550654 requests, 44956893 replies, 9159 reverse, 0 other .   <----Is this only 2-3 minutes of statistics
Sent: 7717897 requests, 12413429 replies (1252377 proxy), 0 reverse
Drop due to input queue full: 5215

That just doesn't seem right that you are getting 132 million arps in a couple minutes. Something doesn't seem right with the output provided.