02-25-2018 09:07 AM - edited 03-08-2019 02:01 PM
Dear All
for couple of days i've been fighting with ARP input drops & throttles on interfaces of the customer VSS (SUP-2T+69XX&68XX LCs).
I was able to avoid of any throttles on the interfaces (with hold-queue in incrementing) of the system, which imho should effectively remove any ARP drops on input (indicated by the drops in ARP section of the "sho ip traffic"output).
I was surprised with the result: even with no drops&throttles indicated on the all interfaces system still shows drops incrementing in the ARP section of "show ip traffic"... Could anybody enlighten on any gaps in the logic?
02-26-2018 02:33 PM
Hello,
Is your CPU high? Are you seeing a lot of arps come into your device?
You could be hitting CoPP or a hardware rate limiter if you are getting overloaded with traffic. I would check the following commands:
show policy-map control-plane <----Look for anything that has exceeded increasing
show fabric utilization detail
show fabric drop
show fabric error
show fabric channel-counter
show proc cpu hist
Feel free to post any of the outputs above that you are comfortable posting and I can take a look at them with you.
Hope that helps!
02-27-2018 04:16 AM
Hello Bradley
Acoording to your advices I've reverified control-place p/m and found some drops on the class below, but class itself has no definition :|
class-map: class-copp-icmp-redirect-unreachable (match-any)
exceeded 12680349 packets action: drop
aggregate-forward 69 pps exceed 5 pps
exceeded 10287152 packets action: drop
aggregate-forward 95 pps exceed 23 pps
exceeded 0 packets action: drop
aggregate-forward 0 pps exceed 0 pps
exceeded 24752412 packets action: drop
aggregate-forward 53 pps exceed 21 pps
exceeded 11361275 packets action: drop
aggregate-forward 83 pps exceed 11 pps
exceeded 0 packets action: drop
aggregate-forward 0 pps exceed 0 pps
Class-map: class-default (match-any)
5 minute offered rate 1780000 bps, drop rate 0000 bps
VSS-Core-Hol#sho class-map class-copp-icmp-redirect-unreachable
Class Map match-any class-copp-icmp-redirect-unreachable (id 1)
Match none
So i wonder what is dropped undeer this section? :)
Fabric utilization seems to be Ok:
VSS-Core-Hol#sho fabric utilization detail
Fabric utilization: Ingress Egress
Module Chanl Speed rate peak rate peak
1 0 40G 0% 34% @23:31 16Feb17 1% 34% @23:31 16Feb17
1 1 40G 0% 34% @23:31 16Feb17 0% 34% @23:31 16Feb17
2 0 20G 2% 34% @23:31 16Feb17 0% 47% @03:36 08May16
5 0 20G 0% 34% @23:31 16Feb17 0% 47% @03:36 08May16
5 1 20G 0% 34% @23:31 16Feb17 0% 34% @23:31 16Feb17
the same for the rest of fabric commands but drops & errors:
VSS-Core-Hol#sho fabric drop
Polling interval for drop counters and timestamp is 15 in seconds
Packets dropped by fabric for different queues:
slot channel Low-Q-drops High-Q-drops
1 0 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
1 1 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
2 0 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
5 0 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
5 1 137259280 @14:18 18Jul13 270852564 @14:18 18Jul13
VSS-Core-Hol#sho fabric error
Module errors:
slot channel crc hbeat sync DDR sync
1 0 0 0 0 0
1 1 0 0 0 0
2 0 0 0 0 0
5 0 0 0 0 0
5 1 0 0 0 0
Fabric errors:
slot channel sync buffer timeout
1 0 0 0 0
1 1 0 0 0
2 0 0 0 0
5 0 0 0 0
5 1 0 0 0
How i can correlate it out with ARP drops indicated in "sho ip traffic | sec ARP"?
02-28-2018 02:01 AM
answer on empty body of default classes for the default policy-map is found:
CoPP is enabled by default on Catalyst 6500 / SUP2T and Catalyst 6880 switches and is based on a preconfigured template. Some class-map configurations do not have corresponding match statements due to the fact that they capture traffic not on the MAC/IP Access Control List (ACL), but rather on internal exceptions that are signalled by the forwarding engine when traffic is received by the switch and a forwarding decision taken.
02-28-2018 03:34 AM
Hello Bradley
i've also found some ARP-relevant commands under platform:
platform qos protocol arp police|pass-through
sho platform qos arp
i've read about it in the https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/15-0SY/configuration/guide/15_0_sy_swcg/denial_of_service.html
but i'm still not clear on the dependencies & effects. Could u pls help me to understand?
02-28-2018 09:06 AM
Hello,
Sorry for the delay. The fabric drops you see are old and from last year or farther back so they are probably unrelated, you can clear them with the command:
clear fabric drop
clear fabric peak-utilization
Then you can monitor to see if they go up again.
You are correct, CoPP has classes for things that don't match an ACL very well. For example unreachable, if we receive a packet that we don't have a route for or is TTL 1 when we get it, we will drop the packet and send a message back to the source saying the destination is unreachable. However, in order to generate that message back, we would have to punt the packet to the CPU. To keep the CPU from getting overloaded if there are a lot of these packets, we limit them with CoPP. You can usually tell by the name what kind of issue it is. Is your drops increasing in the class-copp-icmp-redirect-unreachable" section? If so, that would indicate you are hitting the above issue or a redirect. Neither of which should be arps getting dropped.
Do you have arp policing turned on? If you run "show run all | inc platform qos protocol" do you have an entry for arp?
Hope that helps!
02-28-2018 12:49 PM
02-28-2018 12:51 PM
& BTW output of "show run all | inc platform qos protocol"is empty
02-28-2018 01:11 PM
What command did you run to get that output? Also, can you post the ARP section of show ip traffic? Thanks!
02-28-2018 01:18 PM
02-28-2018 01:23 PM
That is a very low number of drops compared to the number through the switch. How fast does it go up? If you were to clear the counters, how long would it be until you got a drop?
clear ip traffic
03-12-2018 11:53 AM
Hi Brad
it increments with portions of 2K per 1-3 minuts.
Currently i'm looking for detailed explanation of 'platform qos' command. unfortunatly documentation about it is quite poor. May be do u have some relevant boilerplate by any chance?
03-15-2018 08:07 AM
Hello,
Just to confirm,
Drop due to input queue full: 5215 <----This number increases 2k every couple minutes?
If that is the case did you recently clear your counters before grabbing this output?
#sho ip traffic | sec ARP
ARP statistics:
Rcvd: 132550654 requests, 44956893 replies, 9159 reverse, 0 other . <----Is this only 2-3 minutes of statistics
Sent: 7717897 requests, 12413429 replies (1252377 proxy), 0 reverse
Drop due to input queue full: 5215
That just doesn't seem right that you are getting 132 million arps in a couple minutes. Something doesn't seem right with the output provided.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: