Introduction:
On the Nexus 7000 you may see ICMP packet loss when pinging from the CPU to another device depending on the speed in which this traffic is responded and how much ICMP traffic is being sent to the switch at that moment. This is due to the default CoPP (Control Plane Policing) service policy that is enabled by default on the N7k.
The following will be seen when attempting to ping another switch from the N7k:
nexus#ping 14.1.106.1 c 1000
PING 14.1.106.1 (14.1.106.1): 56 data bytes
64 bytes from 14.1.106.1: icmp_seq=0 ttl=254 time=0.852 ms
64 bytes from 14.1.106.1: icmp_seq=1 ttl=254 time=0.977 ms
64 bytes from 14.1.106.1: icmp_seq=2 ttl=254 time=4.101 ms
<snip>
64 bytes from 14.1.106.1: icmp_seq=22 ttl=254 time=3.002 ms
Request 23 timed out
Request 24 timed out
Request 25 timed out
<snip>
--- 14.1.106.1 ping statistics ---
1000 packets transmitted, 987 packets received, 0.98% packet loss
round-trip min/avg/max = 0.847/1.112/11.473 ms
The reason for these drops is that ICMP is policed on the N7k very aggressively. Below is the portion of the CoPP policy which handles ICMP:
IP access list copp-system-acl-icmp
10 permit icmp any any echo
20 permit icmp any any echo-reply
class-map type control-plane match-any copp-system-class-monitoring
match access-group name copp-system-acl-icmp
match access-group name copp-system-acl-icmp6
match access-group name copp-system-acl-traceroute
class copp-system-class-monitoring
set cos 1
police cir 130 kbps bc 1000 ms conform transmit violate drop
We can see that this policy polices ICMP down to 130kbps and that we will drop this traffic if it goes over this amount. Also keep in mind that these policers are performed a per-module basis. That means that each module is limited to 130kbps independently.
We can look to see if we are dropping any traffic due to the policy-map with the following command:
7000-1# show policy-map interface control-plane
Below, I am going to pipe for only the monitoring class:
7000-1# sh policy-map interface Control-plane | b monitoring
class-map copp-system-class-monitoring (match-any)
match access-grp name copp-system-acl-icmp
match access-grp name copp-system-acl-icmp6
match access-grp name copp-system-acl-traceroute
set cos 1
police cir 130 kbps , bc 1000 ms
module 2 :
conformed 397890 bytes; action: transmit
violated 12386 bytes; action: drop
module 3 :
conformed 0 bytes; action: transmit
violated 0 bytes; action: drop
module 4 :
conformed 0 bytes; action: transmit
violated 0 bytes; action: drop
As we can see we have violated this policer on module 2 due to this is where 14.1.106.1 is attached:
7000-1# sh ip arp 14.1.106.1
Flags: * - Adjacencies learnt on non-active FHRP router
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface
IP ARP Table
Total number of entries: 1
Address Age MAC Address Interface
14.1.106.1 00:03:45 0000.0c07.acca Ethernet 2/48
However, we have not sent any ICMP traffic on any other module, since these are independently policed.
We can remove this order to prove that this is the cause of the drops, we can remove this policy-map for a short period of time and perform the ping again.
However, first lets look at the name of the policy-map used for COPP. This can be done using the "show run copp" command.
This command will show the entire CoPP policy. At the bottom of this output will show the same of the service-policy attached to the control-plane:
7000-1# show run copp
<snip>
control-plane
service-policy input copp-system-policy
Now that we have the name lets remove this policy and attempt the ping again:
7000-1(config)# control-plane
7000-1(config-cp)# no service-policy input copp-system-policy
7000-1(config-cp)#ping 14.1.106.1 c 1000
PING 14.1.106.1 (14.1.106.1): 56 data bytes
64 bytes from 14.1.106.1: icmp_seq=0 ttl=254 time=1.16 ms
64 bytes from 14.1.106.1: icmp_seq=1 ttl=254 time=0.976 ms
64 bytes from 14.1.106.1: icmp_seq=2 ttl=254 time=0.841 ms
<snip>
--- 14.1.106.1 ping statistics ---
1000 packets transmitted, 1000 packets received, 0.00% packet loss
round-trip min/avg/max = 0.84/1.131/19.714 ms
As you can see wiwth the CoPP policy removed we are not longer dropping traffic. At this ping we can put back on the CoPP policy to further protect the CPU.
Also, if we want to leave on the CoPP policy and be able to ping wihtout loss we can adjust the policer to a higher value. This can be done in the following manner:
7000-1(config)# policy-map type control-plane copp-system-policy
7000-1(config-pmap)# class copp-system-class-monitoring
7000-1(config-pmap-c)#police cir 500 kbps bc 1000 ms conform transmit violate drop
This will increase the amount of ICMP allowed to the CPU to 500kbps per module. This can be adjusted to your need.
We can see this change with the following command:
F340.11.03-7000-1# sh run copp | b 'control-plane copp-system-policy'
policy-map type control-plane copp-system-policy
class copp-system-class-exception
set cos 1
police cir 360 kbps bc 250 ms conform transmit violate drop
class copp-system-class-critical
set cos 7
police cir 39600 kbps bc 250 ms conform transmit violate drop
class copp-system-class-important
set cos 6
police cir 1060 kbps bc 1000 ms conform transmit violate drop
class copp-system-class-management
set cos 2
police cir 10000 kbps bc 250 ms conform transmit violate drop
class copp-system-class-normal
set cos 1
police cir 680 kbps bc 250 ms conform transmit violate drop
class copp-system-class-redirect
set cos 1
police cir 280 kbps bc 250 ms conform transmit violate drop
class copp-system-class-monitoring
set cos 1
police cir 500 kbps bc 1000 ms conform transmit violate drop
<snip>
control-plane
service-policy input copp-system-policy
Related Information:
Understand CoPP on Nexus 7000 Series Switches
Cisco Nexus 7000 Series Switches Troubleshooting TechNotes
Configuring Control Plane Policing
