ICMP issue on Nexus 5K

[bhoopendra.vishwakarma1]

We have a situation, the oracle exadata database server is unable to ping the default gateway when using LACP (between server and N5k) and otherwise (without LACP between server and N5k) the communication works perfectly fine.

the set-up is like below;

Oracle Server >>>> Port-channel 129, ACCES VLAN 3240 >>> N5k1/N5l2(layer 2) >>TRUNK 3240 >> N7k (layer 3)

The oracle exadata database server is connected to two Nexus 5k chassis using LACP,

The Port-channel on Nexus 5k is up,

We could see the server mac address on port-channel interface.

There is a valid ARP on the N7k.

Any suggestion is highly appreciated, thank you in advance

[Kirk J]

Greetings.

What does the VPC status show? (asking because you mention a port-channel to two N5ks, that would require VPC)

Also, what is the config for your bond interface on your Linux host?

Thanks,

Kirk...

[bhoopendra.vishwakarma1]

Hi Krik,

Thank you for your assistance, please find the information requested are attached herewith.

Regards

Bhoopendra

[bhoopendra.vishwakarma1]

Hi Kirk, did you get a chance to review the logs I have shared with you, please. Awaiting your response.

-Bhoopendra

[sbhadrav@cisco.com]

On the Nexus 7000 you may see ICMP packet loss when pinging from the CPU to another device depending on the speed in which this traffic is responded and how much ICMP traffic is being sent to the switch at that moment.  This is due to the default CoPP (Control Plane Policing) service policy that is enabled by default on the N7k.
 

 The following will be seen when attempting to ping another switch from the N7k:

 
nexus#ping 14.1.106.1 c 1000 
PING 14.1.106.1 (14.1.106.1): 56 data bytes
64 bytes from 14.1.106.1: icmp_seq=0 ttl=254 time=0.852 ms
64 bytes from 14.1.106.1: icmp_seq=1 ttl=254 time=0.977 ms
64 bytes from 14.1.106.1: icmp_seq=2 ttl=254 time=4.101 ms
<snip>
64 bytes from 14.1.106.1: icmp_seq=22 ttl=254 time=3.002 ms
Request 23 timed out
Request 24 timed out
Request 25 timed out
<snip>
--- 14.1.106.1 ping statistics ---
1000 packets transmitted, 987 packets received, 0.98% packet loss
round-trip min/avg/max = 0.847/1.112/11.473 ms

 
The reason for these drops is that ICMP is policed on the N7k very aggressively.  Below is the portion of the CoPP policy which handles ICMP:

 IP access list copp-system-acl-icmp
         10 permit icmp any any echo
         20 permit icmp any any echo-reply

 
class-map type control-plane match-any copp-system-class-monitoring
   match access-group name copp-system-acl-icmp
   match access-group name copp-system-acl-icmp6
   match access-group name copp-system-acl-traceroute

 class copp-system-class-monitoring
     set cos 1
     police cir 130 kbps bc 1000 ms conform transmit violate drop

 
We can see that this policy polices ICMP down to 130kbps and that we will drop this traffic if it goes over this amount.   Also keep in mind that these policers are performed a per-module basis.  That means that each module is limited to 130kbps independently.

 
We can look to see if we are dropping any traffic due to the policy-map with the following command:

 7000-1# show policy-map interface control-plane

 Below, I am going to pipe for only the monitoring class:

 7000-1# sh policy-map interface Control-plane | b monitoring
     class-map copp-system-class-monitoring (match-any)
       match access-grp name copp-system-acl-icmp
       match access-grp name copp-system-acl-icmp6
       match access-grp name copp-system-acl-traceroute
       set cos 1
       police cir 130 kbps , bc 1000 ms
       module 2 :
         conformed 397890 bytes; action: transmit
         violated 12386 bytes; action: drop

   module 3 :
         conformed 0 bytes; action: transmit
         violated 0 bytes; action: drop

   module 4 :
         conformed 0 bytes; action: transmit
         violated 0 bytes; action: drop

 As we can see we have violated this policer on module 2 due to this is where 14.1.106.1 is attached:

 7000-1# sh ip arp 14.1.106.1

 Flags: * - Adjacencies learnt on non-active FHRP routr
        # - Adjacencies Throttled for Glean
        D - Static Adjacencies attached to down interface

 IP ARP Table
Total number of entries: 1
Address         Age       MAC Address     Interface
14.1.106.1      00:03:45  0000.0c07.acca  Ethernet 2/48

 However,  we have not sent any ICMP traffic on any other module, since these are independently policed.
 We can remove this order to prove that this is the cause of the drops, we can remove this policy-map for a short period of time and perform the ping again.

 However, first lets look at the name of the policy-map used for COPP.  This can be done using the "show run copp" command.

 This command will show the entire CoPP policy.  At the bottom of this output will show the same of the service-policy attached to the control-plane:

 7000-1# show run copp
<snip>
control-plane
   service-policy input copp-system-policy

 Now that we have the name lets remove this policy and attempt the ping again:

 7000-1(config)# control-plane
7000-1(config-cp)# no service-policy input copp-system-policy
7000-1(config-cp)#ping 14.1.106.1 c 1000
PING 14.1.106.1 (14.1.106.1): 56 data bytes
64 bytes from 14.1.106.1: icmp_seq=0 ttl=254 time=1.16 ms
64 bytes from 14.1.106.1: icmp_seq=1 ttl=254 time=0.976 ms
64 bytes from 14.1.106.1: icmp_seq=2 ttl=254 time=0.841 ms
<snip>

 --- 14.1.106.1 ping statistics ---
1000 packets transmitted, 1000 packets received, 0.00% packet loss
round-trip min/avg/max = 0.84/1.131/19.714 ms

 As you can see wiwth the CoPP policy removed we are not longer dropping traffic.  At this ping we can put back on the CoPP policy to further protect the CPU.

 Also, if we want to leave on the CoPP policy and be able to ping wihtout loss we can adjust the policer to a higher value.  This can be done in the following manner:

 7000-1(config)# policy-map type control-plane copp-system-policy 
7000-1(config-pmap)# class copp-system-class-monitoring
7000-1(config-pmap-c)#police cir 500 kbps bc 1000 ms conform transmit violate drop

 This will increase the amount of ICMP allowed to the CPU to 500kbps per module.  This can be adjusted to your need.

 We can see this change with the following command:

 F340.11.03-7000-1# sh run copp | b 'control-plane copp-system-policy'
policy-map type control-plane copp-system-policy
   class copp-system-class-exception
     set cos 1
     police cir 360 kbps bc 250 ms conform transmit violate drop
   class copp-system-class-critical
     set cos 7
     police cir 39600 kbps bc 250 ms conform transmit violate drop
   class copp-system-class-important
     set cos 6
     police cir 1060 kbps bc 1000 ms conform transmit violate drop
   class copp-system-class-management
     set cos 2
     police cir 10000 kbps bc 250 ms conform transmit violate drop
   class copp-system-class-normal
     set cos 1
     police cir 680 kbps bc 250 ms conform transmit violate drop
   class copp-system-class-redirect
     set cos 1
     police cir 280 kbps bc 250 ms conform transmit violate drop
  class copp-system-class-monitoring
    set cos 1
    police cir 500 kbps bc 1000 ms conform transmit violate drop
<snip>
control-plane
   service-policy input copp-system-polic