Issue with Nexus 5K dropping ping packets

[r.lundrigan]

I have 2 Nexus 5Ks that drop ping packets when pinging between them at 70-90% drop rate.

The ping drop rate also occurs for the following:

- Between access layer switches and 5Ks

- Between end user devices/servers and 5K

- Between PC and HSRP gateway address on 5K

No drops occur for following:

- From switch to switch passing through Nexus

- From PC to server passing though Nexus

- From PC to device on same network of HSRP gateway address

I am aware of the policing issue on the 5Ks but I don't think that is the case here. This just started happening recently and we've had the 5Ks for over 2 years now.

Any t-shooting or debug command ideas?

[Georg Pauwen]

Hello,

what is the uptime of the switches ? Often slow response times are related to long uptimes...

In order to make sure that the issue is NOT related to the default CoPP policy, can you see any increases in the 'violated' counter for ICMP traffic ?

show policy-map interface control-plane

[r.lundrigan]

Hi,

Uptime is currently 240 days

I ran the sh command and found this:

class-map copp-system-class-icmp-echo (match-any)
match protocol icmp_echo
police cir 64 kbps , bc 3600000 bytes
conformed 150309163572 bytes; action: transmit
violated 4840309833 bytes;

[Georg Pauwen]

Hello,

are the counters increasing when you ping (and have packet loss) ? 240 days is actually not that long, if you have a service window though, you might want to reboot...

[r.lundrigan]

They are definitely increasing even without me pinging but these 5Ks are also using snmp for a Solarwinds server

[Georg Pauwen]

Hello,

you might want to change the default policer in the class map for ICMP traffic and check if that has any effect on your ping responses. If it does, you know it is the CoPP.

[r.lundrigan]

I did try to lower the ping packet size to something lower than 64 and still had the drops. Would that be a similar test?

[Georg Pauwen]

You need to change the policing rate to effectively test...

[r.lundrigan]

Just an update. I did some ICMP debugging and found that the pings are being redirected to a bogus gateway IP and that is most likely causing the ping fails. We did some research and found this bug and this is exactly what we are experiencing. The gateway address showing up in the debug messages happens to be one digit off. So instead of it forwarding packets to something like 10.1.1.1, it's forwarding to 10.0.1.1. It's like it's nulling out the 2nd octet.

https://quickview.cloudapps.cisco.com/quickview/bug/CSCus28969

[Georg Pauwen]

Hello,

very interesting indeed, and kind of a weird bug...

Either way, if that is the solution, good that you have found it...

[alberto.sanchez2]

Hello,

How did you finally solve this issue?

Best regards.

[gilsang kim]

It's the same as my problem.
So I upgraded to a higher OS. But it didn't work out. I'm curious.

[MHM Cisco World]

Make new post  it better 

MHM