Nexus 9332PQ CoPP matching wrong traffic

[s.giunt]

We are experiencing a weird problem with a Nexus 9332PQ (NX-OS 9.3(8)). When there is an SSH transfer between two directly connected hosts, this CoPP policy kicks in:

class-map copp-system-p-class-management (match-any)
match access-group name copp-system-p-acl-ftp
match access-group name copp-system-p-acl-ntp
match access-group name copp-system-p-acl-ssh
match access-group name copp-system-p-acl-http
match access-group name copp-system-p-acl-ntp6
match access-group name copp-system-p-acl-sftp
match access-group name copp-system-p-acl-snmp
match access-group name copp-system-p-acl-ssh6
match access-group name copp-system-p-acl-tftp
match access-group name copp-system-p-acl-https
match access-group name copp-system-p-acl-snmp6
match access-group name copp-system-p-acl-tftp6
match access-group name copp-system-p-acl-radius
match access-group name copp-system-p-acl-tacacs
match access-group name copp-system-p-acl-telnet
match access-group name copp-system-p-acl-radius6
match access-group name copp-system-p-acl-tacacs6
match access-group name copp-system-p-acl-telnet6
set cos 2
police cir 3000 pps , bc 512000 packets
module 1 :
transmitted 535439403 packets;
dropped 964462264 packets;

It seems like the policy is matching traffic that is not directed to the switch. The result is that SSH/SNMP/etc. on the switch stops working.

[s.giunt]

Yes, no ip redirects is present in Vlan10 interface config.

[Christopher Hart]

Understood! I have an interesting (and untested!) theory about this issue revolving around the following static routes you have configured:

ip route 10.10.10.2/32 Vlan10
ip route 10.10.10.3/32 Vlan10

I know that on Cisco IOS, static routes pointing to an egress interface without a next-hop IP address can cause performance issues due to the router attempting to resolve ARP for every IP address that matches against the static route. This is explained in detail here. I wonder if this is causing similar behavior on the Nexus platform (honestly, this is the first time I have seen this type of static route on Nexus switches, so I am not 100% what the behavior would/should be).

I believe you previously stated that you need these IP addresses in order to advertise reachability to these /32 hosts via BGP. NX-OS is a bit unique in that ARP entries are inserted into the routing table as /32 entries under the "AM" protocol (which stands for Adjacency Manager, the software process responsible for maintaining the switch's ARP cache).

Unlike IOS/IOS-XE, NX-OS allows you to easily redistribute AM-learned prefixes in the routing table into routing protocols, including BGP.

N9K(config-router-af)# where
  conf; router bgp 65001; address-family ipv4 unicast      admin@N9K%default

N9K(config-router-af)# redistribute ?
  am      AM routes (learned via ARP)
  direct  Directly connected
  eigrp   Enhanced Interior Gateway Protocol
  hmm     HMM prefix
  isis    ISO IS-IS
  lisp    LISP EID-prefixes in the non-default VRF
  ospf    Open Shortest Path First (OSPF)
  rip     Routing Information Protocol (RIP)
  static  Static routes

You can/should/must attach a route-map to the redistribution statement, which would allow you to filter the specific prefixes you'd like redistributed down to these two /32s (unless you have additional /32s you haven't told us about, which you can also add to the corresponding ACL/prefix-list called by the route-map).

I say all this to ask: Would you be willing to replace your current static route configuration with a redistribution statement under BGP that redistributes these two AM-learned /32s routes into BGP to test whether the static routes could be the source of this issue?

Thank you!

-Christopher

[s.giunt]

I was not aware of that "redistribute ARP" future, I've seen it only in Cumulus switches. It's very interesting, I will test it ASAP and let you know the results.

If every packet is sent to the CPU when there is a static route without explicit nexthop, this could explain everything. Altrough I would expect that only for packets with an incomplete ARP entry.

[MHM Cisco World]

But the CoPP drop ssh not arp. 

The issue not arp

The issue that unusual config. 

Both host have Mac of gw instead of Mac of other host.

For bgp why you need to advertise two host /32 which lead to same point vlan S I?

Instead redistribute direct with filter and use /24