Nexus 9332PQ CoPP matching wrong traffic

[s.giunt]

We are experiencing a weird problem with a Nexus 9332PQ (NX-OS 9.3(8)). When there is an SSH transfer between two directly connected hosts, this CoPP policy kicks in:

class-map copp-system-p-class-management (match-any)
match access-group name copp-system-p-acl-ftp
match access-group name copp-system-p-acl-ntp
match access-group name copp-system-p-acl-ssh
match access-group name copp-system-p-acl-http
match access-group name copp-system-p-acl-ntp6
match access-group name copp-system-p-acl-sftp
match access-group name copp-system-p-acl-snmp
match access-group name copp-system-p-acl-ssh6
match access-group name copp-system-p-acl-tftp
match access-group name copp-system-p-acl-https
match access-group name copp-system-p-acl-snmp6
match access-group name copp-system-p-acl-tftp6
match access-group name copp-system-p-acl-radius
match access-group name copp-system-p-acl-tacacs
match access-group name copp-system-p-acl-telnet
match access-group name copp-system-p-acl-radius6
match access-group name copp-system-p-acl-tacacs6
match access-group name copp-system-p-acl-telnet6
set cos 2
police cir 3000 pps , bc 512000 packets
module 1 :
transmitted 535439403 packets;
dropped 964462264 packets;

It seems like the policy is matching traffic that is not directed to the switch. The result is that SSH/SNMP/etc. on the switch stops working.

[MHM Cisco World]

are you sure the traffic end to host or the traffic end to NSK IP ??
are you using NATing before the NSK ?? this can lead to this behave 

[s.giunt]

I'm 100% sure there is no traffic directed to an IP belonging to the switch.

configuration example:

Host A is connected to port 1, assigned IP address: 10.10.10.2/32 gw: 10.10.10.1

Host B is connected to port 2, assigned IP address: 10.10.10.3/32 gw: 10.10.10.1

port 1 and 2 are in the same Vlan (switchport access vlan10). 

ip route 10.10.10.2/32 Vlan10
ip route 10.10.10.3/32 Vlan10

Interface Vlan10
[...]
hsrp 1
mac-address 0000.0000.0001
ip 10.10.10.1/24
[...]

TCP port 22 flows between 10.10.10.2 and 10.10.10.3 trigger the CoPP policy.

Also, I noticed that:

- The problem only happens when the ARP is resolved. i.e.: if 10.10.10.3 is offline (ARP Incomplete) and I send packets from 10.10.10.2 to 10.10.10.3, packets are not matched by CoPP.

- The problem only happens when both hosts are directly connected to the switch (i.e.: packets coming from the uplink and directed to 10.10.10.2/10.10.10.3 are not matched.

Another strange thing is that hardware rate-limiter L3 glean dropped packet counter keeps increasing even if the ARP is resolved (it should only increase when there is an incomplete ARP entry)

[MHM Cisco World]

ip route 10.10.10.2/32 Vlan10
ip route 10.10.10.3/32 Vlan10

this make me think 
the port in NSK is L2 
and you can config SVI for this VLAN and I see you also config  HSRP with 10.10.10.1/24

so why you need static route !!!
instead assign ip with /24 to both host 
remove the static route 
make sure that IP you assign to host not assign to SVI of VLAN in this NSK or other NSK (if you run vPC).

[s.giunt]

I understand this might be a strange config, but the static routes of the /32s are there because they are redistribuited through BGP. Anyway, this does not explain why packets not destined to the switche are matched by CoPP.

[MHM Cisco World]

sure it explain why the packet is routing not bridging in NSK 

for the view of NSK the frame need to routing and hence send to SVI (CPU) and here the CoPP restrick the connection. 
only do what I mention above and check again. 

[s.giunt]

Do you mean that two devices in the same VLAN cannot communicate over L3?

for the view of NSK the frame need to routing and hence send to SVI (CPU)  --> The switch has both a route and an ARP entry for that IP, so it should not be sent to the CPU. In facts, packets between the two hosts are hardware switched for sure, as I can easily saturate a 10G port.

[Christopher Hart]

Hello!

Is this a single Nexus 9332PQ switch with two hosts connected to it? Or are there two Nexus switches in a vPC domain?

Would you be able share the output of the ethanalyzer local interface inband display-filter "icmp or ip.addr==10.10.10.2" limit-captured-frames 0 command executed while you are reproducing the problem? This is a non-disruptive control plane packet capture filtered on ICMP packets (to capture any potential ICMP Redirect scenarios) as well as packets sourced from or destined to host 10.10.10.2.

Thank you!

-Christopher

[s.giunt]

Hi,

Two Nexus in vPC are connected to a single FEX in active/active mode. Hosts are connected to the FEX.

Regarding the capture, logs are flooded with this message:

2023-02-24 14:42:12.156120 10.10.10.2 -> 10.10.10.3 TCP ssh > 10688 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

There are no ICMP packets.

[Christopher Hart]

Hi!

Understood! Can you execute the ethanalyzer local interface inband display-filter "icmp or ip.addr==10.10.10.2" limit-captured-frames 0 detail command and provide the full output (or at least the headers) of a single packet captured? I'd like to see the headers of these SSH packets between the two hosts.

As a side note, the real "issue" here isn't so much the misclassification of this traffic by CoPP as SSH traffic. It's more so the fact that this data plane traffic between two hosts in the same VLAN is being punted to the control plane of the switch (at which point, CoPP does its best to classify the traffic accordingly - since this traffic is SSH, and SSH traffic is listed highest in the CoPP policy, it's being classified as SSH traffic instead of another class (which, unfortunately, would help us understand why this traffic is being punted in the first place).

[s.giunt]

Thanks, here is the packet details: https://pastebin.com/7T6cxHqr

The packets are hardware switched, otherwise it would be impossible to obtain line-rate trasmission. Also the CoPP police is not actually enforced on the data plane, as there is no packet loss. It seems that even if the packets are hardware switched, for some reason a "copy" is sent to the CPU (encapsulated with 802.1q VLAN tag)

[Christopher Hart]

Can you provide the MAC addresses for 10.10.10.3 (I know 10.10.10.2 is 0cc4.7a07.673e according to the packet capture you provided), as well as the ARP table/cache from both hosts for the other host's respective IP (e.g. the ARP cache for 10.10.10.3 from 10.10.10.2's perspective, as well as the ARP cache for 10.10.10.2 from 10.10.10.3's perspective)? I'd also be curious to see the ARP cache for 10.10.10.2 and 10.10.10.3 from each Nexus 9332PQ's perspective (they should be identical, but I'd like to see it from both just in case).

In the packet capture you provided, I can see the destination MAC address of the SSH packet sent by 10.10.10.2 (0cc4.7a07.673e) is 0000.0000.0001, which seems to match the HSRP virtual MAC address you have configured for HSRP group 1 in VLAN 10. This seems to imply that 10.10.10.2 thinks that for it to get to 10.10.10.3, it needs to send the packet to its default gateway (10.10.10.1)., which is why I'd like to see the ARP cache for 10.10.10.3 from 10.10.10.2's perspective.

Also, just wanted to confirm - hosts 10.10.10.2 and 10.10.10.3 are configured to be in the same 10.10.10.0/24 broadcast domain in VLAN 10, correct? Your original post could be interpreted in a way that suggests both hosts are configured with /32 subnet masks, which would be (in my opinion) a somewhat unusual configuration.

Thank you!

-Christopher

[MHM Cisco World]

Thanks 

[s.giunt]

Hosts are configured as /32, so they reach each other through gateway MAC address. (yes, no doubt it's an unusual configuration, but this could easily lead to a DoS attack (i.e. sending trafic to port 179)) 

ARP tables are identical on both switch

Address Age MAC Address Interface Flags
10.10.10.3 00:09:08 001b.21d3.fd40 Vlan10 +

10.10.10.2 00:09:08 0cc4.7a07.673e Vlan10 +

[Christopher Hart]

Can you confirm ICMP Redirects are disabled on the VLAN 10 SVI of both Nexus switches through the no ip redirects interface configuration command? ICMP Redirects being enabled (which is default configuration) could cause data plane traffic to be punted to the control plane in this specific scenario, since the subnet mask of your hosts does not match the subnet mask configured on the switches.

Thank you!

-Christopher