N7k high latency and dropping

[Ahmed Gaily]

Dears,

We have Cisco Nexus 7K acting as core/gateway for wireless ISP. and we are facing high latency and dropping to this 7K (only to SVI IP) but the traffic going through (traffic from client to internet for example) is working fine.

But any traffic from/to core IP are effected with this behavior (pinging to SVI, routing protocols adjacency).

Kindly note that we have two hardware and both of them have this issue.

Kindly support.

[balaji.bandi]

Provide what Nexus code running. 

 we are facing high latency and dropping to this 7K (only to SVI IP)   < ---can we look the interface drop ?

But any traffic from/to core IP are effected with this behavior (pinging to SVI, routing protocols adjacency).   < -- need to provide that test outcome here .

use troubleshooting guide :

https://www.cisco.com/c/en/us/support/docs/switches/nexus-7000-series-switches/116136-trouble-ethanalyzer-nexus7000-00.html

[Ahmed Gaily]

Hi Balaji,

Not for specific SVI its for all SVIs. Please find attached capture when the issue is happening and one when everything is OK.

for ethanalyzer most of traffic are (PVST and ARP).

Thanks

[MHM Cisco World]

since any traffic bypass the NSK work fine only the traffic direct to SVI, then you need to check 
CoPP and CPU Queue drop. 

[Ahmed Gaily]

Hi MHM,

Please find attached show CPU and Copp output.

Thanks

[Ahmed Gaily]

Kindly note that I changed the copp profile from strict to lenient

[Ahmed Gaily]

Hi MHM, Is any upnormal things on CPU or copp output?

I changed the copp profile from strict to lenient but still same behavior.

Thanks!

[MHM Cisco World]

I check it now

[MHM Cisco World]

 service-policy input copp-system-p-policy-lenient
      match protocol arp
        conformed 663488 bytes,
          5-min offered rate 94784 bytes/sec
          peak rate 94784 bytes/sec at Sat Mar 11 20:00:17 2023
        violated 3536848 bytes,
          5-min violate rate 505264 bytes/sec
          peak rate 505264 bytes/sec at Sat Mar 11 20:00:17 2023

    class-map copp-system-p-class-undesirable (match-any)
      match access-group name copp-system-p-acl-undesirable
      match exception fcoe-fib-miss
        conformed 0 bytes,
          5-min offered rate 0 bytes/sec
          peak rate 0 bytes/sec
        violated 64 bytes,
          5-min violate rate 9 bytes/sec
          peak rate 9 bytes/sec at Sat Mar 11 20:00:17 2023
      module 4:
        conformed 0 bytes,
          5-min offered rate 0 bytes/sec
          peak rate 0 bytes/sec
        violated 60 bytes,
          5-min violate rate 8 bytes/sec
          peak rate 8 bytes/sec at Sat Mar 11 20:00:17 2023

    class-map copp-system-p-class-l2-default (match-any)
      match access-group name copp-system-p-acl-mac-undesirable
      match protocol mpls
      module 3:
        conformed 81651 bytes,
          5-min offered rate 11664 bytes/sec
          peak rate 11664 bytes/sec at Sat Mar 11 20:00:17 2023
        violated 57520 bytes,
          5-min violate rate 8217 bytes/sec
          peak rate 8217 bytes/sec at Sat Mar 11 20:00:17 2023

    class-map class-default (match-any)
      module 3:
        conformed 96646 bytes,
          5-min offered rate 13806 bytes/sec
          peak rate 13806 bytes/sec at Sat Mar 11 20:00:17 2023
        violated 94162 bytes,
          5-min violate rate 13451 bytes/sec
          peak rate 13451 bytes/sec at Sat Mar 11 20:00:17 2023
      module 4:
        conformed 33758 bytes,
          5-min offered rate 4822 bytes/sec
          peak rate 4822 bytes/sec at Sat Mar 11 20:00:17 2023
        violated 4570 bytes,
          5-min violate rate 652 bytes/sec
          peak rate 652 bytes/sec at Sat Mar 11 20:00:17 2023  

 all the CoPP show peak in same time, that meaning something happened in Sat mar 11, 
the CPU history also show us some peak within 72 hours, 
but in CPU sort I dont see anything high except some process which appear in top five in list 
the two must which make network slow is ARP and STP. 
can you do more monitor for the CPU history and CPU sort and CoPP, check if the peak appear in same time the CPU history show high CPU process around 60-70 %.

[Ahmed Gaily]

Hi MHM,

The CPU isn't exceeding the 25% even during the complete time out (we have console access) and the ARP process is taking about 4% only and STP is less than 1%.

For COPP. Yes there is violation but how we can detect/know the source of this traffic and how we can control it. I tried with ethanalyzer but most of traffic are (PVST and ARP) and I think its normal. I'm right?

Thanks

[Ahmed Gaily]

Anyupdates

[Ahmed Gaily]

Any solution?

[Ahmed Gaily]

@MHM Cisco World @balaji.bandi any solution

[Ahmed Gaily]

@MHM Cisco World @balaji.bandi 

Please check below ethanalyzer output

2023-03-16 23:02:45.965367 192.168.123.54 -> 15.218.26.79 TCP 66 52369 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.965382 192.168.123.54 -> 38.98.130.151 TCP 66 52371 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.965396 192.168.123.54 -> 34.130.134.84 TCP 66 52378 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.965410 192.168.123.54 -> 108.38.213.197 TCP 66 52382 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.965441 196.223.157.57 -> 18.198.55.236 ICMP 60 Echo (ping) reply id=0x0015, seq=4372/5137, ttl=255 (request in 817)
2023-03-16 23:02:45.965670 196.29.179.241 -> 13.208.243.220 ICMP 60 Echo (ping) reply id=0x0012, seq=6841/47386, ttl=255 (request in 827)
2023-03-16 23:02:45.966876 192.168.123.54 -> 36.197.250.150 TCP 66 52383 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.966894 192.168.123.54 -> 201.221.55.127 TCP 66 52402 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.966904 192.168.123.54 -> 215.57.169.118 TCP 66 52403 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.966914 192.168.123.54 -> 168.33.139.47 TCP 66 52411 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.967829 192.168.123.54 -> 32.117.229.211 TCP 66 52413 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.967843 192.168.123.54 -> 36.206.71.170 TCP 66 52415 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.967853 192.168.123.54 -> 223.62.87.85 TCP 66 52417 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.967862 192.168.123.54 -> 108.4.239.120 TCP 66 52427 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2023-03-16 23:02:45.967872 192.168.123.54 -> 151.158.227.197 TCP 66 52430 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

Note: 192.168.123.54 isn't our IP

Thanks!

[MHM Cisco World]

2023-03-16 23:02:45.965382 192.168.123.54 -> 38.98.130.151 TCP 66 52371 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1

if the traffic is between two Host in your network why it need to punt to CPU??
are 192.168.123.54 is your VLAN SVI IP ?