Packet Loss pattern 9300 to VMnic

[santi8809]

Hi,

I am having this situation on an Interface connected to a VMware Host, where internally is configured a VMSwitch for MGMT.

Sending 1000, 100-byte ICMP Echos to 172.X.X.200, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!.

[M02@rt37]

Hello @santi8809 

Assess the load on the VMware host. High resource utilization might affect its ability to respond to ping requests promptly.

[santi8809]

This is a new Setup, there is no utilization. It only happens with the IP address asigned on the VMKernels Networks

[MHM Cisco World]

That normal it can CoPP of SW drop icmp packet when it reach specific limit.

MHM

[MHM Cisco World]

share the output of this 

Device# show policy-map control-plane

MHM 

[santi8809]

#show policy-map control-plane
Control Plane

Service-policy input: system-cpp-policy

Class-map: system-cpp-police-ios-routing (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 13000 pps, burst 3173 packets
conformed 27748276 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-ios-feature (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 6000 pps, burst 1464 packets
conformed 279979375 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-data (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 750 pps, burst 183 packets
conformed 53325891 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-sys-data (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 250 pps, burst 61 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-sw-forward (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 1000 pps, burst 244 packets
conformed 11532 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-multicast (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 500 pps, burst 122 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-multicast-end-station (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 2000 pps, burst 488 packets
conformed 7628298 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-punt-webauth (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 1000 pps, burst 244 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-l2-control (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 2000 pps, burst 488 packets
conformed 11054141 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-stackwise-virt-control (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 8000 pps, burst 1953 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-routing-control (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 5500 pps, burst 1342 packets
conformed 155024 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-system-critical (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 1000 pps, burst 244 packets
conformed 12512 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-l2lvx-control (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 1000 pps, burst 244 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-topology-control (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 13000 pps, burst 3173 packets
conformed 16539111 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-dot1x-auth (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 1000 pps, burst 244 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-protocol-snooping (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 2000 pps, burst 488 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-dhcp-snooping (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 500 pps, burst 122 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-forus (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 4000 pps, burst 976 packets
conformed 219013654 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 2000 pps, burst 488 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-high-rate-app (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 13000 pps, burst 3173 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: system-cpp-police-ewlc-control (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: none
police:
rate 13000 pps, burst 3173 packets
conformed 0 bytes; actions:
transmit
exceeded 0 bytes; actions:
drop

Class-map: class-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: any

[MHM Cisco World]

Also share this 

Show controllers cpu-interface 

MHM

[santi8809]

#Show controllers cpu-interface
queue retrieved dropped invalid hol-block
-------------------------------------------------------------------------
Routing Protocol 1253 0 0 0
L2 Protocol 65759 0 0 0
sw forwarding 103 0 0 0
broadcast 214611 0 0 0
icmp gen 0 0 0 0
icmp redirect 0 0 0 0
logging 0 0 0 0
rpf-fail 0 0 0 0
DOT1X authentication 0 0 0 0
Forus Traffic 34026 0 0 0
Forus Resolution 1725852 0 0 0
Inter FED 0 0 0 0
L2 LVX control 0 0 0 0
EWLC control 0 0 0 0
EWLC data 0 0 0 0
L2 LVX data 0 0 0 0
Openflow 0 0 0 0
Topology control 182175 0 0 0
Proto snooping 0 0 0 0
DHCP snooping 0 0 0 0
Transit Traffic 0 0 0 0
Multi End station 44343 0 0 0
Webauth 0 0 0 0
High rate app 0 0 0 0
Exception 0 0 0 0
System Critical 0 0 0 0
NFL sampled data 0 0 0 0
Low latency 0 0 0 0
EGR exception 0 0 0 0
Stackwise Virtual OOB 0 0 0 0
Multicast data 0 0 0 0
Gold packet 38 0 0 0

[MHM Cisco World]

Every thing is OK

The SW CPU and it queue dont drop traffic.

Show interface x

Check the output drop counter 

MHM

[santi8809]

sh int te1/0/2
TenGigabitEthernet1/0/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is cc36.cf29.0002 (bia cc36.cf29.0002)
Description: Uplink-HV1-Vmnic7
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 100/1000/2.5G/5G/10GBaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:17, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:02:03
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 6000 bits/sec, 10 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1629 packets input, 144884 bytes, 0 no buffer
Received 157 broadcasts (8 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 8 multicast, 0 pause input
0 input packets with dribble condition detected
491 packets output, 69308 bytes, 0 underruns
Output 36 broadcasts (73 multicasts)
0 output errors, 0 collisions, 0 interface resets
8 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[MHM Cisco World]

Hi friend can you share the output of this command 
9300_1#show plat hardware fed switch active qos queue stats internal cpu policer

sorry for late reply 
thanks 
MHM

[santi8809]

#show plat hardware fed switch active qos queue stats internal cpu policer

CPU Queue Statistics
============================================================================================
(default) (set) Queue Queue
QId PlcIdx Queue Name Enabled Rate Rate Drop(Bytes) Drop(Frames)
--------------------------------------------------------------------------------------------
0 11 DOT1X Auth Yes 1000 1000 0 0
1 1 L2 Control Yes 2000 2000 0 0
2 14 Forus traffic Yes 4000 4000 0 0
3 0 ICMP GEN Yes 750 750 0 0
4 2 Routing Control Yes 5500 5500 0 0
5 14 Forus Address resolution Yes 4000 4000 0 0
6 0 ICMP Redirect Yes 750 750 0 0
7 16 Inter FED Traffic Yes 2000 2000 0 0
8 4 L2 LVX Cont Pack Yes 1000 1000 0 0
9 19 EWLC Control Yes 13000 13000 0 0
10 16 EWLC Data Yes 2000 2000 0 0
11 13 L2 LVX Data Pack Yes 1000 1000 0 0
12 0 BROADCAST Yes 750 750 0 0
13 10 Openflow Yes 250 250 0 0
14 13 Sw forwarding Yes 1000 1000 0 0
15 8 Topology Control Yes 13000 13000 0 0
16 12 Proto Snooping Yes 2000 2000 0 0
17 6 DHCP Snooping Yes 500 500 0 0
18 13 Transit Traffic Yes 1000 1000 0 0
19 10 RPF Failed Yes 250 250 0 0
20 15 MCAST END STATION Yes 2000 2000 0 0
21 13 LOGGING Yes 1000 1000 0 0
22 7 Punt Webauth Yes 1000 1000 0 0
23 18 High Rate App Yes 13000 13000 0 0
24 10 Exception Yes 250 250 0 0
25 3 System Critical Yes 1000 1000 0 0
26 10 NFL SAMPLED DATA Yes 250 250 0 0
27 2 Low Latency Yes 5500 5500 0 0
28 10 EGR Exception Yes 250 250 0 0
29 5 Stackwise Virtual OOB Yes 8000 8000 0 0
30 9 MCAST Data Yes 500 500 0 0
31 3 Gold Pkt Yes 1000 1000 0 0

* NOTE: CPU queue policer rates are configured to the closest hardware supported value

CPU Queue Policer Statistics
====================================================================
Policer Policer Accept Policer Accept Policer Drop Policer Drop
Index Bytes Frames Bytes Frames
-------------------------------------------------------------------
0 70662468 553388 0 0
1 38064542 221811 0 0
2 208992 3264 0 0
3 2584 38 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 50274798 468879 0 0
9 0 0 0 0
10 0 0 0 0
11 0 0 0 0
12 0 0 0 0
13 11650 104 0 0
14 402707134 6207577 0 0
15 9942556 107046 0 0
16 0 0 0 0
17 0 0 0 0
18 0 0 0 0
19 0 0 0 0

Second Level Policer Statistics
====================================================================
20 88548332 693954 0 0
21 483323808 6868115 0 0

Policer Index Mapping and Settings
--------------------------------------------------------------------
level-2 : level-1 (default) (set)
PlcIndex : PlcIndex rate rate
--------------------------------------------------------------------
20 : 1 2 8 13000 13000
21 : 0 4 7 9 10 11 12 13 14 15 6000 6000
====================================================================

Second Level Policer Config
====================================================================
level-1 level-2 level-2
QId PlcIdx PlcIdx Queue Name Enabled
--------------------------------------------------------------------
0 11 21 DOT1X Auth Yes
1 1 20 L2 Control Yes
2 14 21 Forus traffic Yes
3 0 21 ICMP GEN Yes
4 2 20 Routing Control Yes
5 14 21 Forus Address resolution Yes
6 0 21 ICMP Redirect Yes
7 16 - Inter FED Traffic No
8 4 21 L2 LVX Cont Pack Yes
9 19 - EWLC Control No
10 16 - EWLC Data No
11 13 21 L2 LVX Data Pack Yes
12 0 21 BROADCAST Yes
13 10 21 Openflow Yes
14 13 21 Sw forwarding Yes
15 8 20 Topology Control Yes
16 12 21 Proto Snooping Yes
17 6 - DHCP Snooping No
18 13 21 Transit Traffic Yes
19 10 21 RPF Failed Yes
20 15 21 MCAST END STATION Yes
21 13 21 LOGGING Yes
22 7 21 Punt Webauth Yes
23 18 - High Rate App No
24 10 21 Exception Yes
25 3 - System Critical No
26 10 21 NFL SAMPLED DATA Yes
27 2 20 Low Latency Yes
28 10 21 EGR Exception Yes
29 5 - Stackwise Virtual OOB No
30 9 21 MCAST Data Yes
31 3 - Gold Pkt No

CPP Classes to queue map
======================================================================================
PlcIdx CPP Class : Queues
--------------------------------------------------------------------------------------
0 system-cpp-police-data : ICMP GEN/ BROADCAST/ ICMP Redirect/
10 system-cpp-police-sys-data : Openflow/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/
13 system-cpp-police-sw-forward : Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Transit Traffic/
9 system-cpp-police-multicast : MCAST Data/
15 system-cpp-police-multicast-end-station : MCAST END STATION /
7 system-cpp-police-punt-webauth : Punt Webauth/
1 system-cpp-police-l2-control : L2 Control/
2 system-cpp-police-routing-control : Routing Control/ Low Latency/
3 system-cpp-police-system-critical : System Critical/ Gold Pkt/
4 system-cpp-police-l2lvx-control : L2 LVX Cont Pack/
8 system-cpp-police-topology-control : Topology Control/
11 system-cpp-police-dot1x-auth : DOT1X Auth/
12 system-cpp-police-protocol-snooping : Proto Snooping/
6 system-cpp-police-dhcp-snooping : DHCP Snooping/
14 system-cpp-police-forus : Forus Address resolution/ Forus traffic/
5 system-cpp-police-stackwise-virt-control : Stackwise Virtual OOB/
16 system-cpp-default : Inter FED Traffic/ EWLC Data/
18 system-cpp-police-high-rate-app : High Rate App/
19 system-cpp-police-ewlc-control : EWLC Control/
20 system-cpp-police-ios-routing : L2 Control/ Topology Control/ Routing Control/ Low Latency/
21 system-cpp-police-ios-feature : ICMP GEN/ BROADCAST/ ICMP Redirect/ L2 LVX Cont Pack/ Proto Snooping/ Punt Webauth/ MCAST Data/ Transit Traffic/ DOT1X Auth/ Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Forus traffic/ Forus Address resolution/ MCAST END STATION / Openflow/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/

[balaji.bandi]

You need to provide more information here :

1. what is the Full model of the switch ?

2. what IOS Xe code running on it ?

3. what interface connected to esxi ? is the Bundle interface or single ? can you post the configuration ?

4. what Esxi version ?

5. what interface card you using on esxi ? what speed ?

6. show interface x/x (from switch help us)

7. the Ping you doing from Switch to VM host or exsi ?

8. what is the VM host windows Server or Linux ?

[santi8809]

1. what is the Full model of the switch ? C9300X-48HX
2. what IOS Xe code running on it ?  17.09.04a CAT9K_IOSXE 
3. what interface connected to esxi ? is the Bundle interface or single ? can you post the configuration ? Single
4. what Esxi version ? 8.0.2
5. what interface card you using on esxi ? what speed ? 10GB (Additional CARD)
6. show interface x/x (from switch help us) 

   01#sh run int te1/0/6
   Building configuration...

   Current configuration : 149 bytes
   !
   interface TenGigabitEthernet1/0/6
   description Uplink-HV2-Vmnic2
   switchport trunk allowed vlan 1,50
   switchport mode trunk
   end

7. the Ping you doing from Switch to VM host or exsi ? From Switch to IP configured on ESXI Host
8. what is the VM host windows Server or Linux ? Linux

[balaji.bandi]

post this output : show interface te1/0/6

what vlan exsi and vm host belong to ?

what is the outcome from esxi IP address to host sitting inside Linux box ?

check couple of settings on esxi :

https://kb.vmware.com/s/article/2039495

https://kb.vmware.com/s/article/1004109