Strange Issue on c9200L stack

[Support ACME]

Dear ALL,

We are using the 2 x C9200L-24T-4X(17.12.05)  as stackble L3 core switch(Hold all subnet gateway) and has  4 x C2960 switch as access layer switch(using ether-channel connece to c9200L), the spanning-tree mode is rapid-pvst. We found our network will randomly disconnect around 1-2 min in specify VLAN, then I had setup the server in outside  and using "pinginfoview" software to monitor the  VLANIP under C9200 , I sure this issue only happen in specify VLAN, i have no idea to solve this problem, anyone can help?

[MHM Cisco World]

last chnage occurred 5d ago and after that the disconnect is 3-4 times every day?

check MAC if it correct or not 

MHM

[Support ACME]

last chnage occurred 5d ago and after that the disconnect is 3-4 times every day?

Yes

check MAC if it correct or not 

your means from the host to gateway MAC ,right?

[MHM Cisco World]

Yes 

[Support ACME]

now is normal status, the MAC address is correct. 

[MHM Cisco World]

pinginfoview <<- this key of your issue

When host can not reach GW 

You ping 

Then it successful reach GW

Ping is to many host in same subnet, 

One of them is GW (vlan SVI)

So there is either

Vlan SVI not allow in trunk <<- we check this point

Or 

Host or access SW delete mac or use not correct Mac, which solve by ping. 

When issue happened again, check mac in host and access SW, see it correct point to vlan SVI of stackswitch 

MHM

[Leo Laohoo]

---

@Support ACME wrote:
We found our network will randomly disconnect around 1-2 min in specify VLAN

---

Smells like a network storm.

[Support ACME]

Ｉ did some testing, i setup the two PC(PC1 and PC2) in  VLAN100-PC1 and VLAN101-PC2, then PC1  use  "ＰingＩnfoＶiew" to ping the default gateway(this IP is VLAN interface IP) and the PC2 host, i got packet loss in default gateway, no the PC2 host is any packet lost. 

Is there  limit control of the icmp connection in C9200L?

[MHM Cisco World]

You add two PC in different vlan 

One vlan PC can ping GW 

Other vlan PC can not ping GW ??

MHM

[Support ACME]

I have PC1 in VLAN100:192.168.100.10 & GW:192.168.100.254 and  PC2 in VLAN200:192.168.200.10 GW:192.168.200.254 & GW:192.168.200.254, I used PC1 to ping 192.168.100.254,192.168.200.254 and 192.168.200.10, i only got packet lost on 192.168.100.254,192.168.200.254, but 192.168.200.10 is no any packet lost.

[MHM Cisco World]

I think both issue is relate 
#show plat hardware fed switch active qos queue stats internal cpu policer <<- share this' two times 

One before ping 

Other after ping 100 (repeat 100)

MHM

[Support ACME]

Before:

show plat hardware fed switch active qos queue stats internal cpu policer

CPU Queue Statistics
============================================================================================
(default) (set) Queue Queue
QId PlcIdx Queue Name Enabled Rate Rate Drop(Bytes) Drop(Frames)
--------------------------------------------------------------------------------------------
0 11 DOT1X Auth Yes 800 800 0 0
1 1 L2 Control Yes 2000 2000 0 0
2 14 Forus traffic Yes 1000 1000 48662259 660838
3 0 ICMP GEN Yes 600 600 0 0
4 2 Routing Control Yes 1000 1000 0 0
5 14 Forus Address resolution Yes 1000 1000 21373228 333547
6 0 ICMP Redirect Yes 600 600 1102682279402 16531726238
7 16 Inter FED Traffic Yes 100 200 0 0
8 4 L2 LVX Cont Pack Yes 500 400 0 0
9 19 EWLC Control Yes 100 1000 0 0
10 16 EWLC Data Yes 100 200 0 0
11 13 L2 LVX Data Pack Yes 800 800 0 0
12 0 BROADCAST Yes 600 600 178640400 2179334
13 10 Openflow Yes 100 200 0 0
14 13 Sw forwarding Yes 800 800 1195589460 17771316
15 8 Topology Control Yes 6000 6000 0 0
16 12 Proto Snooping Yes 1000 1000 0 0
17 6 DHCP Snooping Yes 500 400 0 0
18 13 Transit Traffic Yes 800 800 0 0
19 10 RPF Failed Yes 100 200 0 0
20 15 MCAST END STATION Yes 1000 1000 42610 427
21 13 LOGGING Yes 800 800 0 0
22 7 Punt Webauth Yes 800 800 0 0
23 18 High Rate App Yes 3000 3000 0 0
24 10 Exception Yes 100 200 0 0
25 3 System Critical Yes 100 200 0 0
26 10 NFL SAMPLED DATA Yes 100 200 0 0
27 2 Low Latency Yes 1000 1000 0 0
28 10 EGR Exception Yes 100 200 1464648900 21909177
29 5 Stackwise Virtual OOB Yes 100 200 0 0
30 9 MCAST Data Yes 200 200 0 0
31 3 Gold Pkt Yes 100 200 0 0

* NOTE: CPU queue policer rates are configured to the closest hardware supported value

CPU Queue Policer Statistics
====================================================================
Policer Policer Accept Policer Accept Policer Drop Policer Drop
Index Bytes Frames Bytes Frames
-------------------------------------------------------------------
0 59646056665 865180728 1102844937661 16533675261
1 503750741 3120722 0 0
2 702669654 7712747 0 0
3 3536 52 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 1171432484 16424580 0 0
9 0 0 0 0
10 4920868880 70089821 1423667161 21306408
11 0 0 0 0
12 0 0 0 0
13 10442189418 143473728 743156918 10974956
14 6434501840 75658919 0 0
15 5130424 51182 0 0
16 0 0 0 0
17 0 0 0 0
18 0 0 0 0
19 0 0 0 0

Second Level Policer Statistics
====================================================================
20 2377852879 27258049 0 0
21 80869272708 1145830126 579474519 8624252

Policer Index Mapping and Settings
--------------------------------------------------------------------
level-2 : level-1 (default) (set)
PlcIndex : PlcIndex rate rate
--------------------------------------------------------------------
20 : 1 2 8 6000 6000
21 : 0 4 7 9 10 11 12 13 14 15 1000 1000
====================================================================

Second Level Policer Config
====================================================================
level-1 level-2 level-2
QId PlcIdx PlcIdx Queue Name Enabled
--------------------------------------------------------------------
0 11 21 DOT1X Auth Yes
1 1 20 L2 Control Yes
2 14 21 Forus traffic Yes
3 0 21 ICMP GEN Yes
4 2 20 Routing Control Yes
5 14 21 Forus Address resolution Yes
6 0 21 ICMP Redirect Yes
7 16 - Inter FED Traffic No
8 4 21 L2 LVX Cont Pack Yes
9 19 - EWLC Control No
10 16 - EWLC Data No
11 13 21 L2 LVX Data Pack Yes
12 0 21 BROADCAST Yes
13 10 21 Openflow Yes
14 13 21 Sw forwarding Yes
15 8 20 Topology Control Yes
16 12 21 Proto Snooping Yes
17 6 - DHCP Snooping No
18 13 21 Transit Traffic Yes
19 10 21 RPF Failed Yes
20 15 21 MCAST END STATION Yes
21 13 21 LOGGING Yes
22 7 21 Punt Webauth Yes
23 18 - High Rate App No
24 10 21 Exception Yes
25 3 - System Critical No
26 10 21 NFL SAMPLED DATA Yes
27 2 20 Low Latency Yes
28 10 21 EGR Exception Yes
29 5 - Stackwise Virtual OOB No
30 9 21 MCAST Data Yes
31 3 - Gold Pkt No

CPP Classes to queue map
======================================================================================
PlcIdx CPP Class : Queues
--------------------------------------------------------------------------------------
0 system-cpp-police-data : ICMP GEN/ BROADCAST/ ICMP Redirect/
10 system-cpp-police-sys-data : Openflow/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/
13 system-cpp-police-sw-forward : Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Transit Traffic/
9 system-cpp-police-multicast : MCAST Data/
15 system-cpp-police-multicast-end-station : MCAST END STATION /
7 system-cpp-police-punt-webauth : Punt Webauth/
1 system-cpp-police-l2-control : L2 Control/
2 system-cpp-police-routing-control : Routing Control/ Low Latency/
3 system-cpp-police-system-critical : System Critical/ Gold Pkt/
4 system-cpp-police-l2lvx-control : L2 LVX Cont Pack/
8 system-cpp-police-topology-control : Topology Control/
11 system-cpp-police-dot1x-auth : DOT1X Auth/
12 system-cpp-police-protocol-snooping : Proto Snooping/
6 system-cpp-police-dhcp-snooping : DHCP Snooping/
14 system-cpp-police-forus : Forus Address resolution/ Forus traffic/
5 system-cpp-police-stackwise-virt-control : Stackwise Virtual OOB/
16 system-cpp-default : Inter FED Traffic/ EWLC Data/
18 system-cpp-police-high-rate-app : High Rate App/
19 system-cpp-police-ewlc-control : EWLC Control/
20 system-cpp-police-ios-routing : L2 Control/ Topology Control/ Routing Control/ Low Latency/
21 system-cpp-police-ios-feature : ICMP GEN/ BROADCAST/ ICMP Redirect/ L2 LVX Cont Pack/ Proto Snooping/ Punt Webauth/ MCAST Data/ Transit Traffic/ DOT1X Auth/ Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Forus traffic/ Forus Address resolution/ MCAST END STATION / Openflow/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/

After:

show plat hardware fed switch active qos queue stats internal cpu policer

CPU Queue Statistics
============================================================================================
(default) (set) Queue Queue
QId PlcIdx Queue Name Enabled Rate Rate Drop(Bytes) Drop(Frames)
--------------------------------------------------------------------------------------------
0 11 DOT1X Auth Yes 800 800 0 0
1 1 L2 Control Yes 2000 2000 0 0
2 14 Forus traffic Yes 1000 1000 48662259 660838
3 0 ICMP GEN Yes 600 600 0 0
4 2 Routing Control Yes 1000 1000 0 0
5 14 Forus Address resolution Yes 1000 1000 21373228 333547
6 0 ICMP Redirect Yes 600 600 1102838772708 16534027150
7 16 Inter FED Traffic Yes 100 200 0 0
8 4 L2 LVX Cont Pack Yes 500 400 0 0
9 19 EWLC Control Yes 100 1000 0 0
10 16 EWLC Data Yes 100 200 0 0
11 13 L2 LVX Data Pack Yes 800 800 0 0
12 0 BROADCAST Yes 600 600 178666120 2179620
13 10 Openflow Yes 100 200 0 0
14 13 Sw forwarding Yes 800 800 1195589460 17771316
15 8 Topology Control Yes 6000 6000 0 0
16 12 Proto Snooping Yes 1000 1000 0 0
17 6 DHCP Snooping Yes 500 400 0 0
18 13 Transit Traffic Yes 800 800 0 0
19 10 RPF Failed Yes 100 200 0 0
20 15 MCAST END STATION Yes 1000 1000 42610 427
21 13 LOGGING Yes 800 800 0 0
22 7 Punt Webauth Yes 800 800 0 0
23 18 High Rate App Yes 3000 3000 0 0
24 10 Exception Yes 100 200 0 0
25 3 System Critical Yes 100 200 0 0
26 10 NFL SAMPLED DATA Yes 100 200 0 0
27 2 Low Latency Yes 1000 1000 0 0
28 10 EGR Exception Yes 100 200 1464655004 21909253
29 5 Stackwise Virtual OOB Yes 100 200 0 0
30 9 MCAST Data Yes 200 200 0 0
31 3 Gold Pkt Yes 100 200 0 0

* NOTE: CPU queue policer rates are configured to the closest hardware supported value

CPU Queue Policer Statistics
====================================================================
Policer Policer Accept Policer Accept Policer Drop Policer Drop
Index Bytes Frames Bytes Frames
-------------------------------------------------------------------
0 59656540081 865330710 1103001456687 16535976459
1 503839242 3121267 0 0
2 702787910 7714027 0 0
3 3536 52 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
8 1171643984 16427538 0 0
9 0 0 0 0
10 4921658319 70100876 1423673265 21306484
11 0 0 0 0
12 0 0 0 0
13 10443772453 143495071 743156918 10974956
14 6435433516 75668890 0 0
15 5130424 51182 0 0
16 0 0 0 0
17 0 0 0 0
18 0 0 0 0
19 0 0 0 0

Second Level Policer Statistics
====================================================================
20 2378271136 27262832 0 0
21 80883060274 1146022477 579474519 8624252

Policer Index Mapping and Settings
--------------------------------------------------------------------
level-2 : level-1 (default) (set)
PlcIndex : PlcIndex rate rate
--------------------------------------------------------------------
20 : 1 2 8 6000 6000
21 : 0 4 7 9 10 11 12 13 14 15 1000 1000
====================================================================

Second Level Policer Config
====================================================================
level-1 level-2 level-2
QId PlcIdx PlcIdx Queue Name Enabled
--------------------------------------------------------------------
0 11 21 DOT1X Auth Yes
1 1 20 L2 Control Yes
2 14 21 Forus traffic Yes
3 0 21 ICMP GEN Yes
4 2 20 Routing Control Yes
5 14 21 Forus Address resolution Yes
6 0 21 ICMP Redirect Yes
7 16 - Inter FED Traffic No
8 4 21 L2 LVX Cont Pack Yes
9 19 - EWLC Control No
10 16 - EWLC Data No
11 13 21 L2 LVX Data Pack Yes
12 0 21 BROADCAST Yes
13 10 21 Openflow Yes
14 13 21 Sw forwarding Yes
15 8 20 Topology Control Yes
16 12 21 Proto Snooping Yes
17 6 - DHCP Snooping No
18 13 21 Transit Traffic Yes
19 10 21 RPF Failed Yes
20 15 21 MCAST END STATION Yes
21 13 21 LOGGING Yes
22 7 21 Punt Webauth Yes
23 18 - High Rate App No
24 10 21 Exception Yes
25 3 - System Critical No
26 10 21 NFL SAMPLED DATA Yes
27 2 20 Low Latency Yes
28 10 21 EGR Exception Yes
29 5 - Stackwise Virtual OOB No
30 9 21 MCAST Data Yes
31 3 - Gold Pkt No

CPP Classes to queue map
======================================================================================
PlcIdx CPP Class : Queues
--------------------------------------------------------------------------------------
0 system-cpp-police-data : ICMP GEN/ BROADCAST/ ICMP Redirect/
10 system-cpp-police-sys-data : Openflow/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/
13 system-cpp-police-sw-forward : Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Transit Traffic/
9 system-cpp-police-multicast : MCAST Data/
15 system-cpp-police-multicast-end-station : MCAST END STATION /
7 system-cpp-police-punt-webauth : Punt Webauth/
1 system-cpp-police-l2-control : L2 Control/
2 system-cpp-police-routing-control : Routing Control/ Low Latency/
3 system-cpp-police-system-critical : System Critical/ Gold Pkt/
4 system-cpp-police-l2lvx-control : L2 LVX Cont Pack/
8 system-cpp-police-topology-control : Topology Control/
11 system-cpp-police-dot1x-auth : DOT1X Auth/
12 system-cpp-police-protocol-snooping : Proto Snooping/
6 system-cpp-police-dhcp-snooping : DHCP Snooping/
14 system-cpp-police-forus : Forus Address resolution/ Forus traffic/
5 system-cpp-police-stackwise-virt-control : Stackwise Virtual OOB/
16 system-cpp-default : Inter FED Traffic/ EWLC Data/
18 system-cpp-police-high-rate-app : High Rate App/
19 system-cpp-police-ewlc-control : EWLC Control/
20 system-cpp-police-ios-routing : L2 Control/ Topology Control/ Routing Control/ Low Latency/
21 system-cpp-police-ios-feature : ICMP GEN/ BROADCAST/ ICMP Redirect/ L2 LVX Cont Pack/ Proto Snooping/ Punt Webauth/ MCAST Data/ Transit Traffic/ DOT1X Auth/ Sw forwarding/ LOGGING/ L2 LVX Data Pack/ Forus traffic/ Forus Address resolution/ MCAST END STATION / Openflow/ Exception/ EGR Exception/ NFL SAMPLED DATA/ RPF Failed/
ACME-C9200L-Public-HK-KLNB-AC21#

[MHM Cisco World]

2 14 Forus traffic Yes 1000 1000 48662259 660838
5 14 Forus Address resolution Yes 1000 1000 21373228 333547
6 0 ICMP Redirect Yes 600 600 1102682279402 16531726238
12 0 BROADCAST Yes 600 600 178640400 2179334

2 14 Forus traffic Yes 1000 1000 48662259 660838
5 14 Forus Address resolution Yes 1000 1000 21373228 333547
6 0 ICMP Redirect Yes 600 600 1102838772708 16534027150
12 0 BROADCAST Yes 600 600 178666120 2179620

this ping success or not ? forus traffic (i.e. ping to SVI) is same before and after ??

there is huge number of ICMP redirect ? 
broadcast there is 300 drop but I dont think it huge issue 

MHM

[Support ACME]

this ping success or not ? forus traffic (i.e. ping to SVI) is same before and after ??

This two times ping is success.

there is huge number of ICMP redirect ? 

Yes, this switch is the layer3 switch. 
broadcast there is 300 drop but I dont think it huge issue