Hi All,
This morning, we have migrated from an old c3750 switch to a new c9500 stack solution, however, we are facing with a strange issue.
We use the equipment(s) for multicast traffic, and after the migration we noticed packet loss and drops on the control plane. This also affects the multicast/streaming service, some end-devices show "RTP sequence error".
We are suspecting these drops are being caused by the default cpu policer, however, when we have tried to increase the queue rate to 8000, the management of the sw did not respond. We had to console on to the sw and change the value back to the original one.
Have you ever experienced such a problem like this ?
Some outputs:
As you can see the number of drops is huge in Queue RPF failed and MCAST DATA. What can cause this ?
c9500-sw#sh platform hardware fed switch 2 qos queue stats internal cpu policer
CPU Queue Statistics
============================================================================================
(default) (set) Queue Queue
QId PlcIdx Queue Name Enabled Rate Rate Drop(Bytes) Drop(Frames)
--------------------------------------------------------------------------------------------
0 11 DOT1X Auth Yes 1000 1000 0 0
1 1 L2 Control Yes 2000 2000 0 0
2 14 Forus traffic Yes 4000 4000 0 0
3 0 ICMP GEN Yes 750 750 0 0
4 2 Routing Control Yes 5500 5500 0 0
5 14 Forus Address resolution Yes 4000 4000 0 0
6 0 ICMP Redirect Yes 750 750 0 0
7 16 Inter FED Traffic Yes 2000 2000 0 0
8 4 L2 LVX Cont Pack Yes 1000 1000 0 0
9 19 EWLC Control Yes 13000 13000 0 0
10 16 EWLC Data Yes 2000 2000 0 0
11 13 L2 LVX Data Pack Yes 1000 1000 0 0
12 0 BROADCAST Yes 750 750 0 0
13 10 Openflow Yes 250 1000 0 0
14 13 Sw forwarding Yes 1000 1000 0 0
15 8 Topology Control Yes 13000 16000 0 0
16 12 Proto Snooping Yes 2000 2000 0 0
17 6 DHCP Snooping Yes 500 500 0 0
18 13 Transit Traffic Yes 1000 1000 0 0
19 10 RPF Failed Yes 250 1000 1163084619371 683470015
20 15 MCAST END STATION Yes 2000 2000 0 0
21 13 LOGGING Yes 1000 1000 0 0
22 7 Punt Webauth Yes 1000 1000 0 0
23 18 High Rate App Yes 13000 13000 0 0
24 10 Exception Yes 250 1000 0 0
25 3 System Critical Yes 1000 1000 0 0
26 10 NFL SAMPLED DATA Yes 250 1000 0 0
27 2 Low Latency Yes 5500 5500 0 0
28 10 EGR Exception Yes 250 1000 0 0
29 5 Stackwise Virtual OOB Yes 8000 8000 0 0
30 9 MCAST Data Yes 500 1000 14651359982 10118797
31 3 Gold Pkt Yes 1000 1000 0 0
* NOTE: CPU queue policer rates are configured to the closest hardware supported value
CPU Queue Policer Statistics
====================================================================
Policer Policer Accept Policer Accept Policer Drop Policer Drop
Index Bytes Frames Bytes Frames
-------------------------------------------------------------------
0 22787 73 0 0
1 1404977 10536 0 0
2 15951 224 0 0
3 3700714 10901 0 0
4 0 0 0 0
5 2058524250 3719854 0 0
6 0 0 0 0
7 0 0 0 0
8 248547 2448 0 0
9 3373800700 2451384 14651359982 10118797
10 4155548760 2451384 1163084731003 683470085
11 0 0 0 0
12 0 0 0 0
13 118 1 0 0
14 1171146 12179 0 0
15 325970 4920 0 0
16 0 0 0 0
17 0 0 0 0
18 0 0 0 0
19 0 0 0 0
Second Level Policer Statistics
====================================================================
20 1669475 13208 0 0
21 7530628151 4919800 0 0
c9500-sw#ping 10.10.25.46 sou vlan40 rep 400
Type escape sequence to abort.
Sending 400, 100-byte ICMP Echos to 10.10.25.46, timeout is 2 seconds:
Packet sent with a source address of 10.10.25.62
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!.!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!
!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!!!!.!!.!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!
Success rate is 96 percent (384/400), round-trip min/avg/max = 1/1/5 ms
c9500-sw#show platform software fed switch active cpu-interface
queue retrieved dropped invalid hol-block
-------------------------------------------------------------------------
Routing Protocol 102685 0 0 0
L2 Protocol 10812722 0 0 0
sw forwarding 386 3 0 0
broadcast 16479 0 0 0
icmp gen 0 0 0 0
icmp redirect 0 0 0 0
logging 0 0 0 0
rpf-fail 161538486 0 0 0
DOT1X authentication 0 0 0 0
Forus Traffic 6146593 0 0 0
Forus Resolution 127235 0 0 0
Inter FED 0 0 0 0
L2 LVX control 0 0 0 0
EWLC control 0 0 0 0
EWLC data 0 0 0 0
L2 LVX data 0 0 0 0
Openflow 0 0 0 0
Topology control 966672 0 0 0
Proto snooping 0 0 0 0
DHCP snooping 0 0 0 0
Transit Traffic 0 0 0 0
Multi End station 903713 0 0 0
Webauth 0 0 0 0
High rate app 0 0 0 0
Exception 0 0 0 0
System Critical 5698491 0 0 0
NFL sampled data 0 0 0 0
Low latency 0 0 0 0
EGR exception 6 0 0 0
Stackwise Virtual OOB 440307201 0 0 0
Multicast data 311391329 0 0 0
Gold packet 52 0 0 0
CPU utilisation is not more than 50%.
How could we resolve this problem ?
Thank you very much for your help and support in advance.
