Re: nexus3548 multicast packet drop issue

johnyim12345678 · ‎02-13-2022

Hello, I work at a fintech company, and we provide our clients with real-time market data through our nexus 3548 switches. We have been suffering from a significant level of packet drop, and despite the fact we have been troubleshooting on this so long, the issue is still not resolved. We are highly doubting if this might be because of buffer or queue limit configuration, while we haven't changed any default settings regarding this parts. Funny thing here is while some interfaces are not having any issue, the other ones such as eth1/7 are under a severe issue.

Let me attach the results of some commands below.

MDSW-SND1# show interface queuing-drop history brief multicast

slot 1
=======

Multicast Drop History CLI issued at: 02/14/2022 10:39:25
Last cleared at: 02/22/2001 06:43:05
Last sampled at: 02/14/2022 10:39:22
5sec 10sec 60sec 5min 1hr
---------- ---------- ---------- ---------- ----------
Ethernet1/1 0 0 0 0 0
Ethernet1/2 0 0 0 0 0
Ethernet1/3 0 0 0 0 0
Ethernet1/4 0 0 0 0 0
Ethernet1/5 0 0 0 0 0
Ethernet1/6 0 0 0 0 0
Ethernet1/7 0 0 2541 44339 209503
Ethernet1/8 0 0 0 0 0
Ethernet1/9 0 0 9 265 1278
Ethernet1/10 0 0 0 0 0
Ethernet1/11 0 0 0 0 0
Ethernet1/12 0 0 1847 45009 208830
Ethernet1/13 0 0 0 0 0
Ethernet1/14 0 0 0 0 0
Ethernet1/15 0 0 0 0 0
Ethernet1/16 0 0 0 0 0
Ethernet1/17 0 0 2591 46228 225459
Ethernet1/18 0 0 2537 43674 206477
Ethernet1/19 0 0 2718 47635 233091
Ethernet1/20 0 0 2718 47635 233091
Ethernet1/21 0 0 0 0 0
Ethernet1/22 0 0 1847 45009 208830
Ethernet1/23 0 0 0 0 0
Ethernet1/24 0 0 0 0 0
Ethernet1/25 0 0 0 0 0
Ethernet1/26 0 0 0 0 0
Ethernet1/27 0 0 0 0 0
Ethernet1/28 0 0 0 0 0
Ethernet1/29 0 0 0 0 0
Ethernet1/30 0 0 0 0 0
Ethernet1/31 0 0 0 0 0
Ethernet1/32 0 0 0 0 0
Ethernet1/33 0 0 0 0 0
Ethernet1/34 0 0 0 0 0
Ethernet1/35 0 0 0 0 0
Ethernet1/36 0 0 0 0 0
Ethernet1/37 0 0 0 0 0
Ethernet1/38 0 0 0 0 0
Ethernet1/39 0 0 0 0 0
Ethernet1/40 0 0 0 0 0
Ethernet1/41 0 0 0 0 0
Ethernet1/42 0 0 0 0 0
Ethernet1/43 0 0 0 0 0
Ethernet1/44 0 0 0 0 0
Ethernet1/45 0 0 0 0 0
Ethernet1/46 0 0 0 0 0
Ethernet1/47 0 0 0 0 0
Ethernet1/48 0 0 0 0 0

MDSW-SND1# show int status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
mgmt0 -- notconnec routed auto auto --

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 -- xcvrAbsen 10 auto auto --
Eth1/2 CME01:eth2 xcvrAbsen 10 auto auto --
Eth1/3 -- notconnec 15 auto 1000 1000base-T
Eth1/4 CON3:p2p2 notconnec 85 auto auto 1000base-T
Eth1/5 CON4:p2p2 xcvrAbsen 192 auto auto --
Eth1/6 -- xcvrAbsen 1 auto 1000 --
Eth1/7 Meritz (192.168.23 connected 235 full 10G 10Gbase-LR
Eth1/8 -- xcvrAbsen 1 auto 100 --
Eth1/9 SAMSUNG (192.168.2 connected 250 full 100 1000base-T
Eth1/10 CME03:em3 connected 221 full 1000 1000base-T
Eth1/11 -- notconnec routed auto 100 1000base-T
Eth1/12 KIS (192.168.220.2 connected 220 full 1000 1000base-SX
Eth1/13 CMEDEV:eth2 connected 100 full 1000 1000base-T
Eth1/14 CME05:em4 connected 215 full 1000 1000base-T
Eth1/15 -- xcvrAbsen 210 auto 100 --
Eth1/16 -- xcvrAbsen 210 auto 100 --
Eth1/17 CON3:p2p1 connected 192 full 1000 1000base-T
Eth1/18 HyunDai (192.168.2 connected 201 full 100 1000base-T
Eth1/19 Shinhan (192.168.2 connected 205 full 100 1000base-T
Eth1/20 KYOBO (192.168.210 connected 210 full 100 1000base-T
Eth1/21 -- xcvrAbsen 85 auto 100 --
Eth1/22 SK connected 238 full 100 1000base-T
Eth1/23 -- xcvrAbsen 230 auto 100 --
Eth1/24 Yuanta Backup (192 connected 230 full 1000 1000base-T
Eth1/25 Yuanta Primary (19 connected 230 full 1000 1000base-T
Eth1/26 -- notconnec 240 auto 100 1000base-T
Eth1/27 -- notconnec 216 auto 100 1000base-T
Eth1/28 EBEST (192.168.225 connected 225 full 100 1000base-T
Eth1/29 CME20:em3 connected 30 full 1000 1000base-T
Eth1/30 -- xcvrAbsen 1 auto auto --
Eth1/31 -- xcvrAbsen 1 auto auto --
Eth1/32 -- xcvrAbsen 1 auto auto --
Eth1/33 -- xcvrAbsen 1 auto auto --
Eth1/34 -- xcvrAbsen 1 auto auto --
Eth1/35 -- xcvrAbsen 150 auto 100 --
Eth1/36 -- xcvrAbsen 251 auto 100 --
Eth1/37 MIRAE connected 251 full 100 1000base-T
Eth1/38 -- xcvrAbsen 1 auto auto --
Eth1/39 -- xcvrAbsen 1 auto auto --
Eth1/40 ## KB 200MB ## connected 240 full 1000 1000base-T
Eth1/41 -- xcvrAbsen 1 auto auto --
Eth1/42 -- xcvrAbsen 1 auto auto --
Eth1/43 -- xcvrAbsen 1 auto auto --
Eth1/44 -- xcvrAbsen 1 auto auto --
Eth1/45 -- xcvrAbsen 1 auto auto --
Eth1/46 -- xcvrAbsen 1 auto auto --
Eth1/47 -- xcvrAbsen 1 auto 1000 --
Eth1/48 -- xcvrAbsen 1 auto 1000 --

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Vlan1 -- down routed auto auto --
Vlan10 ## send vlan 10.60 down routed auto auto --
Vlan15 ## test send vlan down routed auto auto --
Vlan16 ## inter vlan test down routed auto auto --
Vlan30 ## CME20 10.60.30. connected routed auto auto --
Vlan80 -- down routed auto auto --
Vlan85 ## tcp send vlan # down routed auto auto --
Vlan100 -- connected routed auto auto --
Vlan150 ## CME05 192.168.1 down routed auto auto --
Vlan192 ## tcp recv vlan # connected routed auto auto --
Vlan195 ## CME01_recv ## down routed auto auto --
Vlan201 ## HyundaiFutures connected routed auto auto --
Vlan203 ## Leading 192.168 down routed auto auto --
Vlan205 ## Shinhan 192.168 connected routed auto auto --
Vlan206 -- down routed auto auto --
Vlan210 ## KYOBO ## connected routed auto auto --
Vlan215 ## CME05 192.168.2 connected routed auto auto --
Vlan216 ## Kr Futures 192. down routed auto auto --
Vlan220 ## KIS_Future ## connected routed auto auto --
Vlan221 ## KIS_Option ## connected routed auto auto --
Vlan222 ## KIS_TEST ## down routed auto auto --
Vlan225 ## Ebestsec 192.16 connected routed auto auto --
Vlan230 ## Yuanta 192.168. connected routed auto auto --
Vlan235 ##meritz 192.168.2 connected routed auto auto --
Vlan238 SK connected routed auto auto --
Vlan240 ## HDS ## connected routed auto auto --
Vlan250 ##SAMSUNG FUTURE 1 connected routed auto auto --
Vlan251 ## Mirae 192.168.2 connected routed auto auto --

MDSW-SND1# show queuing interface eth1/7

slot 1
=======

HW MTU of Ethernet1/7 : 1500 bytes

Egress Queuing for Ethernet1/7 [System]
------------------------------------------------------------------------------
QoS-Group# Bandwidth% PrioLevel Shape QLimit
Min Max Units
------------------------------------------------------------------------------
0 100 - - - - 5201691(S)

Mcast pkts dropped : 293304266

+-------------------------------------------------------------------+
| QOS GROUP 0 |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Dropped Pkts | 0| 0| 0|
+-------------------------------------------------------------------+
| QOS GROUP 1 |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Dropped Pkts | 0| 0| 0|
+-------------------------------------------------------------------+
| QOS GROUP 2 |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Dropped Pkts | 0| 0| 0|
+-------------------------------------------------------------------+
| QOS GROUP 3 |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Dropped Pkts | 0| 0| 0|
+-------------------------------------------------------------------+
| QOS GROUP 4 |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Dropped Pkts | 0| 0| 0|
+-------------------------------------------------------------------+
| QOS GROUP 5 |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Dropped Pkts | 0| 0| 0|
+-------------------------------------------------------------------+
| QOS GROUP 6 |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Dropped Pkts | 0| 0| 0|
+-------------------------------------------------------------------+
| QOS GROUP 7 |
+-------------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+-------------------------------------------------------------------+
| Dropped Pkts | 0| 0| 0|
+-------------------------------------------------------------------+

Ingress Queuing for Ethernet1/7
------------------------------------------------------------------
QoS-Group# Pause QLimit
Buff Size Pause Th Resume Th
------------------------------------------------------------------
7 - - - 0(S)
6 - - - 0(S)
5 - - - 0(S)
4 - - - 0(S)
3 - - - 0(S)
2 - - - 0(S)
1 - - - 0(S)
0 - - - 0(S)

PFC Statistics
----------------------------------------------------------------------------
TxPPP: 0, RxPPP: 0
----------------------------------------------------------------------------
COS QOS Group TxCount RxCount
0 - 0 0
1 - 0 0
2 - 0 0
3 - 0 0
4 - 0 0
5 - 0 0
6 - 0 0
7 - 0 0
----------------------------------------------------------------------------

MDSW-SND1# show int counters errors

--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
mgmt0 0 0 -- -- -- --
Eth1/1 0 0 0 0 0 0
Eth1/2 0 0 0 0 0 5005
Eth1/3 0 0 0 0 0 0
Eth1/4 0 0 0 0 0 0
Eth1/5 0 0 0 0 0 48050206
Eth1/6 0 0 0 0 0 0
Eth1/7 0 0 0 0 0 293313744
Eth1/8 0 0 0 0 0 0
Eth1/9 0 0 0 0 0 8391599
Eth1/10 0 0 0 0 0 458647
Eth1/11 0 0 0 0 0 0
Eth1/12 0 0 0 0 0 301131763
Eth1/13 0 0 0 0 0 0
Eth1/14 0 0 0 0 0 0
Eth1/15 0 0 0 0 0 0
Eth1/16 0 0 0 0 0 0
Eth1/17 0 0 0 518 0 372595956
Eth1/18 0 0 0 0 0 294892966
Eth1/19 0 0 0 0 0 409606777
Eth1/20 0 0 0 0 0 409606777
Eth1/21 0 0 0 0 0 0
Eth1/22 0 1 0 1 0 301131763
Eth1/23 0 0 0 0 0 0
Eth1/24 0 0 0 0 0 0
Eth1/25 0 0 0 0 0 64721
Eth1/26 0 0 0 0 0 0
Eth1/27 0 0 0 0 0 0
Eth1/28 0 0 0 0 0 69095
Eth1/29 0 0 0 0 0 0
Eth1/30 0 0 0 0 0 0
Eth1/31 0 0 0 0 0 0
Eth1/32 0 0 0 0 0 0
Eth1/33 0 0 0 0 0 0
Eth1/34 0 0 0 0 0 0
Eth1/35 0 0 0 0 0 0
Eth1/36 0 0 0 0 0 0
Eth1/37 0 0 0 0 0 46812
Eth1/38 0 0 0 0 0 0
Eth1/39 0 0 0 0 0 0
Eth1/40 0 0 0 0 0 72929
Eth1/41 0 0 0 0 0 0
Eth1/42 0 0 0 0 0 0
Eth1/43 0 0 0 0 0 0
Eth1/44 0 0 0 0 0 0
Eth1/45 0 0 0 0 0 0
Eth1/46 0 0 0 0 0 0
Eth1/47 0 0 0 0 0 0
Eth1/48 0 0 0 0 0 0

MDSW-SND1# show policy-map interface ethernet1/7

Global statistics status : enabled

Ethernet1/7

Service-policy (qos) input: default-in-policy

Class-map (qos): class-default (match-any)
Match: any
set qos-group 0

Service-policy (queuing) input: default-in-policy
SNMP Policy Index: 301989925

Class-map (queuing): class-default (match-any)
bandwidth percent 100
queue dropped pkts : 0
queue depth in bytes : 0

Service-policy (queuing) output: default-out-policy
SNMP Policy Index: 301989928

Class-map (queuing): class-default (match-any)
bandwidth percent 100
queue dropped pkts : 0
queue depth in bytes : 0

I really really really would appreciate if any of you could provide me with any feedbacks!

Georg Pauwen · ‎02-13-2022

Hello,

the 3548 by default shares bandwidth between unicast and multicast at an equal rate (50/50).

Not sure what you have already tried, but you could manually change that rate, so that unicast gets less bandwidth e.g.:

N3548(config)#wrr unicast-bandwidth 40

This would cause multicast to get 60 percent.

johnyim12345678 · ‎02-13-2022

Hello, thank you very very much for your kind answer!

Our team has never tried changing such settings as we had not even known that switch seperates the bandwitdth between unicast and multicast. I just asked my boss about this and he said since we send most of our packets through multicasting, maybe we can monitor for today and then change the settings to multi 95 : uni 5 or sth and see if there's any difference tomorrow!

Thank you very much, and will update you tomorrow.

Thank you.

Georg Pauwen · ‎02-14-2022

Hello,

changing it to 95/5 might be a bit extreme, maybe it is a better strategy to start with 60/40, and monitor the results; you can always increase the value afterwards...

Curious to know if that setting makes a difference...

johnyim12345678 · ‎02-15-2022

Hi, so sorry but seems like this doesn't work....

I posted another question on this so pls give it a look if possible, thank you!

https://community.cisco.com/t5/routing/nexus3548-multicast-packet-drop-issue-2-would-really-appreciate/m-p/4552499#M362392

paul driver · ‎02-14-2022

Hello

@johnyim12345678 wrote:

We have been suffering from a significant level of packet drop, and despite the fact we have been troubleshooting on this so long

Possible bug - here

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

johnyim12345678 · ‎02-15-2022

Hi, thank you so much for your reply. I posted once more on this issue, pls take a look if possible. Thank you so much!

https://community.cisco.com/t5/routing/nexus3548-multicast-packet-drop-issue-2-would-really-appreciate/m-p/4552499#M362392

marce1000 · ‎02-14-2022

- Some (other) possible bugs :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuq96923

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva20344

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi93997

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

johnyim12345678 · ‎02-15-2022

Hi, thank you so much for your reply. I posted once more on this issue, pls take a look if possible. Thank you so much!!!

https://community.cisco.com/t5/routing/nexus3548-multicast-packet-drop-issue-2-would-really-appreciate/m-p/4552499#M362392

Georg Pauwen · ‎03-01-2022

Hello,

just out of curiosity, are you making any progress ?