debugging tm drops

[AARON WEINTRAUB]

So we've got the following drops showing up in the tm and I don't know where to go from here to continue troubleshooting it... the np counters themselves look ok, nothing large except for:

  17  MDF_TX_WIRE                                       4240535569     1432592
  21  MDF_TX_FABRIC                                     5048206075     1706913
  33  PARSE_FAB_RECEIVE_CNT                             4240535330     1432592
  45  PARSE_ENET_RECEIVE_CNT                            5047998662     1706835

RP/0/RP1/CPU0:ios#clea controller np tm counters np0 tm tm1 location 0/17/cpu0
Mon Aug 24 23:10:03.411 UTC

                Node: 0/17/CPU0:
----------------------------------------------------------------

==== (NP 0 TM 1) ====
TM Counters successfully cleared.

RP/0/RP1/CPU0:ios#show controllers np  tm counters np0 tm all location 0/17/cpu0
Mon Aug 24 23:10:05.597 UTC

                Node: 0/17/CPU0:
----------------------------------------------------------------

==== TM Counters (NP 0 TM 0) ====
 TM Counters: 
xmt paks: 25815313,  xmt bytes: 10418420412
drop paks: 0,  drop_bytes: 0

==== TM Counters (NP 0 TM 1) ====
 TM Counters: 
xmt paks: 12076465,  xmt bytes: 7522943649
drop paks: 2130315,  drop_bytes: 880240046

RP/0/RP1/CPU0:ios#show controllers np  tm counters np0 tm all location 0/17/cpu0
Mon Aug 24 23:10:06.508 UTC

                Node: 0/17/CPU0:
----------------------------------------------------------------

==== TM Counters (NP 0 TM 0) ====
 TM Counters: 
xmt paks: 25815390,  xmt bytes: 10418443072
drop paks: 0,  drop_bytes: 0

==== TM Counters (NP 0 TM 1) ====
 TM Counters: 
xmt paks: 17094623,  xmt bytes: 10655988130
drop paks: 3007915,  drop_bytes: 1243158216

RP/0/RP1/CPU0:ios#show controllers np  tm counters np0 tm all location 0/17/cpu0
Mon Aug 24 23:10:07.572 UTC

                Node: 0/17/CPU0:
----------------------------------------------------------------

==== TM Counters (NP 0 TM 0) ====
 TM Counters: 
xmt paks: 25815497,  xmt bytes: 10418465671
drop paks: 0,  drop_bytes: 0

==== TM Counters (NP 0 TM 1) ====
 TM Counters: 
xmt paks: 22968196,  xmt bytes: 14314921624
drop paks: 4052432,  drop_bytes: 1678095451

RP/0/RP1/CPU0:ios#show controllers np  tm counters np0 tm all location 0/17/cpu0
Mon Aug 24 23:10:48.401 UTC

                Node: 0/17/CPU0:
----------------------------------------------------------------

==== TM Counters (NP 0 TM 0) ====
 TM Counters: 
xmt paks: 25819295,  xmt bytes: 10419278298
drop paks: 0,  drop_bytes: 0

==== TM Counters (NP 0 TM 1) ====
 TM Counters: 
xmt paks: 248317791,  xmt bytes: 154847889080
drop paks: 42811660,  drop_bytes: 17843489190

[xthuijs]

hey aaron, for this Tomahawk linecard there are 2 TM's (traffic manager). one for ingress, one for egress.

if the TM is dropping packets it means that we are heading up to our port rate.

I believe your issue here was that the TH card perceived the installed cpak as a 10x10 breakout shaping to 10G instead of the 1x100 cpak that you had installed/were using.

that is a sw error in the driver that needs to be corrected.

cheers!

xander

[AARON WEINTRAUB]

For those following along at home, this was determined to be CSCuv98171.  Many thanks to our cisco team for helping us out on this one.

[xthuijs · Accepted solution]

hey aaron, thanks for updating the thread!

For everyone's benefit, the issue appeared to be related to the new Tomahawk/NP5 linecards whereby the traffic manager was rate limited to 10G while the interface is used in 100G mode.

Since the NP5/Tomahawk 100G interfaces can be used in both 100G and 10x10 breakout, there was a sw initialization problem that incorrectly perceived the mode to be 10G/breakout while it was ought to run 100G full.

The situation can be "alleviated" with a linecard reload, but it is probably better to load the (reload, yikes) smu to bypass this issue all together.

This fix will be in the 533 baseline (so is not fixed in 532).

cheers

xander