Cisco 3750 dropping RTP packets

tiwang · ‎10-25-2010

Hi out there

We have a Cisco 3750 running C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1). We have a lot of voice traffice passing through this switch and through sniffing we can see that it is dropping a lots of RTP packets. It is pretty basic configured - no QoS etc defined, no DSCP, WRED etc

We have been looking on the interfaces:

#sh int gi 1/0/7

GigabitEthernet1/0/7 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 001b.2b3e.a807 (bia 001b.2b3e.a807)

Description: CVN002

MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 10/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:17, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 4370000 bits/sec, 4867 packets/sec

5 minute output rate 41241000 bits/sec, 7080 packets/sec

230819752 packets input, 3991716436 bytes, 0 no buffer

Received 250283 broadcasts (0 multicast)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 250249 multicast, 0 pause input

0 input packets with dribble condition detected

340046289 packets output, 2013348553 bytes, 0 underruns

0 output errors, 0 collisions, 1 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

I cannot see any drops there but if I look at the buffers I can see a huge number of missed on a que named RxQ8 and RxQ15

h buffers

Buffer elements:

477 in free list (500 max allowed)

218948661 hits, 0 misses, 0 created

Public buffer pools:

Small buffers, 104 bytes (total 50, permanent 50, peak 179 @ 3d15h):

48 in free list (20 min, 150 max allowed)

109160983 hits, 43 misses, 129 trims, 129 created

0 failures (0 no memory)

Middle buffers, 600 bytes (total 25, permanent 25, peak 49 @ 3d15h):

24 in free list (10 min, 150 max allowed)

6127 hits, 12 misses, 36 trims, 36 created

0 failures (0 no memory)

Big buffers, 1536 bytes (total 50, permanent 50):

36 in free list (5 min, 150 max allowed)

985982 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

VeryBig buffers, 4520 bytes (total 25, permanent 10, peak 26 @ 3d15h):

10 in free list (0 min, 100 max allowed)

234 hits, 8 misses, 1 trims, 16 created

0 failures (0 no memory)

Large buffers, 5024 bytes (total 0, permanent 0):

0 in free list (0 min, 10 max allowed)

0 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

Huge buffers, 18024 bytes (total 4, permanent 0, peak 8 @ 3d15h):

4 in free list (0 min, 4 max allowed)

132230 hits, 4972 misses, 9915 trims, 9919 created

0 failures (0 no memory)

Interface buffer pools:

RxQFB buffers, 2040 bytes (total 904, permanent 904):

898 in free list (0 min, 904 max allowed)

396191 hits, 0 misses

RxQ0 buffers, 2040 bytes (total 1200, permanent 1200):

700 in free list (0 min, 1200 max allowed)

885730 hits, 0 misses

RxQ1 buffers, 2040 bytes (total 128, permanent 128):

2 in free list (0 min, 128 max allowed)

1251686 hits, 13950 fallbacks

RxQ2 buffers, 2040 bytes (total 64, permanent 64):

0 in free list (0 min, 64 max allowed)

64 hits, 0 misses, 0 trims, 0 created

0 failures (0 no memory)

RxQ3 buffers, 2040 bytes (total 128, permanent 128):

2 in free list (0 min, 128 max allowed)

13997270 hits, 176747 fallbacks

RxQ4 buffers, 2040 bytes (total 128, permanent 128):

1 in free list (0 min, 128 max allowed)

137428 hits, 1653 fallbacks

RxQ5 buffers, 2040 bytes (total 96, permanent 96):

32 in free list (0 min, 96 max allowed)

64 hits, 0 misses

RxQ6 buffers, 2040 bytes (total 128, permanent 128):

0 in free list (0 min, 128 max allowed)

131 hits, 3 misses

RxQ7 buffers, 2040 bytes (total 192, permanent 192):

59 in free list (0 min, 192 max allowed)

170653 hits, 0 misses

RxQ8 buffers, 2040 bytes (total 64, permanent 64):

0 in free list (0 min, 64 max allowed)

1102458 hits, 1020513 misses

RxQ9 buffers, 2040 bytes (total 1, permanent 1):

0 in free list (0 min, 1 max allowed)

1 hits, 0 misses

RxQ10 buffers, 2040 bytes (total 64, permanent 64):

1 in free list (0 min, 64 max allowed)

13045721 hits, 203843 fallbacks

RxQ11 buffers, 2040 bytes (total 16, permanent 16):

0 in free list (0 min, 16 max allowed)

16 hits, 0 misses

RxQ12 buffers, 2040 bytes (total 96, permanent 96):

0 in free list (0 min, 96 max allowed)

96 hits, 0 misses

RxQ13 buffers, 2040 bytes (total 16, permanent 16):

0 in free list (0 min, 16 max allowed)

16 hits, 0 misses

RxQ15 buffers, 2040 bytes (total 4, permanent 4):

0 in free list (0 min, 4 max allowed)

6289058 hits, 6289054 misses

IPC buffers, 2048 bytes (total 300, permanent 300):

299 in free list (150 min, 500 max allowed)

1 hits, 0 fallbacks, 0 trims, 0 created

0 failures (0 no memory)

Are these misses an indication of my problem?

best regards /ti

paolo bevilacqua · ‎10-26-2010

Buffer misses do not indicate packet drop expecially on switched.

It is extremely unlikely that the switch is droppping packets,

However, try updating software and reload the switch.

Steven Holl · ‎10-26-2010

You may get more traction on this one in the switching forum. Since you know you have drops, it's not a voice specific issue and the issue is related to the switch queues/QoS, and the issue needs to be looked at from a 3750 packet dropping perspective.

I don't have rights to move threads, though.

Message was edited by: Steven Holl

tiwang · ‎10-28-2010

hi again

Well - I am not sure on how to interpret the output of "show buffers" - since the Cat is dropping RTP packets in transistion (no packets missing before the switch and missing RTP packets after) I would expect to find somewhere where it is low on buffers - and the only buffers I can find which shos "something" are these RxQ8 & RxQ15 buffers which seems to be pretty exhausted.

Could these buffers be my problem?

best regards /ti

paolo bevilacqua · ‎10-28-2010

As mentioned above - buffers do not matter on switches.

If you have qos and queue "tuning" commands, remove them all.

gatlin007 · ‎10-28-2010

Like most switches the ASIC's are oversubscribed on the 3750. I usually find most drops on the ASIC. Try this command:

Switch# show controllers ethernet-controller port-asic statistics

===========================================================================

Switch 1, PortASIC 0 Statistics

---------------------------------------------------------------------------

         0 RxQ-0, wt-0 enqueue frames            0 RxQ-0, wt-0 drop frames

   4118966 RxQ-0, wt-1 enqueue frames            0 RxQ-0, wt-1 drop frames

         0 RxQ-0, wt-2 enqueue frames            0 RxQ-0, wt-2 drop frames

         0 RxQ-1, wt-0 enqueue frames            0 RxQ-1, wt-0 drop frames

       296 RxQ-1, wt-1 enqueue frames            0 RxQ-1, wt-1 drop frames

   2836036 RxQ-1, wt-2 enqueue frames            0 RxQ-1, wt-2 drop frames

         0 RxQ-2, wt-0 enqueue frames            0 RxQ-2, wt-0 drop frames

         0 RxQ-2, wt-1 enqueue frames            0 RxQ-2, wt-1 drop frames

    158377 RxQ-2, wt-2 enqueue frames            0 RxQ-2, wt-2 drop frames

         0 RxQ-3, wt-0 enqueue frames            0 RxQ-3, wt-0 drop frames

         0 RxQ-3, wt-1 enqueue frames            0 RxQ-3, wt-1 drop frames

         0 RxQ-3, wt-2 enqueue frames            0 RxQ-3, wt-2 drop frames

        15 TxBufferFull Drop Count               0 Rx Fcs Error Frames

         0 TxBufferFrameDesc BadCrc16            0 Rx Invalid Oversize Frames

         0 TxBuffer Bandwidth Drop Cou           0 Rx Invalid Too Large Frames

         0 TxQueue Bandwidth Drop Coun           0 Rx Invalid Too Large Frames

         0 TxQueue Missed Drop Statist           0 Rx Invalid Too Small Frames

        74 RxBuffer Drop DestIndex Cou           0 Rx Too Old Frames

         0 SneakQueue Drop Count                 0 Tx Too Old Frames

         0 Learning Queue Overflow Fra           0 System Fcs Error Frames

         0 Learning Cam Skip Count

        15 Sup Queue 0 Drop Frames               0 Sup Queue 8 Drop Frames

         0 Sup Queue 1 Drop Frames               0 Sup Queue 9 Drop Frames

         0 Sup Queue 2 Drop Frames               0 Sup Queue 10 Drop Frames

         0 Sup Queue 3 Drop Frames               0 Sup Queue 11 Drop Frames

         0 Sup Queue 4 Drop Frames               0 Sup Queue 12 Drop Frames

         0 Sup Queue 5 Drop Frames               0 Sup Queue 13 Drop Frames

         0 Sup Queue 6 Drop Frames               0 Sup Queue 14 Drop Frames

         0 Sup Queue 7 Drop Frames               0 Sup Queue 15 Drop Frames

tiwang · ‎11-03-2010

hi again

OK - this looks interesting - even though I cannot find drops on the ports where I expected to find drops - here is a snapshot of the switch - with a few minutes interval:

distrib-sw-001#sh platform port-asic stats drop gigabitEthernet 1/0/24

Interface Gi1/0/24 TxQueue Drop Statistics

Queue 0

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 1

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 2

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 3

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 94587432

distrib-sw-001#sh platform port-asic stats drop gigabitEthernet 1/0/24

Interface Gi1/0/24 TxQueue Drop Statistics

Queue 0

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 1

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 2

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 3

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 94588871

distrib-sw-001#

These weight 2 frames which seems to me to be dropped - what is this? It is pretty hard to find some documentations on what I in fact see here...

best regards /ti