intermittent poor network response to one connected server (voice recorder) on 3750 switch stack

[mitchen]

We have recently set up VoIP (using Mitel phone system) in one of our offices.  Auto QoS enabled and Voice (VLAN55) and Data VLAN (VLAN50) set-up.    We have 5 x Cisco 3750 switches in a stack.

The problem we have is that a connection to one server on the LAN – which carries out the voice recording ( a Red Box Recorder, for what its worth) – seems to suffer from some intermittent high response times/packet loss on the LAN.   This in turn affects the quality of the voice recordings at those times i.e. when you listen to recordings at those times there is some “jitter”.   (This server actually sits in the Voice VLAN)

Now, the fingers are being pointed at this being a “network problem” but there is no sign of any congestion or anything – we monitor utilisation and its relatively low.  And there are no problems with any other servers.  We have tried new network cards, new cables, new switch port, etc but the problem persists.  We monitor response times to the server and you will see normal response times then for periods it will shoot up to 600-700ms+ for small periods of time – we haven’t yet been able to correlate these high response times with any other activity.   There are no such problems for other servers.  (The other strange thing is that ping responses during such periods will typically form a pattern in one good response, one bad response, one good response, one bad response e.g. from another device on the same LAN:

Reply from 192.168.1.7: bytes=32 time=741ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=743ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=726ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=705ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=706ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=695ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=677ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=685ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=679ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Reply from 192.168.1.7: bytes=32 time=679ms TTL=126

Reply from 192.168.1.7: bytes=32 time=<1ms TTL=126

Even pings direct from the network switch itself show the occasional very high response e.g.

Success rate is 100 percent (10000/10000), round-trip min/avg/max = 1/3/772 ms

Here is the interface config for the switchport connected to the affected server:

!

interface FastEthernet1/0/5

switchport access vlan 55

switchport mode access

switchport voice vlan 55

srr-queue bandwidth share 10 10 60 20

priority-queue out

mls qos trust cos

auto qos voip trust

spanning-tree portfast

end

!

Interface statistics don’t show any signs of errors or collisions or high utilisation etc:

FastEthernet1/0/5 is up, line protocol is up (connected)

  Hardware is Fast Ethernet, address is b4a4.e3f4.ec85 (bia b4a4.e3f4.ec85)

  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,

     reliability 255/255, txload 5/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 9w6d, output 00:00:00, output hang never

  Last clearing of "show interface" counters 3w2d

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/0 (size/max)

  5 minute input rate 123000 bits/sec, 2 packets/sec

  5 minute output rate 2195000 bits/sec, 1202 packets/sec

     16584393 packets input, 21459676351 bytes, 0 no buffer

     Received 110362 broadcasts (30992 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 30992 multicast, 0 pause input

     0 input packets with dribble condition detected

     182894914 packets output, 38210765321 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

The only thing I can think of that could have any effect might be QoS settings?  We have just used AutoQoS config.  I’m not entirely sure how to interpret these stats but here are the stats for the interface in question (no sign of dropped packets at least?)

sh mls qos int fa1/0/5 statistics

FastEthernet1/0/5 (All statistics are in packets)

  dscp: incoming

-------------------------------

  0 -  4 :    16427738            0            0            0            0

  5 -  9 :           0            0            0            0            0

10 - 14 :           0            0            0            0            0

15 - 19 :           0            0            0            0            0

20 - 24 :           0            0            0            0            0

25 - 29 :           0            0            0            0            0

30 - 34 :           0            0            0            0            0

35 - 39 :           0            0            0            0            0

40 - 44 :           0            0            0            0            0

45 - 49 :           0            0            0            0            0

50 - 54 :           0            0            0            0            0

55 - 59 :           0            0            0            0            0

60 - 64 :           0            0            0            0

  dscp: outgoing

-------------------------------

  0 -  4 :   180448124            0            0            0            0

  5 -  9 :           0            0            0            0            0

10 - 14 :           0            0            0            0            0

15 - 19 :           0            0            0            0            0

20 - 24 :           0            0            0            0            0

25 - 29 :           0            0            0            0            0

30 - 34 :           0            0            0            0            0

35 - 39 :           0            0            0            0            0

40 - 44 :           0            0            0            0            0

45 - 49 :           0            0            0           44            0

50 - 54 :           0            0            0            0            0

55 - 59 :           0            0            0            0            0

60 - 64 :           0            0            0            0

  cos: incoming

-------------------------------

  0 -  4 :   16582454            0            0            0            0

  5 -  7 :           0            0            0

  cos: outgoing

-------------------------------

  0 -  4 :   181940545            0            0            0            0

  5 -  7 :           0           20           15

  output queues enqueued:

queue: threshold1 threshold2 threshold3

-----------------------------------------

queue 0:           2           0           0

queue 1:      840269       79466     1306139

queue 2:           0           0           0

queue 3:           0           0   181209518

  output queues dropped:

queue: threshold1 threshold2 threshold3

-----------------------------------------

queue 0:            0            0            0

queue 1:            0            0            0

queue 2:            0            0            0

queue 3:            0            0            0

Policer: Inprofile:            0 OutofProfile:            0

The only other issue we’ve had is that some phones intermittently “dropped” off the network – see https://supportforums.cisco.com/message/3238594#3238594 - we never really got to the bottom of that to be honest and, since it happens rarely, its being “lived with” – as part of troubleshooting that, on advice of our VoIP providers, we disabled CDP and enabled LLDP.  I don’t know whether that could have any impact or not?

The only other interesting factor is that the server in question is actually a virtual server (hyper-v) – again, I don’t know if there is any significane in that or not (our server guys assure us the set-up is fine!)

As you can probably gather, I’m struggling on this one!  Can anyone advise on what the possible cause could be or what more I can do to troubleshoot further and pinpoint the cause?   Its getting a bit “political” as we have 3^rd parties that look after the voice side of things and they are pointing the finger at the “network” being at fault!

Thanks.

[Shashank Singh]

Hi Mitchen,

Please session into the stack master and capture 'show platform port-asic stats drop' output. This should tell you if there are any drops because of QoS. If you do not see any drops, please capture 'show platform pm if-numbers' and share the outputs.

Regards,

Shashank

Please rate this answer if you found the content useful

[Shashank Singh]

Hi,

I only see dscp 0 traffic incoming and outgoing on interface fa1/0/5. This means that this interface has all the traffic with same marking. If there is some traffic (voice)  which needs to be prioritized, it should have been classified and marked properly and should have been seen on dscp 46 counters.

If this port connects to a device sending voice traffic, can you check if the traffic is being marked as dscp 46 by the server?

Regards,

Shashank

[mitchen]

Hi,  thanks for the response.   Outputs from those commands are below...

show platform port-asic stats drop

Port-asic Port Drop Statistics - Summary
========================================
  RxQueue 0 Drop Stats: 0
  RxQueue 1 Drop Stats: 0
  RxQueue 2 Drop Stats: 0
  RxQueue 3 Drop Stats: 0

  Port  0 TxQueue Drop Stats: 0
  Port  1 TxQueue Drop Stats: 0
  Port  2 TxQueue Drop Stats: 890
  Port  3 TxQueue Drop Stats: 0
  Port  4 TxQueue Drop Stats: 0
  Port  5 TxQueue Drop Stats: 0
  Port  6 TxQueue Drop Stats: 0
  Port  7 TxQueue Drop Stats: 0
  Port  8 TxQueue Drop Stats: 0
  Port  9 TxQueue Drop Stats: 0
  Port 10 TxQueue Drop Stats: 0
  Port 11 TxQueue Drop Stats: 0
  Port 12 TxQueue Drop Stats: 0
  Port 13 TxQueue Drop Stats: 0
  Port 14 TxQueue Drop Stats: 0
  Port 15 TxQueue Drop Stats: 0
  Port 16 TxQueue Drop Stats: 0
  Port 17 TxQueue Drop Stats: 0
  Port 18 TxQueue Drop Stats: 0
  Port 19 TxQueue Drop Stats: 0
  Port 20 TxQueue Drop Stats: 0
  Port 21 TxQueue Drop Stats: 0
  Port 22 TxQueue Drop Stats: 0
  Port 23 TxQueue Drop Stats: 0
  Port 24 TxQueue Drop Stats: 0
  Port 25 TxQueue Drop Stats: 0

  Supervisor TxQueue Drop Statistics
    Queue  0: 0
    Queue  1: 0
    Queue  2: 0
    Queue  3: 0
    Queue  4: 0
    Queue  5: 0
    Queue  6: 0
    Queue  7: 0
    Queue  8: 33
    Queue  9: 0
  Queue 10: 0
    Queue 11: 0
    Queue 12: 0
    Queue 13: 0
    Queue 14: 0
    Queue 15: 0

Port-asic Port Drop Statistics - Details
========================================
  RxQueue Drop Statistics
    Queue 0
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 1
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 2
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 3
  Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0

  Port 0 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 1 TxQueue Drop Statistics
  Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 2 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
    Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 890

  Port 3 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
    Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 4 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
    Weight 2 Frames 0

  Port 5 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 6 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
    Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 7 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
    Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 8 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
  Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 9 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

Port 10 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 11 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
  Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 12 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
    Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 13 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
    Weight 1 Frames 0
      Weight 2 Frames 0

  Port 14 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 15 TxQueue Drop Statistics
    Queue 0
    Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 16 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
    Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 17 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
    Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 18 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
   
  Port 19 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 20 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
    Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 21 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
  Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 22 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
    Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 23 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 24 TxQueue Drop Statistics
  Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 25 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
    Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
  Supervisor TxQueue Drop Statistics
    Queue  0: 0
    Queue  1: 0
    Queue  2: 0
    Queue  3: 0
    Queue  4: 0
    Queue  5: 0
    Queue  6: 0
    Queue  7: 0
    Queue  8: 33
    Queue  9: 0
    Queue 10: 0
  Queue 11: 0
    Queue 12: 0
    Queue 13: 0
    Queue 14: 0
    Queue 15: 0

show platform pm if-numbers

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Gi1/0/1   1    1    25   0/1  1    1    1    local     Yes     Yes
Gi1/0/2   2    2    26   0/0  1    2    2    local     Yes     Yes
Fa1/0/1   3    3    1    0/2  1    1    3    local     Yes     Yes
Fa1/0/2   4    4    2    0/3  1    2    4    local     Yes     Yes
Fa1/0/3   7    7    5    0/6  1    5    7    local     Yes     Yes
Fa1/0/4   6    6    4    0/5  1    4    6    local     Yes     Yes
Fa1/0/5   5    5    3    0/4  1    3    5    local     Yes     Yes
Fa1/0/6   8    8    6    0/7  1    6    8    local     Yes     Yes
Fa1/0/7   9    9    7    0/8  1    7    9    local     Yes     Yes
Fa1/0/8   10   10   8    0/9  1    8    10   local     Yes     Yes
Fa1/0/9   11   11   9    0/10 1    9    11   local     Yes     Yes
Fa1/0/10  12   12   10   0/11 1    10   12   local     Yes     Yes
Fa1/0/11  13   13   11   0/12 1    11   13   local     Yes     Yes
Fa1/0/12  14   14   12   0/13 1    12   14   local     Yes     Yes
Fa1/0/13  15   15   13   0/14 1    13   15   local     Yes     Yes
Fa1/0/14  16   16   14   0/15 1    14   16   local     Yes     Yes
Fa1/0/15  17   17   15   0/16 1    15   17   local     Yes     Yes
Fa1/0/16  18   18   16   0/17 1    16   18   local     Yes     Yes
Fa1/0/17  19   19   17   0/18 1    17   19   local     Yes     Yes
Fa1/0/18  20   20   18   0/19 1    18   20   local     Yes     Yes
   
interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa1/0/19  21   21   19   0/20 1    19   21   local     Yes     Yes
Fa1/0/20  22   22   20   0/21 1    20   22   local     Yes     Yes
Fa1/0/21  23   23   21   0/22 1    21   23   local     Yes     Yes
Fa1/0/22  24   24   22   0/23 1    22   24   local     Yes     Yes
Fa1/0/23  25   25   23   0/24 1    23   25   local     Yes     Yes
Fa1/0/24  26   26   24   0/25 1    24   26   local     Yes     Yes
Gi2/0/1   55   55   25   0/1  2    1    1    remote    No      Yes
Gi2/0/2   56   56   26   0/0  2    2    2    remote    No      Yes
Fa2/0/1   57   57   1    0/2  2    1    3    remote    No      Yes
Fa2/0/2   58   58   2    0/3  2    2    4    remote    No      Yes
Fa2/0/3   59   59   3    0/4  2    3    5    remote    No      Yes
Fa2/0/4   60   60   4    0/5  2    4    6    remote    No      Yes
Fa2/0/5   61   61   5    0/6  2    5    7    remote    No      Yes
Fa2/0/6   62   62   6    0/7  2    6    8    remote    No      Yes
Fa2/0/7   63   63   7    0/8  2    7    9    remote    No      Yes
Fa2/0/8   64   64   8    0/9  2    8    10   remote    No      Yes
Fa2/0/9   65   65   9    0/10 2    9    11   remote    No      Yes
Fa2/0/10  66   66   10   0/11 2    10   12   remote    No      Yes
Fa2/0/11  67   67   11   0/12 2    11   13   remote    No      Yes
Fa2/0/12  68   68   12   0/13 2    12   14   remote    No      Yes
Fa2/0/13  69   69   13   0/14 2    13   15   remote    No      Yes
Fa2/0/14  70   70   14   0/15 2    14   16   remote    No      Yes
Fa2/0/15  71   71   15   0/16 2    15   17   remote    No      Yes
   
interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa2/0/16  72   72   16   0/17 2    16   18   remote    No      Yes
Fa2/0/17  73   73   17   0/18 2    17   19   remote    No      Yes
Fa2/0/18  74   74   18   0/19 2    18   20   remote    No      Yes
Fa2/0/19  75   75   19   0/20 2    19   21   remote    No      Yes
Fa2/0/20  76   76   20   0/21 2    20   22   remote    No      Yes
Fa2/0/21  77   77   21   0/22 2    21   23   remote    No      Yes
Fa2/0/22  78   78   22   0/23 2    22   24   remote    No      Yes
Fa2/0/23  79   79   23   0/24 2    23   25   remote    No      Yes
Fa2/0/24  80   80   24   0/25 2    24   26   remote    No      Yes
Gi3/0/1   109  109  25   0/1  3    1    1    remote    No      Yes
Gi3/0/2   110  110  26   0/0  3    2    2    remote    No      Yes
Fa3/0/1   111  111  1    0/2  3    1    3    remote    No      Yes
Fa3/0/2   112  112  2    0/3  3    2    4    remote    No      Yes
Fa3/0/3   113  113  3    0/4  3    3    5    remote    No      Yes
Fa3/0/4   114  114  4    0/5  3    4    6    remote    No      Yes
Fa3/0/5   115  115  5    0/6  3    5    7    remote    No      Yes
Fa3/0/6   116  116  6    0/7  3    6    8    remote    No      Yes
Fa3/0/7   117  117  7    0/8  3    7    9    remote    No      Yes
Fa3/0/8   118  118  8    0/9  3    8    10   remote    No      Yes
Fa3/0/9   119  119  9    0/10 3    9    11   remote    No      Yes
Fa3/0/10  120  120  10   0/11 3    10   12   remote    No      Yes
Fa3/0/11  121  121  11   0/12 3    11   13   remote    No      Yes
Fa3/0/12  122  122  12   0/13 3    12   14   remote    No      Yes
   
interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa3/0/13  123  123  13   0/14 3    13   15   remote    No      Yes
Fa3/0/14  124  124  14   0/15 3    14   16   remote    No      Yes
Fa3/0/15  125  125  15   0/16 3    15   17   remote    No      Yes
Fa3/0/16  126  126  16   0/17 3    16   18   remote    No      Yes
Fa3/0/17  127  127  17   0/18 3    17   19   remote    No      Yes
Fa3/0/18  128  128  18   0/19 3    18   20   remote    No      Yes
Fa3/0/19  129  129  19   0/20 3    19   21   remote    No      Yes
Fa3/0/20  130  130  20   0/21 3    20   22   remote    No      Yes
Fa3/0/21  131  131  21   0/22 3    21   23   remote    No      Yes
Fa3/0/22  132  132  22   0/23 3    22   24   remote    No      Yes
Fa3/0/23  133  133  23   0/24 3    23   25   remote    No      Yes
Fa3/0/24  134  134  24   0/25 3    24   26   remote    No      Yes
Gi4/0/1   163  163  25   0/1  4    1    1    remote    No      Yes
Gi4/0/2   164  164  26   0/0  4    2    2    remote    No      Yes
Fa4/0/1   165  165  1    0/2  4    1    3    remote    No      Yes
Fa4/0/2   166  166  2    0/3  4    2    4    remote    No      Yes
Fa4/0/3   167  167  3    0/4  4    3    5    remote    No      Yes
Fa4/0/4   168  168  4    0/5  4    4    6    remote    No      Yes
Fa4/0/5   169  169  5    0/6  4    5    7    remote    No      Yes
Fa4/0/6   170  170  6    0/7  4    6    8    remote    No      Yes
Fa4/0/7   171  171  7    0/8  4    7    9    remote    No      Yes
Fa4/0/8   172  172  8    0/9  4    8    10   remote    No      Yes
Fa4/0/9   173  173  9    0/10 4    9    11   remote    No      Yes
   
interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa4/0/10  174  174  10   0/11 4    10   12   remote    No      Yes
Fa4/0/11  175  175  11   0/12 4    11   13   remote    No      Yes
Fa4/0/12  176  176  12   0/13 4    12   14   remote    No      Yes
Fa4/0/13  177  177  13   0/14 4    13   15   remote    No      Yes
Fa4/0/14  178  178  14   0/15 4    14   16   remote    No      Yes
Fa4/0/15  179  179  15   0/16 4    15   17   remote    No      Yes
Fa4/0/16  180  180  16   0/17 4    16   18   remote    No      Yes
Fa4/0/17  181  181  17   0/18 4    17   19   remote    No      Yes
Fa4/0/18  182  182  18   0/19 4    18   20   remote    No      Yes
Fa4/0/19  183  183  19   0/20 4    19   21   remote    No      Yes
Fa4/0/20  184  184  20   0/21 4    20   22   remote    No      Yes
Fa4/0/21  185  185  21   0/22 4    21   23   remote    No      Yes
Fa4/0/22  186  186  22   0/23 4    22   24   remote    No      Yes
Fa4/0/23  187  187  23   0/24 4    23   25   remote    No      Yes
Fa4/0/24  188  188  24   0/25 4    24   26   remote    No      Yes
Gi5/0/1   217  217  25   0/1  5    1    1    remote    No      Yes
Gi5/0/2   218  218  26   0/0  5    2    2    remote    No      Yes
Fa5/0/1   219  219  1    0/2  5    1    3    remote    No      Yes
Fa5/0/2   220  220  2    0/3  5    2    4    remote    No      Yes
Fa5/0/3   221  221  3    0/4  5    3    5    remote    No      Yes
Fa5/0/4   222  222  4    0/5  5    4    6    remote    No      Yes
Fa5/0/5   223  223  5    0/6  5    5    7    remote    No      Yes
Fa5/0/6   224  224  6    0/7  5    6    8    remote    No      Yes
   
interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa5/0/7   225  225  7    0/8  5    7    9    remote    No      Yes
Fa5/0/8   226  226  8    0/9  5    8    10   remote    No      Yes
Fa5/0/9   227  227  9    0/10 5    9    11   remote    No      Yes
Fa5/0/10  228  228  10   0/11 5    10   12   remote    No      Yes
Fa5/0/11  229  229  11   0/12 5    11   13   remote    No      Yes
Fa5/0/12  230  230  12   0/13 5    12   14   remote    No      Yes
Fa5/0/13  231  231  13   0/14 5    13   15   remote    No      Yes
Fa5/0/14  232  232  14   0/15 5    14   16   remote    No      Yes
Fa5/0/15  233  233  15   0/16 5    15   17   remote    No      Yes
Fa5/0/16  234  234  16   0/17 5    16   18   remote    No      Yes
Fa5/0/17  235  235  17   0/18 5    17   19   remote    No      Yes
Fa5/0/18  236  236  18   0/19 5    18   20   remote    No      Yes
Fa5/0/19  237  237  19   0/20 5    19   21   remote    No      Yes
Fa5/0/20  238  238  20   0/21 5    20   22   remote    No      Yes
Fa5/0/21  239  239  21   0/22 5    21   23   remote    No      Yes
Fa5/0/22  240  240  22   0/23 5    22   24   remote    No      Yes
Fa5/0/23  241  241  23   0/24 5    23   25   remote    No      Yes
Fa5/0/24  242  242  24   0/25 5    24   26   remote    No      Yes
St1       872  872  0    16/0  1    0    0    internal  Yes     Yes
St2       873  873  0    16/0  2    0    0    internal  No      Yes
St3       874  874  0    16/0  3    0    0    internal  No      Yes
St4       875  875  0    16/0  4    0    0    internal  No      Yes
St5       876  876  0    16/0  5    0    0    internal  No      Yes

Does anything look out of the ordinary there?

[Shashank Singh]

Hi,

Please session into switch 1 and share 'show platform port-asic stats drop port 4 asic 0' output .

Regards,

Shashank

[mitchen]

Hi Shashank,

output, as requested:

show platform port-asic stats drop port 4 asic 0

Port-asic Port Drop Statistics - Summary
========================================
  RxQueue 0 Drop Stats: 0
  RxQueue 1 Drop Stats: 0
  RxQueue 2 Drop Stats: 0
  RxQueue 3 Drop Stats: 0

  Port  0 TxQueue Drop Stats: 0
  Port  1 TxQueue Drop Stats: 0
  Port  2 TxQueue Drop Stats: 890
  Port  3 TxQueue Drop Stats: 0
  Port  4 TxQueue Drop Stats: 0
  Port  5 TxQueue Drop Stats: 0
  Port  6 TxQueue Drop Stats: 0
  Port  7 TxQueue Drop Stats: 0
  Port  8 TxQueue Drop Stats: 0
  Port  9 TxQueue Drop Stats: 0
  Port 10 TxQueue Drop Stats: 0
  Port 11 TxQueue Drop Stats: 0
  Port 12 TxQueue Drop Stats: 0
  Port 13 TxQueue Drop Stats: 0
  Port 14 TxQueue Drop Stats: 0
  Port 15 TxQueue Drop Stats: 0
  Port 16 TxQueue Drop Stats: 0
  Port 17 TxQueue Drop Stats: 0
  Port 18 TxQueue Drop Stats: 0
  Port 19 TxQueue Drop Stats: 0
  Port 20 TxQueue Drop Stats: 0
  Port 21 TxQueue Drop Stats: 0
  Port 22 TxQueue Drop Stats: 0
  Port 23 TxQueue Drop Stats: 0
  Port 24 TxQueue Drop Stats: 0
  Port 25 TxQueue Drop Stats: 0

  Supervisor TxQueue Drop Statistics
    Queue  0: 0
    Queue  1: 0
    Queue  2: 0
    Queue  3: 0
    Queue  4: 0
    Queue  5: 0
    Queue  6: 0
    Queue  7: 0
    Queue  8: 33
    Queue  9: 0
    Queue 10: 0
    Queue 11: 0
    Queue 12: 0
    Queue 13: 0
    Queue 14: 0
    Queue 15: 0

Port-asic Port Drop Statistics - Details
========================================
  RxQueue Drop Statistics
    Queue 0
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 1
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 2
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 3
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0

  Port 4 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
  Supervisor TxQueue Drop Statistics
    Queue  0: 0
    Queue  1: 0
    Queue  2: 0
    Queue  3: 0
    Queue  4: 0
    Queue  5: 0
    Queue  6: 0
    Queue  7: 0
    Queue  8: 33
    Queue  9: 0
    Queue 10: 0
    Queue 11: 0
    Queue 12: 0
    Queue 13: 0
    Queue 14: 0
    Queue 15: 0

Thanks again for your assistance - very much appreciated.

[mitchen]

Hi,

as i understand it, this server (its a Red Box Voice Recorder, I don't know if that helps any!) simply takes an RTP stream from the Mitel phone system to record the voice calls so I don't think it should be marked as DSCP 46 so I think what we see is correct?  However, that side of things was all set-up by a 3rd party and the information from them was very sketchy (or to be more precise, non-existent!)

[Shashank Singh]

Hi,

With the outputs it is clear that this switch is not dropping the packets (even though all the traffic on fa1/0/5 is dscp 0), either because of QoS or congestion. If there are other switches involved, we may probably have to check them too and we may also need to check the server.

We can perform a sniffer capture for fa1/0/5 to investigate deeper if we are facing TCP retransmissions for this traffic. I would suggest to open a TAC case for in-depth troubleshooting.

Regards,

Shashank

Please rate the answer if you found the content useful

[mitchen]

Hi Shashank,

thanks for your assistance on this.

I will request some more investigation on the server itself (I'm suspicious of the virtual server set-up) but will open a TAC case if neccessary.