Packet Loss on ME3400

[Hassan Mwangi]

I have deployed the Cisco ME3400 in the access part of my network (we are a big ISP providing WIMAX fixed internet connectivity). Recently, several sites have been having issues with packet drops whereas the bandwith provisioned is more than the existing traffic (at times even double) but i am still experiencing high latencies and packet drops.

However, when replaced with a new switch, the link stabilizes and no more packet drops.

Anyone who has a solution to this?

Anyone who can demystify the "show tech support" output?

Need urgent assistance as i might end up replacing all my switches.

[Wael Abdelaliem]

Hi Hassan,

Try to change the default queue-limit settings. Prior to 12.2(35)SE1, the maximum queue-size setting is 272, with 12.2(35)SE1

and later, the maximum queue-size is 544.  The default queue-size setting is 48 for all releases.

You can use a simple policy-map like:

policy-map QLIMIT
  class class-default
    queue-limit 544

interface FastEthernetX/Y
......
 service-policy output QLIMIT

HTH

Wael

[Hassan Mwangi]

Thanks Wael.

I had already done that and did not have any effect.

Also they were no queue drops on the interface.

Are you able to decrypt the show tech support output??

[rsimoni]

Hi Hassan,

decoding the show tech won't help you that much as when you troubleshoot drops you need 'dynamic' information and not just a static snap shot that you can get from the show tech.

to troubleshoot output drops first you need to check in which queue they are happening.

sh platf port-asic stats drop x/x

also it might be useful to check which traffic is received

sh platform qos statistics interface x/x

However the most important advice is to make sure that the we don't have speed mismatch between ingress and egress interfaces

for instance if you have

---------- gi0/1 ME34000 fa0/1 ------- 

and traffic flows from left to right you will likely see output drops on fa0/1 even if you use the policy setting bigger queue-limit (to 544).

Workaround would be to set speed port of Giga interface to 100Mbs (I know that you would loose bandwidth but there is no other way on this box when traffic is bursty).

Lastly, if you configure that policy map to multiple interface the use of it is limited as it just instructs the box to use extra buffer from the shared pool. If multiple ports try to get extra buffers from the shared pool (in chunks of 256 bytes) at the end of the day the benefit will be limited.

Bottom line this device has limited buffer capabilities.

regards,

Riccardo

[Hassan Mwangi]

Hello,

The traffic flows to and from Fa0/23 and Fa0/22.

Below is the output of the command and the interface configuration;

TST_SWITCH#sh platform qos statistics interface fastEthernet 0/22
FastEthernet0/22 (All statistics are in packets)

  dscp: incoming 
-------------------------------

  0 -  4 :     1002386          169            0            0            0 
  5 -  9 :           0            0            0         1638            0 
10 - 14 :           9            0            0            0            0 
15 - 19 :           0            4            0            0            0 
20 - 24 :           0            0            0            0            0 
25 - 29 :           0            0            0            0            0 
30 - 34 :           0            0            0            0            2 
35 - 39 :           0            0            0            0            0 
40 - 44 :           0            0            0            0            0 
45 - 49 :           0            0            0        50136            0 
50 - 54 :           0            0            0            0            0 
55 - 59 :           0            3            0            0            0 
60 - 64 :           0            0            0            0 
  dscp: outgoing
-------------------------------

  0 -  4 :        6929            0            0            0            0 
  5 -  9 :           0            0            0            0            0 
10 - 14 :           0            0            0            0            0 
15 - 19 :           0            0            0            0            0 
20 - 24 :           0            0            0            0            0 
25 - 29 :           0            0            0            0            0 
30 - 34 :           0            0            0            0            0 
35 - 39 :           0            0            0            0            0 
40 - 44 :           0            0            0            0            0 
45 - 49 :           0            0            0          344            0 
50 - 54 :           0            0            0            0            0 
55 - 59 :           0            0            0            0            0 
60 - 64 :           0            0            0            0 
  cos: incoming 
-------------------------------

  0 -  4 :     1017820          622            0            0            1 
  5 -  7 :           0        39577        43801 
  cos: outgoing
-------------------------------

  0 -  4 :        6521            0            0            0            0 
  5 -  7 :           0          344          474 
  output queues enqueued:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           2           0           0
queue 1:           0        6545         975
queue 2:           0           0           0
queue 3:           0           0        2240

  output queues dropped:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           0           0           0
queue 1:           0           0           0
queue 2:           0           0           0
queue 3:           0           0           0

TST_SWITCH#
TST_SWITCH#
TST_SWITCH#sh platform qos statistics interface fastEthernet 0/23
FastEthernet0/23 (All statistics are in packets)

  dscp: incoming 
-------------------------------

  0 -  4 :       10460            0            0            0            0 
  5 -  9 :           0            0            0            0            0 
10 - 14 :           0            0            0            0            0 
15 - 19 :           0            0            0            0            0 
20 - 24 :           0            0            0            0          914 
25 - 29 :           0            0            0            0            0 
30 - 34 :           0            0            0            0            0 
35 - 39 :           0            0            0            0            0 
40 - 44 :           0            0            0            0            0 
45 - 49 :           0            0            0          298            0 
50 - 54 :           0            0            0            0            0 
55 - 59 :           0            0            0            0            0 
60 - 64 :           0            0            0            0 
  dscp: outgoing
-------------------------------

  0 -  4 :       10817            0            0            0            0 
  5 -  9 :           0            0            0         1051            0 
10 - 14 :           7            0            0            0            0 
15 - 19 :           0            3            0            0            0 
20 - 24 :           0            0            0            0          576 
25 - 29 :           0            0            0            0            0 
30 - 34 :           0            0            0            0            0 
35 - 39 :           0            0            0            0            0 
40 - 44 :           0            0            0            0            0 
45 - 49 :           0            3            0         2232            0 
50 - 54 :           0            0            0            0            0 
55 - 59 :           0            3            0            0            0 
60 - 64 :           0            0            0            0 
  cos: incoming 
-------------------------------

  0 -  4 :       10993            0            0            0            0 
  5 -  7 :           0            0          825 
  cos: outgoing
-------------------------------

  0 -  4 :       12823            1            0            0            0 
  5 -  7 :           0         2184           29 
  output queues enqueued:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           2           0           0
queue 1:           0         303         335
queue 2:           0           0           0
queue 3:           0           0       14896

  output queues dropped:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           0           0           0
queue 1:           0           0           0
queue 2:           0           0           0
queue 3:           0           0           0

TST_SWITCH#

-------------------------------------------------------------------------------------------------

TST_SWITCH#show run int fa0/22
Building configuration...

Current configuration : 179 bytes
!
interface FastEthernet0/22
description Connection to METRO FIBRE PROVIDER
port-type nni
switchport trunk allowed vlan 604,1022
switchport mode trunk
load-interval 30
duplex full
end

TST_SWITCH#show run int fa0/23
Building configuration...

Current configuration : 172 bytes
!
interface FastEthernet0/23
description UPLINK TO BTS
port-type nni
switchport trunk allowed vlan 1-1012,1014-4094
switchport mode trunk
speed 100
duplex full
end

TST_SWITCH#

The Provider is a Metro Fibre provider providing connetivity from the BTS to the IP core. This is the link having packet drops and intermitency whereas the is enough provisioned capacity. The interfaces have no qeue drops.

Rgds,

Hassan M. Mwangi

[ปลาวาฬทราย RMUTT CPE IX]

#sho platform port-asic stats drop gigabitEthernet 0/5

  Interface Gi0/5 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 223677075

#sho platform qos statistics interface gi0/5
GigabitEthernet0/5 (All statistics are in packets)

  dscp: incoming  
-------------------------------

  0 -  4 :  1144255751           68        99134            0         1041  
  5 -  9 :          39           10           25         1975            0  
 10 - 14 :        3943            0            3            0         5425  
 15 - 19 :           0          223            0           68            0  
 20 - 24 :          53            0            0            1        13810  
 25 - 29 :           0            6            0            8           27  
 30 - 34 :           0            0          155            0            2  
 35 - 39 :           0            2            0            0           33  
 40 - 44 :           0            0            0            0            2  
 45 - 49 :           0            0            1        79353          125  
 50 - 54 :         575            7          234           40           80  
 55 - 59 :           8            0            1            0            0  
 60 - 64 :           1            1           24            2  
  dscp: outgoing
-------------------------------

  0 -  4 :   568408678         1837       109021     14342416         4521  
  5 -  9 :       20734         1082          340       407027            9  
 10 - 14 :   423559649          110          108            5      3020882  
 15 - 19 :           1        51289            0         9284            0  
 20 - 24 :       23280            0           20            1          341  
 25 - 29 :           0          215            0         2311         1493  
 30 - 34 :           5            0         4011            1           50  
 35 - 39 :           0           47            0            2         2196  
 40 - 44 :          44            5            4            0           45  
 45 - 49 :        2052          406            2        26162            5  
 50 - 54 :         287           30           15          277          366  
 55 - 59 :           1         1397            1            2            0  
 60 - 64 :           3            6         3733           42  
  cos: incoming  
-------------------------------

  0 -  4 :  1144533605            0            0            0            0  
  5 -  7 :           0            0            0  
  cos: outgoing
-------------------------------

  0 -  4 :  1010005892            0            0            0            0  
  5 -  7 :           0            0            0  
  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:           0       16347     2459761
 queue 2:           0           0           0
 queue 3:           0           0  1010070312

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:           0           0           0
 queue 2:           0           0           0
 queue 3:           0           0        6486

#sho cdp nei gi0/5
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
AY-CDC882_17  Gig 0/5           129             R S I  1841      Fas 0/0

#sho int status | inc 0/5
Gi0/5     CDC882_AYAY     connected    520        a-full  a-100 10/100/1000BaseTX

How should I do?

Thank you very much.

[pbrogniart]

Hello Assan,

Any news about your problem ?

I'm having the same problem on My ME3400 used to connect GSM BTS via microwave links. First i thought it was due to bad quality of links, but after several tests and measure the links look fine.

So i put in place ip sla icmp-echo and track it, so i discover some icmp failure, but when i'm checking the platform queue stat, it is Ok, no drop ???

i'm a bit lost...

if you have any feedback

Pieter.

[Hans Martinez]

Hi, you have any update about this issue?

I have the same problem whit a ME340x Software (ME340x-METROBASE-M), Version 12.2(35)SE5

When the traffic is less a 17Mb don't have problemes, but when the output traffic up to 20-30 Mb loss 5 or 7 porcent of the traffic.

If somebody have news, share please.

Regards,

Hans

[rsimoni]

as said already tx buffers on this platform are pretty limited.

workaround is to use the policy map suggested before and make sure there is no oversubscription between ingress and egress port.

if drops are still there there is little to add unfortunately