Error in CEM circuit Excessive Packet Loss Rate Packet Loss (Error en CEM circuit)

[Ruben Lozada]

Good afternoon,
   
    I have the pleasure to address you looking for support because we have a failure at the level of signaling our network, the detail is that there are many locations with the same error, which is excessive packet loss on local computers, but we do not know the source Of the same we have discarded the Tunnel-TE, the STM-1 where they are transported, however we can not give either with the origin or the solution of it, I attach the different show of the team in relation.

********************************************************************

controller SONET 2/2/0
framing sdh
clock source line
aug mapping au-4
!
au-4 1 tug-3 1
mode c-12

tug-2 1 e1 1 cem-group 0 timeslots 24

*********************************************************************

interface Tunnel14284
description Side_A Tu18442 (SIGNALING-B)
ip unnumbered Loopback0
logging event link-status
tunnel mode mpls traffic-eng
tunnel destination 10.24.127.246
tunnel mpls traffic-eng priority 0 0
tunnel mpls traffic-eng bandwidth 1000
tunnel mpls traffic-eng affinity 0x2 mask 0x2
tunnel mpls traffic-eng path-option 1 dynamic
tunnel mpls traffic-eng record-route
!
interface CEM2/2/0
no ip address
cem 0
service-policy input SET-EXP5-IN
dejitter-buffer 150
xconnect 10.24.127.246 3086 encapsulation mpls pw-class PLANO_B_A
!

***********************************************************************

Side_B#sh cem cir 0

CEM2/2/0, ID: 0, Line: UP, Admin: UP, Ckt: ACTIVE
Controller state: up, T1/E1 state: up
Idle Pattern: 0xFF, Idle CAS: 0x8
Dejitter: 150 (In use: 0)
Payload Size: 32
Framing: Framed (DS0 channels: 24)
CEM Defects Set
Excessive Pkt Loss Rate Packet Loss

Signalling: No CAS
RTP: No RTP

Ingress Pkts: 37501    Dropped: 0
Egress Pkts: 0            Dropped: 0

CEM Counter Details
Input Errors: 0                   Output Errors: 0
Pkts Missing: 37500          Pkts Reordered: 0
Misorder Drops: 0             JitterBuf Underrun: 0
Error Sec: 0                    Severly Errored Sec: 0
Unavailable Sec: 150            Failure Counts: 0
Pkts Malformed: 0            JitterBuf Overrun: 0

Side_B#sh mpls l2t vc

Local intf Local circuit Dest address VC ID Status
------------- -------------------------- --------------- ---------- ----------

CE2/2/0 CESoPSN Basic 0 10.24.127.246 3086 UP

**************************************************************************

Side_B#sh xconnect all
Legend: XC ST=Xconnect State S1=Segment1 State S2=Segment2 State
UP=Up DN=Down AD=Admin Down IA=Inactive
SB=Standby HS=Hot Standby RV=Recovering NH=No Hardware

XC ST Segment 1 S1 Segment 2 S2
------+---------------------------------+--+---------------------------------+--
UP pri ac CE2/2/0:0 (CESoPSN Basic) UP mpls 10.24.127.246:3086 UP

******************************************************************************

Side_B#sh mpls l2t vc 3086 de
Local interface: CE2/2/0 up, line protocol up, CESoPSN Basic 0 up
Destination address: 10.24.127.246, VC ID: 3086, VC status: up
Output interface: Tu14284, imposed label stack {16064 653}
Preferred path: Tunnel14284, active
Default path: ready
Next hop: point2point
Create time: 01:02:09, last status change time: 01:02:09
Last label FSM state change time: 01:02:09
Signaling protocol: LDP, peer 10.24.127.246:0 up
Targeted Hello: 10.23.127.252(LDP Id) -> 10.24.127.246, LDP is UP
Status TLV support (local/remote) : enabled/supported
LDP route watch : enabled
Label/status state machine : established, LruRru
Last local dataplane status rcvd: No fault
Last BFD dataplane status rcvd: Not sent
Last BFD peer monitor status rcvd: No fault
Last local AC circuit status rcvd: No fault
Last local AC circuit status sent: No fault
Last local PW i/f circ status rcvd: No fault
Last local LDP TLV status sent: No fault
Last remote LDP TLV status rcvd: No fault
Last remote LDP ADJ status rcvd: No fault
MPLS VC labels: local 23, remote 653
Group ID: local 0, remote 0
MTU: local 0, remote 0
Remote interface description:
Sequencing: receive disabled, send disabled
Control Word: On (configured: autosense)
SSO Descriptor: 10.24.127.246/3086, local label: 23
Dataplane:
SSM segment/switch IDs: 4177/4176 (used), PWID: 1
VC statistics:
transit packet totals: receive 0, send 928318
transit byte totals: receive 0, send 37132720
transit packet drops: receive 0, seq error 0, send 0

[Vinit Jain]

Are you running this on ASR903 platform?

Missing packets Counters - This indicates CEM did not receive packets from the PW side. If the rate of increase of missing packets is very very slow as in one in a thousand packets of ingress or lesser, then suspect the clocking between the two PEs. If the rate is very high there is a problem in the PW side packet path either on the PW side.

from the show mpls l2 vc output, i do not see the receive counters incrementing. What is the remote end PE router. Could you please share the configuration from both the PE routers in a file.

Thanks

Vinit

[Ruben Lozada]

It's a Cisco 7609-S

[Douglas Ramirez]

Hi Ruben,

You said, you have discarded TE tunnel and STM-1. How did you that. Did you check STM-1 controller stats ?. Have you tried to use TE explicit path instead of dynamic ?.

As Vinit, already stated, this is an indication that CEM interface is not receiving packets from PW side. please run ping mpls pseudowire between both routers and check if there are packets lost.

thx,

Douglas

[Ruben Lozada]

Hi douglas

    Actually we did ping pseudowire mpls and these are completed, the STM-1 section we discarded since it does not present us errors in the interfaces and also more services by the same STM-1 without affection; And the TE also for hemo s done tests of ping with interface of origin the TE and they are sastifactorias.

    The last tests we did were to erase all the configurations of the circuits in question and were created again using other resources in the STP as well as in the SDH node and in the BBO IP / MPLS.

[Giuseppe Larosa]

Hello Ruben,

as noted by Vinit you need to provide the configurations of both PE nodes and the show version in order to get better help.

We see:

>> transit packet totals: receive 0, send 928318
transit byte totals: receive 0, send 37132720

You have a unidirectional xconnect in the forwarding plane, but it is not possible to say more without the above additional information.

Hope to help

Giuseppe

[Ruben Lozada]

Good afternoon, thanks for the support I attach the configurations of the 2 extremes to see in which they can support us still we continue with the fault and we do not know how to give an answer.

[Giuseppe Larosa]

Hello Ruben,

I see from configuration that there are other pseudowires configured to carry CEM frames between the two pair of PE nodes.

Are all the other pseudowires between these two PE nodes behaving correctly?

If so I would follow Douglas' suggestion to use ping mpls pseudowire on both ends so you can check the status of the forwarding plane for this specific pseudowire.

CEM 0 side B is made of 24 timeslots DS0 64 kbps each

tug-2 1 e1 1 cem-group 0 timeslots 24

on the A side the cem group 189 that is associated to pseudowire 3086 has only 3 timeslots.

>> tug-2 3 e1 1 cem-group 189 timeslots 3

Is this wanted ?

Hope to help

Giuseppe

[Telefonica Nicaragua]

Hello Ruben

We have the same issue, this is our topology ....

RBS --- ASR901 ----- MPLS CLOUD --- A903 --- BSC

A901
VC statistics:
transit packet totals: receive 0, send 608335
transit byte totals: receive 0, send 311467520
transit packet drops: receive 0, seq error 0, send 0

A903
VC statistics:
transit packet totals: receive 660071, send 0
transit byte totals: receive 345877204, send 0
transit packet drops: receive 0, seq error 0, send 0

A901_LAB#ping mpls pseudowire 172.20.16.119 310001 source 172.20.16.214
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
Total Time Elapsed 16 ms

A903##ping mpls pseudowire 172.20.16.214 310001 source 172.20.16.119
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
Total Time Elapsed 17 ms

Did you find a solution?

Thanks 

[Ruben Lozada]

Hi, we still have the flaw.

[3Dg4RIP]

Hi, Did you find the solution? Is relationed to Syncronization or there another issues?