How to fix packet loss across EoMPLS?

[ปลาวาฬทราย RMUTT CPE IX]

Diagram:
CE1 - PE1 - PE2 - CE2

CE1#sho run int vla 253
Building configuration...

Current configuration : 84 bytes
!
interface Vlan253
 description test
 ip address 1.1.1.2 255.255.255.252
end

CE1#ping 1.1.1.1 siz 1526 df rep 9000
Success rate is 99 percent (8992/9000), round-trip min/avg/max = 1/5/108 ms

Thank you very much.

[Akash Agrawal]

Hi,

One way to troubleshoot is to check input/output drops for each link involved in topology. Other way is break L2circuit in few L3 segments and ping hop-by-hop, isolate culprit segment and check further.

You can break complete path in 3 segment

CE1 - PE1 - PE2 - CE2

Segment-1 : CE1--PE1

Segment-2 : PE1--PE2

Segment-3 : PE2--CE2

To check each segment, we need to break L2 into L3 circuit. On PE1 attachment circuit, instead of configuring xconnect, configure CE2 ip address, ping to CE1 ip address, check if there are drops. Similarly other side attachment circuit can be tested. For PE1-PE2, you can do loopback to loopback ping test.

Regards,

Akash

[mwandu]

Hi,

We have a similar issue of packet loss over EoMPLS.

We are observing the following consistent packet loss on all pseudowires originating from one of the switches - ME3600x running IOS code - System image file is "flash:/me360x-universalk9-mz.153-3.S3/me360x-universalk9-mz.153-3.S3.bin

current uptime;

show ver | i uptim
-UPE-01 uptime is 8 weeks, 5 days, 5 hours, 46 minutes

     timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 98 percent (988/1000), round-trip min/avg/max = 1/2/52 ms
 Total Time Elapsed 24944 ms

UPE-01#ping mpls pseudowire 10.220.31.184 2000202 size 1500 repeat 1000
Sending 1000, 1500-byte MPLS Echos to 10.220.31.184,
     timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface,
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
  'P' - no rx intf label prot, 'p' - premature termination of LSP,
  'R' - transit router, 'I' - unknown upstream index,
  'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 98 percent (988/1000), round-trip min/avg/max = 1/2/16 ms
 Total Time Elapsed 24804 ms

The pattern of packet drops is very consistent as seen above, two consecutive pings with same result. I have checked system resources and all are within acceptable utilisation thresholds.

What do you think could be issue, platform or code? all was working fine until recently when issue surfaced.

[aacole]

dialondemand, I see your ping send interval is 0ms with large size frames. I have seen similar regular packet drops in other networks under these conditions, where QoS policy thresholds were kicking in and causing regular drops.

You could try reducing the rate or set the ToS byte (if OS supports the option) to a value that your network will grant higher priority, see if you get the same behavior.

Andy

dialondemand

dialondemand

[Vinit Jain]

Adding to what Andy said, if you are seeing drops in a pattern, then most likely its a rate-limiter that is kicking in.

[ปลาวาฬทราย RMUTT CPE IX]

Should the ping I repeat for check hop-by-hop?

Thank you very much.