packet loss on SONET interface

evan.r.moore · ‎06-29-2015

This story began with this conversation:

https://supportforums.cisco.com/discussion/12502321/packet-drop-hunting-asr-9010-sip-700-and-xr-513

It now continues, after the commands Xander recommended shed no light for me. Here's the current environment:

R1 is a Cisco 2901 router, R2 is a non-Cisco router, and they are linked via multilink PPP over 4 DS1 circuits. The circuits are provided by TDM and SONET equipment in the lab, and conventional BERT tests for extended periods (over 48 hours) have shown zero errors in the TDM/SONET equipment.

When I attach a packet generator to the ethernet port of R1, a packet receiver to the ethernet port of R2, and send 4.5 Mbps of 256 byte packets, I drop zero packets. If I move the generator onto R2 and the receiver onto R1, testing the other direction with 4.5Mbps of 256 byte packets, I drop zero packets.

This is all what I expect. The problem comes when I change R1 to a Cisco ASR 9010, with a SIP-700 card and a SPA-2XCHOC12/DS0. I still define a bundle of four DS1 circuits in the multilink PPP as the logical link.

In this scenario, with the packet receiver attached to R1 (the ASR) and the packet generator attached to R2, I have zero packet loss. In the other direction, so the traffic stream is being transmitted by the ASR (R1) to R2, I have a packet loss rate of roughly 1 to 5 per 100,000 (depending on the test run). It's a bursty pattern, with 10s of packets getting lost sporadically.

The question, of course, is why. This is a lab environment, with no other traffic passing through the ASR, and little terminating on the ASR.

Let me know what configuration or hardware information to post, and I'll get it up.

Thanks for any help.

Bitwrangler for FirstLight Fiber Corp

Vinit Jain · ‎07-04-2015

Hello,

Can you try with a ping test. i wanted to understand if there was any pattern in the packets being dropped. It may be possible that LPTS might be dropping those packets. Though the question might be why.

Please let me know if you can try with ping test with MTU size of the interface and do-not-frag bit set and perform the test with the repeat count of 10000 packets.

I think you should still see drops.

please let me know the results.

Vinit

Thanks
--Vinit

evan.r.moore · ‎07-05-2015

Vinit,

I'll get that set up and report, but I'm rather skeptical. I'm testing with a traffic flow (UDP on port 10001) which passes through R1 and R2 in all cases. Since neither R1 nor R2 is the source or the destination, why would LPTS come into play?

ERM

Bitwrangler for FirstLight Fiber Corp

Vinit Jain · ‎07-05-2015

I agree to your point, but what i am trying to understand is, if the packets are getting software switched or hardware switched and its good to be sure that if LPTS is playing any role here or not. The plan is just for isolation purpose :)

Vinit

Thanks
--Vinit

evan.r.moore · ‎07-06-2015

Vinit,

One packet was lost out of the 10,000.

ERM

Bitwrangler for FirstLight Fiber Corp

Vinit Jain · ‎07-06-2015

Thanks, Could you please let me know what is the linecard and the XR version that you are running.

Thanks
--Vinit

evan.r.moore · ‎07-06-2015

RP/0/RSP0/CPU0:cr1.st.blfl.vt#show inventory
NAME: "module 0/RSP0/CPU0", DESCR: "ASR9K Route Switch Processor with 440G/slot Fabric and 12GB "
PID: A9K-RSP440-SE, VID: V02, SN: FOC1636N0XB

NAME: "module 0/0/CPU0", DESCR: "80G Modular Linecard, Packet Transport Optimized"
PID: A9K-MOD80-TR, VID: V06, SN: FOC1644N0MY

NAME: "module 0/0/0", DESCR: "ASR 9000 4-port 10GE Modular Port Adapter"
PID: A9K-MPA-4X10GE, VID: V03, SN: FOC1639N1QD

NAME: "module mau 0/0/0/0", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR , VID: V03 , SN: ONT162403D1

NAME: "module 0/2/CPU0", DESCR: "Cisco ASR 9000 Series SPA Interface Processor-700"
PID: A9K-SIP-700, VID: V02, SN: FOC1718N1KH

NAME: "module 0/2/0", DESCR: "2 port channelized OC12 to DS0 Shared Port Adapter"
PID: SPA-2XCHOC12/DS0, VID: V02, SN: SAL1628GP9Y

NAME: "module mau 0/2/0/0", DESCR: "OC12 SR-1/STM4 I-4"
PID: SFP-OC12-SR , VID: V01 , SN: OPM16240754

NAME: "module mau 0/2/0/1", DESCR: "OC12 SR-1/STM4 I-4"
PID: SFP-OC12-SR , VID: V01 , SN: OPM16240761

NAME: "module 0/2/1", DESCR: "3-port OC3c SFP Optics ATM Shared Port Adapter"
PID: SPA-3XOC3-ATM-V2, VID: V01, SN: JP612430230

NAME: "module mau 0/2/1/2", DESCR: "OC3 SR-1/STM1 I-1"
PID: SFP-OC3-SR , VID: V01 , SN: OCP10335460

NAME: "module 0/2/3", DESCR: "3-port OC3c SFP Optics ATM Shared Port Adapter"
PID: SPA-3XOC3-ATM-V2, VID: V01, SN: JP61243022P

NAME: "module mau 0/2/3/0", DESCR: "OC3 SR-1/STM1 I-1"
PID: SFP-OC3-SR , VID: V01 , SN: OCP10336027

NAME: "module mau 0/2/3/1", DESCR: "OC3 SR-1/STM1 I-1"
PID: SFP-OC3-SR , VID: V01 , SN: OCP10335505

NAME: "module mau 0/2/3/2", DESCR: "OC3 SR-1/STM1 I-1"
PID: SFP-OC3-SR , VID: V01 , SN: OCP10336038

NAME: "module 0/5/CPU0", DESCR: "Cisco ASR 9000 Series SPA Interface Processor-700"
PID: A9K-SIP-700, VID: V02, SN: FOC1718N1L6

NAME: "module 0/5/0", DESCR: "2 port channelized OC12 to DS0 Shared Port Adapter"
PID: SPA-2XCHOC12/DS0, VID: V02, SN: SAL1628GP9P

NAME: "module mau 0/5/0/0", DESCR: "OC12 SR-1/STM4 I-4"
PID: SFP-OC12-SR , VID: V01 , SN: OPM16220810

NAME: "module mau 0/5/0/1", DESCR: "OC12 SR-1/STM4 I-4"
PID: SFP-OC12-SR , VID: V01 , SN: OPM16220820

NAME: "module 0/5/1", DESCR: "2 port channelized OC12 to DS0 Shared Port Adapter"
PID: SPA-2XCHOC12/DS0, VID: V02, SN: SAL1628GPAK

NAME: "module mau 0/5/1/0", DESCR: "OC12 SR-1/STM4 I-4"
PID: SFP-OC12-SR , VID: V01 , SN: OPM16220791

NAME: "module mau 0/5/1/1", DESCR: "OC12 SR-1/STM4 I-4"
PID: SFP-OC12-SR , VID: V01 , SN: OPM16220852

NAME: "module 0/6/CPU0", DESCR: "80G Modular Linecard, Packet Transport Optimized"
PID: A9K-MOD80-TR, VID: V02, SN: FOC1640N1UU

NAME: "module 0/6/1", DESCR: "ASR 9000 20-port 1GE Modular Port Adapter"
PID: A9K-MPA-20X1GE, VID: V01, SN: FOC161288SM

NAME: "module mau 0/6/1/0", DESCR: "Unknown or Unsupported XFP Module"
PID: GLC-LH-SM , VID: V01 , SN: ADDLX11498

NAME: "module 0/7/CPU0", DESCR: "80G Modular Linecard, Service Edge Optimized"
PID: A9K-MOD80-SE, VID: V02, SN: FOC1637N7HM

NAME: "module 0/7/0", DESCR: "ASR 9000 4-port 10GE Modular Port Adapter"
PID: A9K-MPA-4X10GE, VID: V03, SN: FOC1639N1QQ

NAME: "module mau 0/7/0/0", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR , VID: V03 , SN: ONT162403F2

NAME: "module mau 0/7/0/1", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR , VID: V03 , SN: ONT162403WC

NAME: "module 0/7/1", DESCR: "ASR 9000 4-port 10GE Modular Port Adapter"
PID: A9K-MPA-4X10GE, VID: V03, SN: FOC1639N1Q7

NAME: "module mau 0/7/1/0", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR , VID: V03 , SN: ONT162204LH

NAME: "module mau 0/7/1/1", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR , VID: V04 , SN: SPC174508L5

NAME: "module mau 0/7/1/2", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR , VID: V03 , SN: ECL144400ZX

NAME: "module mau 0/7/1/3", DESCR: "Multirate 10GBASE-LR and OC-192/STM-64 SR-1 XFP, SMF"
PID: XFP-10GLR-OC192SR , VID: V03 , SN: ONT162403E9

RP/0/RSP0/CPU0:cr1.st.blfl.vt#
RP/0/RSP0/CPU0:cr1.st.blfl.vt#
RP/0/RSP0/CPU0:cr1.st.blfl.vt#
RP/0/RSP0/CPU0:cr1.st.blfl.vt#show install active summary
Default Profile:
SDRs:
    Owner
Active Packages:
    disk0:asr9k-fpd-px-5.1.3
    disk0:asr9k-mpls-px-5.1.3
    disk0:asr9k-mgbl-px-5.1.3
    disk0:asr9k-mini-px-5.1.3
    disk0:asr9k-k9sec-px-5.1.3

Bitwrangler for FirstLight Fiber Corp