CISCO default data pattern getting packet loss

[nkowlahar]

I am trying to bring up a MPLS connection between a Juniper MX960 and a Cisco 3750 through a Ethernet over Sonet connection.  We modified the configs so it is just a simple IP'd interface on both devices and we are continuing to get about 30%-50% packet loss when the Cisco is generating the rapid ping requests with datagrams above 300bytes.  However if the Juniper generates the rapid ping requests we are able to pass giant sized datagrams with a df-bit and do not experience frame loss.  If we replace the Cisco 3750 with a laptop running a rapid ping software or a JDSU smartclass test set we do not have any issues with ICMP.  After further research we noticed that we are only losing packets when the derfault data pattern 0xabcd generated by the Cisco is being sent went preforming a rapid ping request.  If we just modify the data pattern by one Hexidecimal we no longer experience this behavior.  We do have suspect of maybe the Metro Ethernet over Sonet connection may be filtering or policing ICMP traffic, but this is a Layer 1 platform with Layer 2 switching subscriber cards and all other data patterns run clean across it.  We are also able to pass live datastreams proving throughput with Ethernet test sets.  It seems as though this only happens with ICMP and the default cisco data pattern.   

Has anyone come across this before or can provide any insight on this behavior?

N09CA1C3:578:#ping 216.149.18.37 source Gi0/0.1609 size 1500 repeat 100          

Type escape sequence to abort.

Sending 100, 1500-byte ICMP Echos to 216.149.18.37, timeout is 2 seconds:

Packet sent with a source address of 216.149.18.38

..!.!..!...!!!.!!.!!!..!!!...!!.!!!.!..!.!.........!.!.!!.!.!..!!.!!.!

!!!!.!...!!...!...!!...!!!..!.

Success rate is 47 percent (47/100), round-trip min/avg/max = 6/8/62 ms

N09CA1C3:578:#ping 216.149.18.37 source Gi0/0.1609 size 1500 repeat 100 data abcd

Type escape sequence to abort.

Sending 100, 1500-byte ICMP Echos to 216.149.18.37, timeout is 2 seconds:

Packet sent with a source address of 216.149.18.38

.!.!!.!...!!!.!.!!...!.!..!!..!...!.!.!!.!!..!.!!..!!!..!!!!.!!!!!!..!

...!!!!.....!!.!!....!!.!.....

Success rate is 49 percent (49/100), round-trip min/avg/max = 6/9/74 ms

N09CA1C3:578:#ping 216.149.18.37 source Gi0/0.1609 size 1500 repeat 100 data abcc

Type escape sequence to abort.

Sending 100, 1500-byte ICMP Echos to 216.149.18.37, timeout is 2 seconds:

Packet sent with a source address of 216.149.18.38

Packet has data pattern 0xABCC

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (100/100), round-trip min/avg/max = 6/8/36 ms

N09CA1C3:578:#ping 216.149.18.37 source Gi0/0.1609 size 1500 repeat 1000 data abce

Type escape sequence to abort.

Sending 1000, 1500-byte ICMP Echos to 216.149.18.37, timeout is 2 seconds:

Packet sent with a source address of 216.149.18.38

Packet has data pattern 0xABCE

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 6/8/68 ms

[rsavena14]

I have not encountered this before, but i would like to know also the cause of this.

[nkowlahar]

posted this late last night.. bringing it back up to see if this has been encountered by anyone

[rsimoni]

Hey Nick,

first step is to understand who is actually dropping the packets and in which direction.

So I suggest you to sniff the traffic (you can add another switch between metro circuit and your 3750 and the juniper) and make sure where you have the loss and in which direction (requests are lost or replies).

The idea is to make sure that the 3750 really puts on wire the icmp packets (it should but it worth double-checking) and more important confirm whether the Juniper is receiving some (drops happenong in the cloud) or all of them (dropping the replies).

R

[nkowlahar]

Thanks..  We have already ran this through a switch and did a port mirror with a packet sniffer on the line.  the ICMP packets are leaving the 3750, however they never get back to the Juniper.  This happens only in the direction from the 3750 to the Juniper when the 3750 is intiating the ping request. 

I have done some further research and found a couple of suspects that might  point to the TDM design.  the Cisco data pattern might actually be triggering some TDM test patterns.  Or the scrambling function might need to be turned off on the Metro Ethernet connection if there are any POS ports involved.  I am still working on verifying these theories, but thats all I have for now.  thoughts ???

[Grant Woertendyke]

I know this is months later, but I hav having a very similar issue with an MPLS connection to a remote site. I was hoping something was found, and the fix could be posted here.

If this was a provider issue where there was no cisco related fix, I'd be happy to know how it all worked out.

-G

[kiansimcisco]

Juniper router ------- UTP cable ------- convector ---------- fiber ---------------- convector ------------- UTP cable ------------ Cisco router

I have encounter the same issue. Ping with 0xABCD is always getting drop; Ping with 0xABCC is not dropping.

Ping test from Juniper default data pattern is all zeros, so it cannot detect the problem. You have to manually define the data pattern to 0xABCD.

The issue is resolved by replacing both convectors.