ME3600 transit fragmented mpls packets loss.

kzryumov1 · ‎04-11-2012

Hello colleagues,

We noticed packet loss on ME3600X 24FS

It is transit fragmented mpls labeled packets loss on ME switch.

For example: we have next scheme

prom20 (7609) --- prom20-2 (ME3600) --- novaya-sw-1 (3750me) ---

> zyx_ies1248 (target for ping)

:

1. When we ping from novaya-sw-1 to zyx_ies1248 – there is no packet loss

:

novaya-sw-1#ping 172.31.168.197 size 1500 rep 1000

Type escape sequence to abort.

Sending 1000, 1500-byte ICMP Echos to 172.31.168.197, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/2/17 ms

novaya-sw-1#ping 172.31.168.197 size 2500 rep 1000

Type escape sequence to abort.

Sending 1000, 2500-byte ICMP Echos to 172.31.168.197, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/5/302 ms

2. Ping from me3600 without loss

prom20-2#ping 172.31.168.197 sou 172.31.232.255 size 1500 rep 1000

Type escape sequence to abort.

Sending 1000, 1500-byte ICMP Echos to 172.31.168.197, timeout is 2 seconds:

Packet sent with a source address of 172.31.232.255

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/2/16 ms

prom20-2#ping 172.31.168.197 sou 172.31.232.255 size 2500 rep 1000

Type escape sequence to abort.

Sending 1000, 2500-byte ICMP Echos to 172.31.168.197, timeout is 2 seconds:

Packet sent with a source address of 172.31.232.255

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/4/84 ms

3. Ping from 7600 to end of the link – packets loss exist on ONLY FRAGMENTED LABELED MPLS packets

prom20#ping 172.31.168.197 sou 172.31.175.193 siz 1500 rep 1000

Type escape sequence to abort.

Sending 1000, 1500-byte ICMP Echos to 172.31.168.197, timeout is 2 seconds:

Packet sent with a source address of 172.31.175.193

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/1/12 ms

prom20#ping 172.31.168.197 sou 172.31.175.193 siz 2500 rep 1000

Type escape sequence to abort.

Sending 1000, 2500-byte ICMP Echos to 172.31.168.197, timeout is 2 seconds:

Packet sent with a source address of 172.31.175.193

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 98 percent (981/1000), round-trip min/avg/max = 1/4/92 ms

prom20#trace 172.31.168.197 sou 172.31.175.193

Type escape sequence to abort.

Tracing the route to novaya-ies1248-1.mgmt.redcom.net (172.31.168.197)

1 212.19.1.190 [MPLS: Label 698 Exp 0] 0 msec 0 msec 0 msec

2 novaya-sw-1-gi1-1-1.redcom.net (212.19.1.178) 4 msec 0 msec 4 msec

3 * *

3. Ping from 7600 to me3600 – without loss:

prom20#ping prom20-2 size 1500 rep 1000

Translating "prom20-2"...domain server (212.19.2.1) [OK]

Type escape sequence to abort.

Sending 1000, 1500-byte ICMP Echos to 212.19.0.54, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/2/16 ms

prom20#ping prom20-2 size 2500 rep 1000

Translating "prom20-2"...domain server (212.19.2.1) [OK]

Type escape sequence to abort.

Sending 1000, 2500-byte ICMP Echos to 212.19.0.54, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/2/48 ms

LFIB on me3600 for interface to 3750me:

prom20-2#sh mpls forw int gi0/1

Local Outgoing Prefix Bytes Label Outgoing Next Hop

Label Label or Tunnel Id Switched interface

521 687 172.17.254.8/29 0 Gi0/1 212.19.1.178

577 639 172.30.68.44/30 0 Gi0/1 212.19.1.178

599 2618 172.30.69.44/30 0 Gi0/1 212.19.1.178

615 81 172.30.70.208/28 0 Gi0/1 212.19.1.178

671 2432 172.31.136.80/29 0 Gi0/1 212.19.1.178

680 1853 172.31.136.160/28 \

0 Gi0/1 212.19.1.178

698 Pop Label 172.31.168.192/28 \

59411830 Gi0/1 212.19.1.178

740 1679 172.31.171.176/28 \

0 Gi0/1 212.19.1.178

797 1998 172.31.175.160/28 \

0 Gi0/1 212.19.1.178

804 2276 172.31.175.240/28 \

0 Gi0/1 212.19.1.178

805 1170 172.31.176.0/21 0 Gi0/1 212.19.1.178

820 1956 172.31.193.8/29 0 Gi0/1 212.19.1.178

821 495 172.31.193.16/29 0 Gi0/1 212.19.1.178

837 129 172.31.193.184/29 \

0 Gi0/1 212.19.1.178

862 Pop Label 172.31.194.232/29 \

30312 Gi0/1 212.19.1.178

Local Outgoing Prefix Bytes Label Outgoing Next Hop

Label Label or Tunnel Id Switched interface

941 Pop Label 212.19.0.15/32 7501818396 Gi0/1 212.19.1.178

946 1372 212.19.0.20/32 0 Gi0/1 212.19.1.178

989 1911 212.19.0.71/32 0 Gi0/1 212.19.1.178

1038 2187 212.19.1.148/30 0 Gi0/1 212.19.1.178

2833 Pop Label 212.19.25.32/30 7210 Gi0/1 212.19.1.178

2850 2473 212.19.26.28/30 0 Gi0/1 212.19.1.178

LFIB from 7609 to me3600:

prom20#sh mpls forw int te8/2

Local Outgoing Prefix Bytes Label Outgoing Next Hop

Label Label or VC or Tunnel Id Switched interface

231 2833 212.19.25.32/30 0 Te8/2 212.19.1.190

523 Pop Label 212.19.1.176/30 0 Te8/2 212.19.1.190

1560 Pop Label 212.19.0.54/32 165617 Te8/2 212.19.1.190

1655 698 172.31.168.192/28 0 Te8/2 212.19.1.190

1917 862 172.31.194.232/29 0 Te8/2 212.19.1.190

2192 941 212.19.0.15/32 7011643469 Te8/2 212.19.1.190

3496 Pop Label 172.31.232.255/32 0 Te8/2 212.19.1.190

We've done a lot of tests with software versions, MTU, ip mpls mtu, ip mtu size changing but haven't got any luck.

Also there does not any drops / discard / miss etc. on interfaces.

Thanks.

Best regards, Konstantin.

nkarpysh · ‎04-11-2012

Hi Konstantin,

Just wondering if you tried a quick test setting the ICMP timeout to more than 2 seconds (e.g. 3 or 5) for ping where you see drops. This is to make sure that we indeed missing replies and not just delay those due to fragment re-assembeling.

Nik

HTH,
Niko

kzryumov1 · ‎04-11-2012

Thank you for reply.

prom20#ping 172.31.168.197 sou 172.31.175.193 siz 2500 rep 1000 tim 3

Type escape sequence to abort.

Sending 1000, 2500-byte ICMP Echos to 172.31.168.197, timeout is 3 seconds:

Packet sent with a source address of 172.31.175.193 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!

Success rate is 98 percent (981/1000), round-trip min/avg/max = 1/4/60 ms

prom20#ping 172.31.168.197 sou 172.31.175.193 siz 2500 rep 1000 tim 5

Type escape sequence to abort.

Sending 1000, 2500-byte ICMP Echos to 172.31.168.197, timeout is 5 seconds:

Packet sent with a source address of 172.31.175.193 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!.!!!!!!

Success rate is 98 percent (980/1000), round-trip min/avg/max = 1/4/16 ms

izlber24w · ‎04-11-2012

hey, i replaced a ME6524 with ME3600x and i got the same problems. Next to the ME3600x are other ME6524 Switches and they work fine. On the ME3600x i cant´s see any giant packets!!???

he´re my interface config

int gi 0/1

mtu 9000

ip mtu 1500

ip addr..

mpls ip

mpls label protocol ldp

any idea?

nkarpysh · ‎04-12-2012

Hi,

You can try checking following command for all interfaces in the path:

show interface GiX/Y switching (where X/Y is the mod/port numbers - please also not try to use ? for this command - it is hidden - just copy paste). You can see by that if any interface is sending the packet to RP (CPU) and we get drops there.

By that we can narrow down it to particular node.

Nik

HTH,
Niko

kzryumov1 · ‎04-19-2012

Thank you for your reply but there is no drops anywere.

But I found one interesting thong.

ME3600X drop Checksum 0xFFFF packet when act as a label disposition LSR

Symptoms: Cisco ME3600X act as a label disposition Edge-LSR.
When received MPLS packets with Checksum 0xFFFF that will keep continue drop with Ipv4HeaderErr and Ipv4ChecksumError at Nile.

Conditions: This symptom is seen while label pop action at the Edge-LSR.

Workaround: There is no workaround

I suppose we faced with this bug, didn't I?

kzryumov1 · ‎10-03-2012

The Nile HW forwarding ASIC used by ME3600 doesn't support fragmentation in hardware so it punts the packet that need fragmentation to CPU queue 20 i.e. ip_mpls_mtu_fail_q.

The CPU queue has rate-limiting policer because of which intermittent packet drops happen ME-3600X

#sh pl qos policer cpu 20 0 ########### Stats for CPU queue 20 ########## Internal Qnum: 18196 Policer conform: 99 (packets) 150381 (bytes) Policer exceed: 1 (packets) 1519 (bytes) RM Drops: 1 (packets) 1547 (bytes) #############################################

ME-3600X#

This is root-cause of drops that we are seeing.

kzryumov1 · ‎10-03-2012

The Nile HW forwarding ASIC used by ME3600 doesn't support fragmentation in hardware so it punts the packet that need fragmentation to CPU queue 20 i.e. ip_mpls_mtu_fail_q.

The CPU queue has rate-limiting policer because of which intermittent packet drops happen ME-3600X

#sh pl qos policer cpu 20 0 ########### Stats for CPU queue 20 ########## Internal Qnum: 18196 Policer conform: 99 (packets) 150381 (bytes) Policer exceed: 1 (packets) 1519 (bytes) RM Drops: 1 (packets) 1547 (bytes) #############################################

ME-3600X#

This is root-cause of drops that you are seeing.

Robert Placencia · ‎08-07-2013

Was there ever a fix action found for this? I think I may be running into the same issue.