Two hops with the same IP address.

[Ruben Leal]

Hello Everyone,

I have a problem with a client complaining about assymmetric routing, I already performed a traceroute from source to destination and backward and for me all seems to be fine, but now the client is complaining about the hops 2 and 3  with IP: 172.31.234.166, he says there's a routing loop and that's why he has its conection with slowness.

Could please  share an idea about why a router could answer twice to a traceroute. Thanks in advance.

Router1#tracer vrf vpn-1111 10.135.47.30

Type escape sequence to abort.

Tracing the route to 10.135.47.30

  1 62.237.110.46 0 msec 0 msec 0 msec                

  2 172.31.234.166 0 msec 0 msec 9 msec

  3 172.31.234.166 0 msec 0 msec 0 msec

  4 94.142.119.106 42 msec 33 msec 34 msec

  5 172.31.232.177 33 msec 25 msec 34 msec

  6  *  *  *

  7

  From Destination to Source

------------------------------------------------

  Router2#tracer vrf vpn-1111 10.121.76.51

Type escape sequence to abort.

Tracing the route to 10.121.76.51

  1 172.31.232.177 [AS 65534] 1 msec 1 msec 1 msec

  2 94.142.125.21 [AS 65534] [MPLS: Labels 300880/24 Exp 2] 42 msec 30 msec 30 msec

  3 172.31.234.166 [AS 65534] 31 msec 30 msec 31 msec

  4 62.237.110.47 [AS 65534] 31 msec 31 msec 31 msec

  5  *  *  *

6  *  *  *

[Alessio Andreoli]

Hi Ruben,

I do not believe that it is a routing loop because it does not exist the condition required for a loop :

R1 route to R2

R2 routes to R3

R3 routes back to R1

The last part would need some kind of redistribution or a particular combination of metric and AD to form a loop. In your case THE SAME router is just querying twice. This is a symptom of suboptimal routing or load-sharing issues. Of course many causes more can be the reason of this slowness but start to investigate on this double-querying router the next-hop table and particular route-map/PBR/filters applied to it. Even a show CDP neighbour will help you to understand the topology of the routers directly connected to it.

One more. Is that ip repeated twice a loop back or a physical interface? If you could post some config of that router that would be great.

One thing more. The MPLS encapsulation is also introducing some CoS values (exp bits) , you should check if this so low value of your experimental bits (EXP) is not introducing delay and retransmissions. Can you share your topology?

Good night

Alessio

Sent from Cisco Technical Support iPad App

[Ruben Leal]

Hi Alessio,

Thank you very much for your prompt response.

The first thing I thought was in a load sharing weird behavior due that the IPs of the 4th hop in the first traceroute is diferent from the 2nd hop in the second one, but the duplicated IPs appears before those hops ( the 172.31.234.166 ip is a physical interface).

The problem is that I don't have access to the core equipments or diagrams of the core MPLS network, just to the CPE routers.

The times of the traceroute and ping tests are within the SLA, also our nework monitoring tools shows no packet loss and no congestion at all. I just want to have a good answer to give to the client since for me we don't have a problem in our network ( in this case we're just responsible for the 4 hops that I shared, the complete path has 22 hops) and they're trying to pass the hot potato to us. The issue start with a client complaining about packet loss, we provide the tests and they changed the issue to slowness, when we share the times of the traceroutes changed to an asymmetric route issue.

I still thinking about the possibility of a load sharing behavior but without a diagram is going to be hard to explain.

If you  can think in something else with the info a provided, it will be great. I'll try to obtain a network diagram or something that helps in this incident.

Thanks again

Ruben

[milan.kulik · Accepted solution]

Hi,

to avoid possible load sharing issues, I'd try

Router1#tracer vrf vpn-1111 10.135.47.30 probe 1

If you still see the same hop in the output, you would know it's received from a single packet sent with particular TTL value.

I'd also think about some tunnels involved possibly?

In some cases, a tunnel interface can bring one hop visible in a traceroute output twice.

You are running your traceroute through an MPLS backbone, aren't you?

This article might help you, too

http://www.cisco.com/en/US/tech/tk436/tk428/technologies_tech_note09186a008020a42a.shtml#no_mpls

Last but not least:

It's strange you see the same IP addresses in the output while running traceroute both directions.

Usually, the incoming interface IP adddress is used in the ICMP error messages, so you see a different IP address while running traceroute back from the target to the source.

(See 94.142.119.106 and 94.142.125.21 in your case.)

Possibly unnumbered interfaces being used in the backbone?

HTH,

Milan

[Ruben Leal]

Hi Milan,

I was reading the article and I think this is the best possible answer:

" Router B decrements the TTL value in the MPLS header, drops the packet, and sends an ICMP TTL-exceeded message to the source. Since it was an MPLS packet that was dropped, the return address for the ICMP message must be derived from the source address in the IP header inside the MPLS packet. But that IP address actually may not be known to Router B, so Router B forwards the ICMP messages along the same label switched path (LSP) that the dropped packet was traveling (in the direction toward router C). At the end of the LSP, the label is removed and the ICMP messages are forwarded according to the destination address in the IP header (toward Router C1). "

What is weird is that in the backward path there's no repetition, yesterday

I was performing some other traceroute tests about other cases and I realized that this "problem" occurs in various scenarios on our network even on different vrfs and different path.

I already performed the test you suggested

Router1#tracer vrf vpn-1111 10.135.47.30 probe 1

and I got the hop repeated twice as well.

I've another question:

In which case a traceroutes could show the egress ip address in one of its hops.

Thank you for your response, and thanks for the article's link, very very useful.

[milan.kulik]

Hi Ruben,

just some ideas coming to my mind:

a) possibly some tunnel involved?

I had an interesting discussion here some time ago:

https://supportforums.cisco.com/message/3751246

regarding traceroute through GRE tunnels.

So possibly it might be also valid for MPLS TE tunnels somehow?

Read http://www.cisco.com/en/US/tech/tk436/tk428/technologies_tech_note09186a0080125b01.shtml#intro

for some details.

b) Or even MPLS penultimate hop popping (PHP) feature could be the reason?

In fact, there is a stack of two MPLS headers used for MPLS VPNs, isn't it?

So possibly the TTL value from the original IP packest is copied to both MPLS headers but only the outer header TTL value is decremented when the MPLS packet is traversing the backbone?

So imagine the original IP packet having TTL=1 when received by the PE router.

Then a stack of two MPLS headers would be added (both having the TTL=1) by the PE router and the MPLS packet would be sent to the MPLS backbone.

The first P router would receive the packet, decrease the TTL in the outer MPLS header to 0 and sent an ICMP error the way desribed in the article we discussed.

In a case this P router would be also running PHP, it should now forward the packet with the inner MPLS header only. But as the TTL in that header would be also 1, would't it trigger another ICMP error message sent by the same P router?

I'm not an MPLS expert, I'm not sure how TTL values are being copied in a case of MPLS header stack, just guessing here.

On the other hand, I'd expect observing the same behaviour while running traceroute in the opposite direction.

So I can only imagine the backbone not being configured a consistent way and some routers not providing PHP?

HTH,

Milan