10-02-2012 01:35 PM - edited 03-04-2019 05:44 PM
Hello Everyone,
I have a problem with a client complaining about assymmetric routing, I already performed a traceroute from source to destination and backward and for me all seems to be fine, but now the client is complaining about the hops 2 and 3 with IP: 172.31.234.166, he says there's a routing loop and that's why he has its conection with slowness.
Could please share an idea about why a router could answer twice to a traceroute. Thanks in advance.
Router1#tracer vrf vpn-1111 10.135.47.30
Type escape sequence to abort.
Tracing the route to 10.135.47.30
1 62.237.110.46 0 msec 0 msec 0 msec
2 172.31.234.166 0 msec 0 msec 9 msec
3 172.31.234.166 0 msec 0 msec 0 msec
4 94.142.119.106 42 msec 33 msec 34 msec
5 172.31.232.177 33 msec 25 msec 34 msec
6 * * *
7
From Destination to Source
------------------------------------------------
Router2#tracer vrf vpn-1111 10.121.76.51
Type escape sequence to abort.
Tracing the route to 10.121.76.51
1 172.31.232.177 [AS 65534] 1 msec 1 msec 1 msec
2 94.142.125.21 [AS 65534] [MPLS: Labels 300880/24 Exp 2] 42 msec 30 msec 30 msec
3 172.31.234.166 [AS 65534] 31 msec 30 msec 31 msec
4 62.237.110.47 [AS 65534] 31 msec 31 msec 31 msec
5 * * *
6 * * *
Solved! Go to Solution.
10-04-2012 07:48 AM
Hi,
to avoid possible load sharing issues, I'd try
Router1#tracer vrf vpn-1111 10.135.47.30 probe 1
If you still see the same hop in the output, you would know it's received from a single packet sent with particular TTL value.
I'd also think about some tunnels involved possibly?
In some cases, a tunnel interface can bring one hop visible in a traceroute output twice.
You are running your traceroute through an MPLS backbone, aren't you?
This article might help you, too
http://www.cisco.com/en/US/tech/tk436/tk428/technologies_tech_note09186a008020a42a.shtml#no_mpls
Last but not least:
It's strange you see the same IP addresses in the output while running traceroute both directions.
Usually, the incoming interface IP adddress is used in the ICMP error messages, so you see a different IP address while running traceroute back from the target to the source.
(See 94.142.119.106 and 94.142.125.21 in your case.)
Possibly unnumbered interfaces being used in the backbone?
HTH,
Milan
10-02-2012 04:29 PM
Hi Ruben,
I do not believe that it is a routing loop because it does not exist the condition required for a loop :
R1 route to R2
R2 routes to R3
R3 routes back to R1
The last part would need some kind of redistribution or a particular combination of metric and AD to form a loop. In your case THE SAME router is just querying twice. This is a symptom of suboptimal routing or load-sharing issues. Of course many causes more can be the reason of this slowness but start to investigate on this double-querying router the next-hop table and particular route-map/PBR/filters applied to it. Even a show CDP neighbour will help you to understand the topology of the routers directly connected to it.
One more. Is that ip repeated twice a loop back or a physical interface? If you could post some config of that router that would be great.
One thing more. The MPLS encapsulation is also introducing some CoS values (exp bits) , you should check if this so low value of your experimental bits (EXP) is not introducing delay and retransmissions. Can you share your topology?
Good night
Alessio
Sent from Cisco Technical Support iPad App
10-02-2012 05:49 PM
Hi Alessio,
Thank you very much for your prompt response.
The first thing I thought was in a load sharing weird behavior due that the IPs of the 4th hop in the first traceroute is diferent from the 2nd hop in the second one, but the duplicated IPs appears before those hops ( the 172.31.234.166 ip is a physical interface).
The problem is that I don't have access to the core equipments or diagrams of the core MPLS network, just to the CPE routers.
The times of the traceroute and ping tests are within the SLA, also our nework monitoring tools shows no packet loss and no congestion at all. I just want to have a good answer to give to the client since for me we don't have a problem in our network ( in this case we're just responsible for the 4 hops that I shared, the complete path has 22 hops) and they're trying to pass the hot potato to us. The issue start with a client complaining about packet loss, we provide the tests and they changed the issue to slowness, when we share the times of the traceroutes changed to an asymmetric route issue.
I still thinking about the possibility of a load sharing behavior but without a diagram is going to be hard to explain.
If you can think in something else with the info a provided, it will be great. I'll try to obtain a network diagram or something that helps in this incident.
Thanks again
Ruben
10-04-2012 07:48 AM
Hi,
to avoid possible load sharing issues, I'd try
Router1#tracer vrf vpn-1111 10.135.47.30 probe 1
If you still see the same hop in the output, you would know it's received from a single packet sent with particular TTL value.
I'd also think about some tunnels involved possibly?
In some cases, a tunnel interface can bring one hop visible in a traceroute output twice.
You are running your traceroute through an MPLS backbone, aren't you?
This article might help you, too
http://www.cisco.com/en/US/tech/tk436/tk428/technologies_tech_note09186a008020a42a.shtml#no_mpls
Last but not least:
It's strange you see the same IP addresses in the output while running traceroute both directions.
Usually, the incoming interface IP adddress is used in the ICMP error messages, so you see a different IP address while running traceroute back from the target to the source.
(See 94.142.119.106 and 94.142.125.21 in your case.)
Possibly unnumbered interfaces being used in the backbone?
HTH,
Milan
10-04-2012 04:20 PM
Hi Milan,
I was reading the article and I think this is the best possible answer:
" Router B decrements the TTL value in the MPLS header, drops the packet, and sends an ICMP TTL-exceeded message to the source. Since it was an MPLS packet that was dropped, the return address for the ICMP message must be derived from the source address in the IP header inside the MPLS packet. But that IP address actually may not be known to Router B, so Router B forwards the ICMP messages along the same label switched path (LSP) that the dropped packet was traveling (in the direction toward router C). At the end of the LSP, the label is removed and the ICMP messages are forwarded according to the destination address in the IP header (toward Router C1). "
What is weird is that in the backward path there's no repetition, yesterday
I was performing some other traceroute tests about other cases and I realized that this "problem" occurs in various scenarios on our network even on different vrfs and different path.
I already performed the test you suggested
Router1#tracer vrf vpn-1111 10.135.47.30 probe 1
and I got the hop repeated twice as well.
I've another question:
In which case a traceroutes could show the egress ip address in one of its hops.
Thank you for your response, and thanks for the article's link, very very useful.
10-05-2012 06:39 AM
Hi Ruben,
just some ideas coming to my mind:
a) possibly some tunnel involved?
I had an interesting discussion here some time ago:
https://supportforums.cisco.com/message/3751246
regarding traceroute through GRE tunnels.
So possibly it might be also valid for MPLS TE tunnels somehow?
Read http://www.cisco.com/en/US/tech/tk436/tk428/technologies_tech_note09186a0080125b01.shtml#intro
for some details.
b) Or even MPLS penultimate hop popping (PHP) feature could be the reason?
In fact, there is a stack of two MPLS headers used for MPLS VPNs, isn't it?
So possibly the TTL value from the original IP packest is copied to both MPLS headers but only the outer header TTL value is decremented when the MPLS packet is traversing the backbone?
So imagine the original IP packet having TTL=1 when received by the PE router.
Then a stack of two MPLS headers would be added (both having the TTL=1) by the PE router and the MPLS packet would be sent to the MPLS backbone.
The first P router would receive the packet, decrease the TTL in the outer MPLS header to 0 and sent an ICMP error the way desribed in the article we discussed.
In a case this P router would be also running PHP, it should now forward the packet with the inner MPLS header only. But as the TTL in that header would be also 1, would't it trigger another ICMP error message sent by the same P router?
I'm not an MPLS expert, I'm not sure how TTL values are being copied in a case of MPLS header stack, just guessing here.
On the other hand, I'd expect observing the same behaviour while running traceroute in the opposite direction.
So I can only imagine the backbone not being configured a consistent way and some routers not providing PHP?
HTH,
Milan
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide