tracerout issue in MPLS experiment

[morris0117]

Hey guys, I got a problem.

In the picture attached, R1 & R4 are running iBGP with 192.168.12.1, 192.168.34.4 and "next-hop-self" eabled. Now R6 wants to ping 5.5.5.5 which is R5's loopback0, and I know it would fail. Because the next-hop of 5.5.5.5 is 192.168.12.1 for R4 (learned from BGP), R4 would push a label for the prefix 192.168.12.0/24 and send the packet to R3. But R3 would pop the label off because for R2 the 192.168.12.0/24 is directly connected.

The problem is when I perform traceroute on R6, it shows:
   1 192.168.46.4 44 msec 32 msec 28 msec
   2 * * *
   3 * * *
It looks like the packet drops at R3...?

But I also check debug icmp of R2, R3 and R4. And it seems the packet did drop at R2 whose debug log shows:
   *Mar 1 00:07:34.463: ICMP: dst (5.5.5.5) host unreachable sent to 192.168.46.6

Why does the output of R6 show like that? Maybe I don't really understand what the asterisks mean.

Thanks all  

[MHM Cisco World]

MHM

[MHM Cisco World]

I think @Harold Ritter  correct 
check the ping is success then share the config of BGP 
MHM

[Harold Ritter]

Hi @MHM Cisco World ,

"no mpls ip propagate-ttl" sets the TTL to 255 on the ingress PE, so only the ingress PE and egress PE will show in the traceroute output. It will not help if the R4 to R6 subnet is not known on the egress PE and CE.

Regards,

[MHM Cisco World]

Totally correct' 

He can ping if he success but the trace route is show missing hop then he need no mpls ip propagate-ttl

MHM

[Harold Ritter]

Hi @MHM Cisco World ,

> He can ping if he success

Well, the issue in the OP scenario, is that the ping from R6 to R5 will not work even if he specifies the source address because the LSP is not end to end between R4 and R1. The LSP for 192.168.12.1 stops at R2 and R2 doesn't have a route to 5.5.5.5.

This is why in such a scenario you would always use the PE loopback address to ensure the LSP is end to end between the ingress and egress PEs. 

Regards,

[MHM Cisco World]

Sorry but R5 and R6 is CE not PE.

The label started show when traffic hit PE (ingress or egress) or P.

MHM

[Harold Ritter]

Hi @MHM Cisco World ,

I know R5 and R6 are CEs. The LSP between the ingress and egress PE R4 and R1 is incomplete and that is the issue and the OP is aware of it.

His question was why the trace route stops at R3 rather than R2. I provided the explanation for it.

Regards,

[M02@rt37]

Hello @morris0117 

The asterisks '*' in the traceroute output typically indicate that the device is not sending ICMP Time Exceeded messages back. In a typical traceroute, each hop along the path sends back an ICMP Time Exceeded message to the source indicating that the TTL has expired.

In your case, the absence of ICMP Time Exceeded messages in the traceroute output might be due to specific configurations on the routers, such as the `no mpls ip propagate-ttl` command. This command instructs the router to not propagate the original IP TTL value in the MPLS label stack.

When MPLS is in use, the routers along the path may not decrement the TTL for the MPLS label stack, leading to the original TTL reaching the destination without any intermediate devices decrementing it. As a result, the routers might not generate ICMP Time Exceeded messages for the traceroute.

https://packetlife.net/blog/2008/dec/22/disabling-mpls-ttl-propagation/

[Harold Ritter]

Hi @morris0117 ,

>But R3 would pop the label off because for R2 the 192.168.12.0/24 is directly connected. 

The behavior you described is indeed the expected behavior. To avoid this behavior, you should use the loopback interface addresses to peer between R1 and R4.

The packets are not being dropped at R3, but R3 can't send the ICMP TTL Expired message back to the source (192.168.46.6). When a TTL expires and the packet was received via MPLS, the router forwards the ICMP TTL expired message using the original LSP and the LSP tail end router forwards it using its local routing table. In your scenario, R2 is the LSP tail end and it doesn't have a route to 192.168.46.6, which causes the ICMP TTL expired to be dropped. 

Regards,

[morris0117]

Thanks guys! I found the answer myself which I can truly understand, mainly referring to the following information:
https://www.cisco.com/c/en/us/support/docs/multiprotocol-label-switching-mpls/mpls/200510-Trace-route-in-MPLS-network.html#anc3
https://www.youtube.com/watch?v=X0xcefcA3sw&list=PLREYIVnkcgLUl2WWoZ0smeoNdQFcxKddw&index=6

On LSP (Label Switched Path), the ICMP pkt of traceroute travels in a different way. When the TTL=2 pkt arrives at R3 (TTL→0), since it's a label pkt, it cannot just return to R6 (and neither does it know the route). R3 will reset the source IP address as the incoming interface of the pkt (192.168.34.3), reset the destination address as the original source address (192.168.46.6), reset the TTL=255, pop the label (or swap the label if R3 wasn't the penultimate hop), and then forward the pkt to R2. However, R2 doesn't know the route to 192.168.46.6 so the time exceeded message of R3 couldn't make it to R6.

In a properly configured scenario by using loopback interface address in iBGP peering between R1 & R4 mentioned by @Harold Ritter, the label pkt would first make it to R1 and R1 knows how to send it back to 192.168.46.6.