Traceroute with multiple next hops

[mip]

Hi All,

In brief- we have iWAN in our environment and all branch routers have 2 ISP connections ( both are public internet, layer 3). Now we are in a process of replacing one of these DIA (direct internet access) connections with a layer 2 circuit from the service provider. I believe it is called VPLS service from ISP perspective.

We have migrated only one location at the moment to this layer 2 service and what we are seeing now is when using traceroute, we have multiple next hops:

 R1-4331#traceroute vrf TEST 8.8.8.8
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 10.254.250.1 28 msec
10.254.250.2 5 msec
10.254.250.1 28 msec
2 10.16.2.1 7 msec
10.16.1.1 31 msec
10.16.2.1 12 msec

..............................

9 75.149.230.110 32 msec
64.233.174.9 9 msec
75.149.230.26 31 msec
10 8.8.8.8 9 msec * 10 msec

I did some researching and what i found is that , this means that from router side, there are multiple routes that these packets can take to reach the destination.

Is this behavior normal? Is this fine from a routing perspective? Why do we see 10.254.250.1 twice in the first hop and so on....?

My boss told me that this is not normal.

I would appreciate any inputs!!!

Thank you.

 Note: the rest of the branches which are still running on layer 3 service (DIA) are showing one ip address as next hop.

[balaji.bandi]

In the path found 2 best routes and the path elected best route to reach the destination.

BB

[mip]

Hi Balaji,

In that case then , that is a normal behavior?! Correct?

[Joseph W. Doherty]

Yes, it's normal (as also noted by John).

Traceroute will round-robin across equal cost path, per hop. So, to your earlier question, you'll see the same hop IP reappear depending on how many traceroute packets you've generated. (Increase the traceroute packet count, and the round-robin will likely be more clear.)

Generally, flows of the same packet will stay on one path, i.e. those packets will not usually round-robin at any one hop. As traceroute packets aren't considered one flow, again, you normally see them round-robin.

[Richard Burts]

The original poster had really figured out that this behavior was because there are multiple entries in the routing table for that destination. He then asks 3 questions. Here are my responses to those questions. 

1) Is this normal? Yes this is normal. And if your boss says it is not normal then you need to find a polite way to inform him that he is not correct.

2) Is this fine? Yes this is the expected behavior when there is a redundant path toward the destination. And having redundancy is a good thing.

3) Why does 10.254.250.1 show up twice? To explain this I would start with a review of the way that trace route works. It works by sending probe packets toward the destination address and by manipulating the TTL of the packets. Typically it sends 3 probe packets with each RRL (though this can be changed). So this trace route sends the first probe packet with TTL equal 1 and gets a response from 10.254.250.1. The second probe packet with TTL equal 1 is sent using the second path in the routing table and receives a response from 10.254.250.2. The third probe packet with TTL equal 1 is sent using the first path (there are only two paths toward this destination in the routing table) and receives a response from 10.254.250.1. (if you had changed the trace route to use 4 probes for each TTL then the response for the fourth probe would have been 10.254.250.2)

HTH

Rick

[johnlloyd_13]

hi,

it's normal. it could be due to load balancing on the routing protocol used on the service provider cloud.

[mip]

Thank you All for your inputs.

I was having hard time explaining this behavior to my self! But now things looks clearer and the day is better:)
Richard,
I tested the traceroute with 4 probes count and I indeed got what you said:

R1-4331#traceroute vrf TEST 8.8.8.8 probe 4
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 10.254.250.1 28 msec
10.254.250.2 5 msec
10.254.250.1 28 msec
10.254.250.2 6 msec
2 10.16.2.1 7 msec
10.16.1.1 31 msec
10.16.2.1 12 msec
10.16.1.1 8 msec
You are right:)
Thank you all for your help!

[Richard Burts]

You are welcome. I am glad that my explanation was helpful and that now you have a better understanding of what is happening with the trace route. These forums are excellent places to ask questions and to learn about networking. I hope to see you continue to be active in the forums.

HTH

Rick