Traceroute is bouncing between redundant hops that have an equal cost.

[BruteForce]

Below is my topology. I have a very basic OSPF setup and every layer 3 switch sees every network on 'sh ip route', and every advertising router is known in the ospf database. If I try to do a traceroute from DN1 to VLAN2000 hosted on DN2 or its loopback, here's the output I see.

DN1_Lab#traceroute 10.10.2.1
Type escape sequence to abort.
Tracing the route to 10.10.2.1
VRF info: (vrf in name/id, vrf out name/id)
1 5.5.5.9 2 msec
5.5.5.1 2 msec
5.5.5.9 2 msec
2 *
5.5.5.14 2 msec *

As shown above, the traceroute bounces between the two CNs and both routes have a cost of 2. Oddly, if I do a traceroute to 5.5.5.14 on DN2 from DN1, there's no bouncing between the CNs.

DN1_Lab#traceroute 5.5.5.14
Type escape sequence to abort.
Tracing the route to 5.5.5.14
VRF info: (vrf in name/id, vrf out name/id)
1 5.5.5.9 2 msec 2 msec 1 msec
2 5.5.5.14 1 msec 2 msec *

Something odd about this network is that every single interface is a layer 2 interface.. The "point to point" vlans are trunked across but they still participate in OSPF. I'm not sure if this is causing some spanning tree issues which would explain the bouncing between CNs.

[balaji.bandi]

Post all the config and show route and ospf information.

EDIT - I was reading different post thinking that - it was loop ---

[Kanan Huseynli]

Hi,

there is no loop in your topology.

Most probably, ip cef load-sharing algorithm determines different hops based on hashing result when you try to reach 10.10.2.1, but the same next-hop when you try to reach 5.5.5.14 (by default "universal" method is configured on most devices).

As i understand it is lab environment, so safely you can use "ip cef load-sharing algorithm original" global config mode command on all routers/ switches and check. It will do hash based on source/destination IP, thus always the same path will be used for the same source-destination.

Below is one of the cef load balancing documents:

https://www.cisco.com/c/en/us/td/docs/ios/12_4t/ip_switch/configuration/guide/tceflbs.html

[MHM Cisco World]

I think this your second post about this issue ?
anyway I have two issue now need to analysis, I will run lab see the TTL in SVI of L3SW. 
I will update you soon 

[BruteForce]

Sorry for two posts, they're related. My original issue was related to the command "no ip unreachables" and the affect it has on traceroute, but after digging and applying this knowledge to what I was ultimately trying to fix (the traceroute bouncing between the CNs) I learned that there was more than one issue. I'm not at the CCNP level so I'm trying to troubleshoot and learn about the CEF process right now and how it applies to what I know about traceroute. 

[MHM Cisco World]

Yes, we you mark as solved I return to home do lab and see even with no ip unreachable the traceroute is success.

anyway  
VLAN1000 SVI is config in which SW? in DN1?

[BruteForce]

Vlan1000 is hosted on DN1 and Vlan2000 is hosted on DN2. I have it trunked on a port facing the CN for each respective DN (but only on the DN side) so I can get an up/up status.

For example gi1/0/1 on DN1 has: switchport trunk allowed vlan 100, 1000. The CN only has 100 trunked back which is intentional. I only want the SVIs to use OSPF unlike the point to point vlans which are actually trunked across and participate in ospf process 1 area 0.

DN2 has a similar setup to CN2 on port 4 with vlans 400 and 2000.

[Kanan Huseynli]

Hi @BruteForce ,

did you try what I mentioned before regarding CEF issue?

[Richard Burts]

First let me deal with the suggestion that this behavior might be some spanning tree issue. Spanning tree has nothing to do with this behavior. This is about basic ip routing behavior and not about spanning tree.

To explain the issue I would start with how traceroute works. Traceroute sends probe packets to the destination and varies the Time To Live. Many implementations, including Cisco, send 3 packets with each TTL but it is possible to do traceroute with a different number of packets. So if you traceroute 10.10.2.1 it will start with 3 packets with TTL of 1. Pretty clearly the routing table of DN1 has 2 entries for that destination (so 2 paths toward the destination). The first probe packet uses path 1, the second probe packet uses path 2, and the third probe packet uses path 1. Then it sends 3 probe packets with TTL of 2. The first probe uses path 2, then path 1, then path 2. This alternation is not a problem but is a basic expected behavior.

Clearly in the routing table of DN1 there is a single entry for 5.5.5.14. I do not have enough information to explain the traceroute to 5.5.5.14. Seeing the configurations and especially the content of the routing tables would be very helpful in this.

[Kanan Huseynli]

Hi @Richard Burts ,

I'm sorry but when I talked about spanning-tree?)

My first comment about loop is related to L3 loop (OSPF loop) and I commented this because balaji.bandi talked about OSPF loop prevention. I just mentioned that there is no loop in this scenario.

Regarding your traceroute explaination - I also mentioned this in my first comment. Hashing result may end up this behaviour and depending on hash result for next-hop we may or may not see 2 hops in the traceroute output.

I suggested to author to test with old method of cef load-sharing in which only source-destionation IP are considered (so, the same hash for each trace probe).

[BruteForce]

Hello, yes, I've been looking at CEF, and I'm still trying to learn how to use it better. I wasn't able to see the algorithms for which path is preferred on my device, but I do think traceroute is behaving this way due to load balancing. I'm still doing some testing, but I noticed that if I make traceroute send only 1 probe, there's no bouncing. If I send 2 probes it hit both CNs once. If I send 3 probes, which is the default anyway as I understand, one of the CNs is hit twice. If I send 4, both CNs are hit twice. 

I've gone down a deeper rabbit hole than I thought I would but I'm learning. Right now my theory is there's no bouncing between the CNs at all, but I haven't actually done a packet capture to prove it. The DN is simply sending out 3 packets, and per-destination load balancing is enabled by default, and the 3 packets traceroute sends out are different flows, so they get load balanced. As for why one path is picked twice over the other I don't know yet. 

[Kanan Huseynli · Accepted solution]

---

@BruteForce wrote:

 As for why one path is picked twice over the other I don't know yet. 

---

Hello,

I have already explained the reason. Reason is different hashing result (result means chosen next-hop). You have 3 probes, Cisco trace is based on UDP protocol and uses random source port with 34343 being destination address. Default algorithm for hashing considers source-destination L4 port numbers.

I have simple topology to explain this:

(loopback interface) R1--R2 (loopback interface), between routers I have 2 interlinks 10.10.10.1-2 /24 and 10.10.20.1-2/24.

Now, lets see what happens when I do trace from R2 loopback to R1 loopback (4 times):

Router2#traceroute 8.8.8.8 so lo 0 num probe 3
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 10.10.10.1 1 msec
10.10.20.1 1 msec
10.10.10.1 0 msec
Router2#traceroute 8.8.8.8 so lo 0 num probe 3
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 10.10.20.1 1 msec
10.10.10.1 0 msec
10.10.20.1 0 msec
Router2#traceroute 8.8.8.8 so lo 0 num probe 3
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 10.10.10.1 1 msec
10.10.20.1 1 msec
10.10.10.1 0 msec
Router2#traceroute 8.8.8.8 so lo 0 num probe 3
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 10.10.20.1 0 msec
10.10.10.1 1 msec
10.10.20.1 0 msec

as you see, in each test either 10.1 or 20.1 two times, but another one time. If you do with probe 1 it will be either 10.1 or 20.1.

Depending on OS it can be played with different algorithm type and may result only one next-hop.

[mlund]

For the second traceroute, if nothing special is configured, but just enabling ospf. Then this is what is expected. Looking at the route entries, both cn2 and dn2 will advertise the 5.5.5.12/30 as connected network. So from the dn1 point of view there is one hop to reach 5.5.5.12/30 via cn2, but it is 2 hops via cn1.

[BruteForce]

Yes I see how this makes sense. the 5.5.5.12/30 network simply has a preferred path because one route takes less hops. 

[MHM Cisco World]

check solution