Weird Traceroute's

[colin.painter]

Hello all,

Hoping someone can provide some suggestions on t-shooting an issue i'm seeing in my network.

Some configuration insight, my gateway (172.16.0.10 is a router that also has 10.10.10.192/26 configured on another interface (10.10.10.253). There is another router on this subnet on 10.10.10.252 and a switch, (the destination in question) 10.10.10.242.

When doing a traceroute to the switch on 10.10.10.242, i get the following result:

1. 172.16.0.10 (gateway)

2. 10.10.10.252

3. 10.10.10.253

4. 10.10.10.252

5. 10.10.10.253

6. 10.10.10.252

7. 10.10.10.253

8. 10.10.10.252

9. 10.10.10.253

10. 10.10.10.242

So eventually it does reach the destination but it bounces between the two routers multiple times first. So far, I can't figure out why - on my default gateway the best route to 10.10.10.242 is via 'connected' and all devices have proxy arp disabled. So i can't see any reason for the traffic bouncing like this.

Any one come across this before or able to offer a t-shooting suggestion i have missed?

Thanks.

[Richard Burts]

This is a pretty strange issue and I am not sure that I have seen this particular thing before. It would help if you could post a diagram that clarifies the topology of the network. To get things started here are a few questions and suggestions:

- can you clarify whether this is happening with real routers and switches, or is done on some simulator?

- what type of router is your gateway? and what type of router is the router that is connected to the gateway?

- what type of connection is between the two routers? Is it a router physical Ethernet interface, a switch routed interface, or a switch vlan access port?

- you tell us that your gateway router has another interface (10.10.10.253) and on this network is a router (10.10.10.252) and a switch (10.10.10.242). Does the switch physically connect to the gateway, to the other router, or to both?

- before you start this traceroute are there entries in the arp table on the gateway for 10.10.10.252 and 10.10.10.242? And a similar question about whether the second router has arp entries for 10.10.10.253 and 10.10.10.242?

- it might be helpful to run debug arp on both routers, run the traceroute, and look to see if any arp activity is generated.

HTH

Rick

[colin.painter]

Hi Rick,

Thanks for the reply, i've attached a topology. Where Switches 1,2,3 & 4 are concerned the VLAN for the 10.10.10.192/26 network is tagged traffic and passes at a L2 level.

- can you clarify whether this is happening with real routers and switches, or is done on some simulator?

This is a live network, not a simulated one.

- what type of router is your gateway? and what type of router is the router that is connected to the gateway?

The gateway is a Cisco ASR1001-X, as is Router-2 in the topology.

- what type of connection is between the two routers? Is it a router physical Ethernet interface, a switch routed interface, or a switch vlan access port?

As per the topology, each router goes in to a switch on trunk ports, tagged for the specific VLAN. They communicate between a switch ring of 10Gb fibres that is controlled by STP to prevent loops (STP has been checked and an interface is disabled correctly)

- you tell us that your gateway router has another interface (10.10.10.253) and on this network is a router (10.10.10.252) and a switch (10.10.10.242). Does the switch physically connect to the gateway, to the other router, or to both?

As per topology via switches

- before you start this traceroute are there entries in the arp table on the gateway for 10.10.10.252 and 10.10.10.242? And a similar question about whether the second router has arp entries for 10.10.10.253 and 10.10.10.242?

Yes, there are ARP entries already and the MAC's correspond to the devices owning those IP's. Have also tried an ARP-Cache clear to get them to re-learn.

For reference, switches 1-4 are HP Procurves and the switch that has 10.10.10.242 configured is a HP Comware switch. All have STP configured.

A traceroute to 10.10.10.252 does not result in the same thing and appears to route correctly. It shows:

1. 172.16.0.10

2. 10.10.10.252

Thanks

Colin

[Richard Burts]

Colin

Thanks for the additional information. Given the drawing showing the topology I am surprised that the router at 10.10.10.252 shows up in the traceroute at all. So it raises some more questions. Do any of the switches have layer 3 routing enabled? Or are they all acting as layer 2 switches? 

Given the fact that the first router has an interface in the subnet where the destination is located and that it does have an arp entry which has the correct mac address I would expect this to be the behavior:

- the router would build the first traceroute packet as a unicast frame with a destination mac address of the switch at 10.10.10.242 and transmit the frame on its Ethernet interface.

- the first switch would receive the unicast frame, look for the destination mac address in its switching table, and forward the frame to the next switch.

- the next switch would repeat those steps, receiving a unicast frame, look for the destination mac address in its switching table, and forward the frame to the destination switch.

- note that in this behavior there is no reason why the router at 10.10.10.253 should process the traceroute frame or send a response to it.

Since the observed behavior is different we need to evaluate reasons why the switch is forwarding the frame not as a unicast frame with a known destination mac address but as some other type of frame (most likely as a broadcast)

.- is there some reason why the router is not sending the frame as unicast? Perhaps a packet capture could verify the addressing of the frame that the router is sending? Perhaps seeing the configuration of the router might shed some light on this.

- is there some reason why the switch is applying some type of layer 3 forwarding logic rather than the simple layer 2 forwarding of a frame with a known destination mac address?

- is there some reason why the switch would not have the destination mac address in its switching table, resulting in flooding of the frame to all interfaces in the vlan?

HTH

Rick

[colin.painter]

Hi Rick,

Thanks for the additional response. After spending a large amount of time t-shooting, getting captures etc and finding absolutely no rhyme or reason, I arranged a maintenance window and tore down my entire config and rebuilt it, with no topology or firmware changes at all.

Issue went away. Most bizarre... but glad it seems to be resolved.

Thanks for the detailed replies to assist, much appreciated.

Colin

[Richard Burts]

Colin

thanks for the update. Glad to know that you got it working by starting over and rebuilding the configs, with no changes made. Looks like something was hung or out of sync somehow and rebuilding configs got it straightened out.

HTH

Rick