Every Hop is the Same in Traceroute from Cisco 3650, but have Connectivity

[Dean Romanelli]

Any ideas if this is a bug with Cisco 3650 03.06.04.E? Both destination IP's are across the country from where this switch is located, and they are not accessed over VPN's. Just regular commercial grade internet circuit. 

SW2ElizabethCore-PLA3650A#trace 8.28.0.1
Type escape sequence to abort.
Tracing the route to 8.28.0.1
VRF info: (vrf in name/id, vrf out name/id)
1 8.28.0.1 0 msec 10 msec 0 msec
2 8.28.0.1 10 msec 10 msec 10 msec
3 8.28.0.1 10 msec 20 msec 10 msec
4 8.28.0.1 10 msec 10 msec 20 msec
5 8.28.0.1 10 msec 20 msec 10 msec
6 8.28.0.1 10 msec * 10 msec
7 * 10 msec *
8 8.28.0.1 40 msec 30 msec 40 msec

SW2ElizabethCore-PLA3650A#trace 192.84.18.11
Type escape sequence to abort.
Tracing the route to 192.84.18.11
VRF info: (vrf in name/id, vrf out name/id)
1 192.84.18.11 10 msec 0 msec 10 msec
2 192.84.18.11 10 msec 10 msec 10 msec
3 192.84.18.11 10 msec 20 msec 10 msec
4 192.84.18.11 10 msec 10 msec 20 msec
5 192.84.18.11 20 msec 10 msec 20 msec
6 192.84.18.11 10 msec * 10 msec
7 * * *
8 192.84.18.11 60 msec * 80 msec
9 192.84.18.11 60 msec 60 msec 60 msec

SW2ElizabethCore-PLA3650A#ping 8.28.0.1 repeat 50
Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 8.28.0.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 30/39/50 ms

SW2ElizabethCore-PLA3650A#ping 192.84.18.11 repeat 50
Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 192.84.18.11, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 60/62/70 ms

[Richard Burts]

This is a very odd situation. The results of ping and of traceroute show that you do have IP connectivity to those addresses. The strange thing is that the output of trace is showing the destination address as the source of the trace probe packets at the intermediate steps but it should be showing the address of the intermediate device. I would assume that this is some bug. I am not sure what kind of configuration could produce this result, but it is probably worth asking whether there is anything unusual in the configuration of this switch. Also might be helpful to know something about the topology of this network. What is the next hop device and is that device doing any type of address translation or other manipulation that might impact the address used for probe responses.

HTH

Rick

[Dean Romanelli]

Hi Rick,

Thanks for replying.  The next hop is a Cisco ASA firewall, which has a VPN to the data center, and another VPN where all of our internet bound traffic gets dumped to for processing before it enters the internet.  The address in the trace above is a flow that I've allowed to break out of that tunnel, because it is VOIP, so it goes direct out the ASA firewall to the ISP via the catchall NAT (nat (inside,outside) source dynamic any interface). 

I have 4 other sites with the same topology and traces to the same IP's from the same hardware platform do not produce these results, so I suspect you are right that it is a bug.

[Richard Burts]

Thanks for the information. Would I be correct in assuming that the version of software on this ASA is different from the version on the other ASAs? If so it would support the belief that this is a bug. 

HTH

Rick

[paul driver]

Hello

Looks like the trace is possibly hitting a Firewall and the reply's you see is the natted address on that firewall