Hey Kevin,
Your network is working properly and what you see in
debug ip icmp
is perfectly expected.
The traceroute tool has a gotcha: How does it know that it finally reached the end destination and can stop running (terminate)? It can detect the intermediary routers by sending out packets with TTL 1, displaying the incoming ICMP TTL Exceeded messages, and progressively increasing the TTL - that's all okay. But the end destination may have multiple IP addresses and may decide to respond from a different IP address than the one you're performing traceroute to. If your termination detection for traceroute was based on expecting a response from the same IP address you're doing traceroute to, you could end up with the traceroute never terminating.
So traceroute uses a different trick: It sends out UDP segments destined to high ports that are very likely closed on the destination device (no process is listening on them), and of course, it packages those UDP segments into IP packets with gradually increasing TTL. By definition, UDP payload is never processed on intermediary devices, only on the destination host. Intermediary routers do not process UDP payload. Only when the IP packets arrive at their true destination when their TTL is high enough, the destination host starts processing the UDP segments inside them, but when it finds out they are destined to a closed UDP port, it will send back an ICMP Port Unreachable message.
So the traceroute implementation you're looking at is expecting two types of ICMP messages to come back:
- ICMP TTL Exceeded messages come from intermediary routers and indicate to traceroute it needs to keep running and increasing the TTL
- ICMP Port Unreachable messages come from the final destination and signal traceroute that the TTL is sufficient and it can stop
Hence, what you see is 100% correct.
Please feel welcome to ask further!
Best regards,
Peter
