Using "debug ip packet detail" to follow ICMP traffic across multiple hops

[Jim R]

Hi all,

I'm doing a bit of an experiment at the moment tracking an ICMP reply and the route it takes through our lab network however I've come to a bit of a dead end.  If I do a "debug packet ip detail" on our destination router (Where we are trying to ping to) and the router I am using to send the packet I can clearly see three four things:

1) Sending router: The ICMP type 8 message from the sending router

2) Sending router: The packet going out via the next hop router

3) Recieving router: The ICMP type 0 message going out from the recieving router

4) Recieving router: The packet going out via the next hop router.

However what I was expecting to see was some sort of IP traffic on the routers in the middle stating that they were passing on the traffic but alas I don't see anything.

So my question is:

1) Using "debug ip packet detail" should I be able to see ICMP send/replies/associated packets traversing the network including the routers in the middle.

2) If not what command should I be using.

Cisco details ping usage etc on it's site but only shows pinging a host which is the next hop router.  The other examples just show using "debug ip packet detail" to get back information on the packet being dropped as there is no route to the host.

Perhaps someone could fill in the blanks for me?

Many thanks,

Jamie

[Richard Burts]

Jamie

I believe that the issue is that debug can only report on what the CPU of the router sees. So the sending router must create the packet using its CPU and debug sees the packet. The receiving router must use its CPU to process the packet and debug can report the packet. But on the routers in the middle they are forwarding the packet (probably using CEF or some efficient forwarding mechanism) without needing their CPU to see the packet. If you really want the middle router to have debug report the traffic you would need to disable CEF (and Fast Switching or any other enhanced forwarding mechanism). If the router in the middle must process switch the packet then its CPU will be used and debug would report the traffic.

HTH

Rick

[xthuijs]

Hey Jamie, only packets that are process switched can be seen with debug ip packet detail.

Intermediate routers (if cisco) will CEF switch (hopefully ) the packet and therefore not print any debug messages.

If you have hardware forwarding routers (such as the ASR1K, C10K etc) then that is another level of forwarding abstraction in which you may not see the packet either with these debug commands .

I believe that if you'd disable ip cef (if your platform/sw version) supports that, which means that packets will get process switched (or fast switched) you might be able to see the debug ip packet.

Be careful doing this as it slows down your router significantly and more then you wish for debugs might show

xander

[Rozsa Illes]

Hi Jamie,

An addition to the previous replies:

From 12.4(24)T, you can use "debug ip cef packet" to see CEF switched packets (traffic passing through the router). It needs to have the interface, direction of the traffic, ACL and rate limiting configured to minimize the risk of crashing/hanging the device.

If you just want to see the packets, embedded packet capture is also an option from 12.4(24)T and it can be exported and viewed by wireshark:

http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_packet_capture.pdf

(The size of the capture buffer is quite small on 12.4(24)T, only 512K. It can be increased more in later versions.)

Cheers,

Rose

[Jim R]

Hi guys,

Thanks for the prompt replies very much appreciated.  I think I have 12.3 on these lab devices so at the moment shutting off CEF is probably the way forward for my testing. Interesting that CEF debug wasn't implemented until 12.4, you'd have thought it a very useful tool for seeing exactly what the hell is going on inside the little black box of tricks.

Also as a bit of a side note I am assuming that multicast traffic is not CEF'd?  As I do see quite a bit of multicast traffic flying around when "debug ip packet" is on. 

Anyways, I'll have a play and report back.  Thanks once again for your support, It's the first time in ages I've posted a query and actually had a useful reply back on any forum

Regards,

Jamie

[Jim R]

Just as a bit of a follow up.  I disabled CEF and fast switching on all interfaces I wished to monitor and it worked like a charm.  You could clearly see the packets entering and leaving the interface and being routed appropriatly.

For other I used the "no ip cef" command globally and confirmed CEF was not running by using "show ip cef" and "show ip cef interface " and then by disabling, on a per interface basis, route caching using the interface command "no ip route-cache"

This then forced all traffic via the CPU so I could debug it correctly.

Thanks again all, Much appreciated.

Jamie

[Willie Walton]

Another solution to this is to use traceroutes. That way the router's CPU will have to process the packet in order to decrement the TTL value, and the traffic will show up in the debug. 

traceroute <destination ip>

You can also modify the probe value to send more than 3 packets to each router in the path.

traceroute <destination ip> probe 10