At Cisco TAC, when the phones ring for P1/P2 outages, a typical problem we hear from customers is about connectivity issues between end-hosts and/or end-users' application performance issues.
In an enterprise core/distribution and datacenter network, where virtual switching solutions (like, VSS or vPC) are highly deployed with large number of port-channels / redundant links, it is challenging and critical to trace the path of the packet to identify the device, link and port causing the above-mentioned problems.
How does a TAC engineer really find the device / link / port in issue ? The answer is ... by using the innovative tools discussed here.
(1) NetDR / EthAnalyzer
Net Driver (NetDR) tool is used in Catalyst 6500 and Cisco 7600 platforms to capture the traffic sent to (and received from) the CPU. EthAnalyzer performs the same functions in Nexus platforms and is an implementation of TShark in NX-OS.
When poor performance is experienced with specific application, using these tools (in the devices along the traffic path) helps us to confirm that specific traffic is getting hardware-switched and NOT processed by the CPU.
Here is a sample NetDR capture - reporting layer2, layer3 and layer4 information along with ingress interface.
This tool allows network administrators to replicate network traffic, especially when there is a need to monitor subset of traffic on high bandwidth links (thats how it differs from traditional R/ER/SPAN). VACL Capture is supported only on Catalyst 6500 / Cisco 7600 platforms. ACL Capture is supported only on Nexus7000 M1-series modules in 5.2(1) release onwards.
The examples discussed are very simple but powerful in terms of narrowing down the device(s)
It can be very invaluable if number of packets for a specific flow can be counted at each layer-2 hop, specifically for faster problem resolution.
(3) Embedded Logic Analyzer Module (ELAM)
First of all, ELAM is NOT a supported feature and is very specific to the hardware architecture. Anything and everything about it may change from version to version without any notice. ELAM is supported in Catalyst 6500, Cisco 7600, Nexus 7000 platforms.
ELAM is used by TAC as a capturing tool to validate:
(a) Was the packet received ?
(b) What did the packet look like ?
(c) Was the packet rewritten/alterted ? If so, how ?
(d) Where was the packet sent ?
The following diagram shows the workflow of using the ELAM tool:
As you can see, the first step is to identify the ingress forwarding engine, which requires knowledge on hardware architecure. It is highly recommened for customers to review a detailed document on ELAM available here.
A relevant TS Newsletter Article has discussed different packet capturing tools available in Cisco devices:
Hi Guys,When I plug in my switch the system LED is blinking green and all the ports led lit up for a few seconds then the ports led goes off, and a few seconds after the system LED is off. The switch seems like its shutdown after. Any ide...
I am trying to ping Loopback interface IP from PE Router in Underlay SDWAN vpn 0. But its not Pinging from either side.we are using Static Routing between PE and vEdge.is there anything special we need to do?
I have a Cisco ISR 4451-X running version 16.8.1 (Fuji) and I'm trying to configure a couple of 24-port SM-X-ES3-24-P (running 15.0(2)EJ) into it using BDI. THIS document says to use the commands below. However, in v16.8.1, #3 is not a valid command on th...
dear alli have :c2900-universalk9-mz.SPA.154-3.M1.bin" (Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M1, RELEASE SOFTWARE (fc1)) with memory as belowCisco CISCO2911/K9 (revision 1.0) with 483328K/40960K bytes of memory.Processo...