Here is the troubleshooting checklist which should be ready before customers/partners contact Cisco TAC:
- Firmware Version of APIC and Switch
- Hardware Model
- Download Switch and APIC techsupport logs
- Problem description (Symptoms with details)
- Business impact (eg, what kind of services are impacted, how many users are affected?)
- When did the problem start
- What changes done before the problem started
- What troubleshooting actions that has been performed, and what's the initial analysis.
If you couldn't provide all the information, just try your best to provide as much details as you could.
Useful commands set to get more info of the problem details, and start initial troubleshooting.
On APIC:
# show version
# show controller
# show switch
if It's Endpoint connectivity issue, using following show commands to retrieve the EP details, alternatively, you may use EP tracker tool under APIC GUI Operations.
# show endpoints ip x.x.x.x
# show endpoints mac x:x:x:x:x:x:x
e.g.
apic1# show endpoints ip 10.10.10.100
Legends:
(P):Primary VLAN
(S):Secondary VLAN
Dynamic Endpoints:
Tenant : lindawa
Application : lindawa-vmm
AEPg : lindawa-db
End Point MAC IP Address Node Interface Encap Multicast Address
----------------- ---------------------------------------- ---------- ------------------------------ --------------- ---------------
00:50:56:91:5B:F0 10.10.10.100 101 102 vpc lindawa-vpc-to-srv37 vlan-829 not-applicable
On Switch:
# show module
# show vlan ex | grep <encap_vlan>
# show vrf
# show ip route vrf <vrf-name>
# show endpoint ip x.x.x.x
# show endpoint mac x:x:x:x:x:x:x
# show system internal epm vpc (if vPC is configured)
# show system internal epm endpoint ip x.x.x.x
# show system internal epm endpoint mac x:x:x:x:x:x:x
# show logging ip access-list internal packet-log deny | grep <ip/mac> // you may use this command to check contract drops on leaf switch, if it's vPC case, remember to check this command on both vPC member.
Here is an example for a detailed problem description.
ACI version: 3.1(1i)
Inter-Pod connectivity issue within same EPG.
ping from the src in pod1 could not be able to reach the dst in pod2.
src ip: 10.15.20.22 (pod1, leaf201/202)
dst ip: 10.15.20.59 (pod2, leaf203/204)
Recent changes:
customer did hardware replacement for both IPN routers(from Nexus N9K-C93180YC-EX to N9K-C9372PX-E), then started seeing this issue.
In some complicated scenarios, we also need to collect audit log, event history, fault record history .etc from APIC, this is very general information which is good to have before opening TAC case, please attach it to TAC case once case is opened.
Login to one of APIC by ssh, then:
#bash
Then run below:
mkdir /tmp/tac-outputs
cd /tmp/tac-outputs
show running-config > show_run.txt
icurl 'http://localhost:7777/api/class/firmwareARunning.xml' > firmwareARunning.xml
icurl 'http://localhost:7777/api/class/fabricNode.xml' > fabricNode.xml
icurl 'http://localhost:7777/api/class/faultInfo.xml' > faultInfo.xml
icurl 'http://localhost:7777/api/class/aaaModLR.xml?order-by=aaaModLR.created|desc&page-size=100000' > aaaModLR.xml
icurl 'http://localhost:7777/api/class/faultRecord.xml?order-by=faultRecord.created|desc&page-size=100000' > faultRecord.xml
icurl 'http://localhost:7777/api/class/eventRecord.xml?order-by=eventRecord.created|desc&page-size=100000' > eventRecord.xml
cd /tmp
tar zcvf tac-outputs.tgz tac-outputs
cp tac-outputs.tgz /data/techsupport
Please download the file tac-outputs.tgz and upload it to TAC case.
