Hello Kevin,
>> I am wondering if there is an IOS command that can backtrace what process hogged the CPU and found the root cause
not after a reload, but I understand that you needed to find a quick workaround.
if possible in similar cases you should take sh proc cpu and sh log and to save them in a text file, before reloading the device (logs could be retrivied on syslog server if the device exports log messages to an external server).
Also sw interrupts could be the main users of cpu resources, the second number in 5 seconds cpu usage says how much cpu is used by SW interrupts so it is possible to have cpu at 100% and sh proc cpu sorted does not show processes for more then 20 percent just to make an example.
Very high cpu usage can be triggered by a bridging loop that leads to a broadcast storm that makes the cpu to process a rapidly increasing amount of broadcast frames ( no TTL exists at OSI layer 2 so frames are not dropped after circulating many times in the loop and each cycle a multiplication effect happens).
Combined to a bridging loop there can be other effects for example on HSRP groups: if two Vlans broadcast domains are joined the device can receive on Vlan X the multicast HSRP hello packets of Vlan Y and this can cause problems even to C6500 devices. A fixup for this is to use different MD5 password for different HSRP groups in different vlans so that if for any accident two vlans are joined HSRP group N does not consider frames of group M and viceversa.
Hope to help
Giuseppe
