11-17-2011 12:24 PM - edited 03-07-2019 03:27 AM
I have a situation where a vlan has periodically high utilization (240/255 tx/rx or higher) and at the same time CPU goes up past 90% (its normally around 5%) on a 6500 with two SUP720-3B. The host trying to send traffic via that vlan sees very slow rates - like 10 - 20Mbps or so on a gig link. No ports have high error counts during the problem (via a collect top).
During backups:
CPU utilization for five seconds: 99%/74%; one minute: 51%; five minutes: 47%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
145 19201223521565136805 1226 23.08% 10.66% 10.04% 0 IP Input
20 3554055924279033665 0 0.87% 0.30% 0.18% 0 IPC Seat Manager
301 2838486282006984232 141 0.55% 0.22% 0.21% 0 Port manager per
139 27996776 230313636 121 0.23% 0.07% 0.06% 0 CDP Protocol
141 1736 1664 1043 0.23% 0.09% 0.07% 2 Virtual Exec
343 44781136 115349940 388 0.23% 0.58% 0.19% 0 SNMP ENGINE
129 289725521388813235 20 0.07% 0.02% 0.00% 0 Earl NDE Task
42 8861440 95430240 92 0.07% 0.02% 0.00% 0 Per-Second Jobs
147 7457980 70432232 105 0.07% 0.00% 0.00% 0 ADJ resolve proc
Any thoughts on what to look for?
thanks
John
11-20-2011 12:15 AM
Hi John,
CPU is high due to interrupts - value after slash - /74%. Those are for traffic to be handled by CPU. As you find particular VLAN getting excess traffic - it seems to be that that traffic mostly sent to CPU and we need to understand why.
To do that we can sniff those packets:
On VLAN buffer:
show buffer input-int vlan VLAN_NUM packet (do it multiple times to catch multiple packets)
Also sniff CPU (debug safe to run - specificaly for High CPU issues):
debug netdr cap rx (you can also narrowed it down to particular vlan enterning "vlan X" in the end of that command)
show netdr cap
So you will see packets sent to CPU for processing causing spikes - you then can diagnose the sources and destination to understand why those are sent to CPU and if possible to stop those closer to the source.
Hope this helps.
Nik
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide