high cpu/vlan utilization

[John Biederstedt]

I have a situation where a vlan has periodically high utilization (240/255 tx/rx or higher) and at the same time CPU goes up past 90% (its normally around 5%) on a 6500 with two SUP720-3B.  The host trying to send traffic via that vlan sees very slow rates - like 10 - 20Mbps or so on a gig link.  No ports have high error counts during the problem (via a collect top). 

During backups:

CPU utilization for five seconds: 99%/74%; one minute: 51%; five minutes: 47%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

145  19201223521565136805       1226 23.08% 10.66% 10.04%   0 IP Input

  20   3554055924279033665          0  0.87%  0.30%  0.18%   0 IPC Seat Manager

301   2838486282006984232        141  0.55%  0.22%  0.21%   0 Port manager per

139    27996776 230313636        121  0.23%  0.07%  0.06%   0 CDP Protocol

141        1736      1664       1043  0.23%  0.09%  0.07%   2 Virtual Exec

343    44781136 115349940        388  0.23%  0.58%  0.19%   0 SNMP ENGINE

129    289725521388813235         20  0.07%  0.02%  0.00%   0 Earl NDE Task

  42     8861440  95430240         92  0.07%  0.02%  0.00%   0 Per-Second Jobs

147     7457980  70432232        105  0.07%  0.00%  0.00%   0 ADJ resolve proc

Any thoughts on what to look for?

thanks

John

[nkarpysh]

Hi John,

CPU is high due to interrupts -  value after slash - /74%. Those are for traffic to be handled by CPU. As you find particular VLAN getting excess traffic - it seems to be that that traffic mostly sent to CPU and we need to understand why.

To do that we can sniff those packets:

On VLAN buffer:

show buffer input-int vlan VLAN_NUM  packet  (do it multiple times to catch multiple packets)

Also sniff CPU (debug safe to run - specificaly for High CPU issues):

debug netdr cap rx    (you can also narrowed it down to particular vlan enterning "vlan X" in the end of that command)

show netdr cap

So you will see packets sent to CPU for processing causing spikes - you then can diagnose the sources and destination to understand why those are sent to CPU and if possible to stop those closer to the source.

Hope this helps.

Nik