cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
899
Views
0
Helpful
5
Replies

6509-E Switch High CPU (IOS-Base process)

hi@alll

we have two 6509 switches configured with VSS. Few days ago we had a power cut and both switches was down.

After both switches coming up again but with issues.

installed IOS image is: s72033_rp_ADIPSERVICESK9_WAN_VM 12.2(33)SXI2a

I could find the issue on interface port-channel 110 for VSS Link.

When i disable this interface, the cpu comes down.

Problem is, both switch goes to active and not active/standby.

When i enable this interface again then VSS status come up and i have one active and one standby switch, but CPU is then very high.

see below the show command:

SolCen_VSS_Core#show switch virtual role

Switch Switch Status Preempt Priority Role Session ID
Number Oper(Conf) Oper(Conf) Local Remote
------------------------------------------------------------------
LOCAL 1 UP FALSE(N ) 100(100) ACTIVE 0 0
REMOTE 2 UP FALSE(N ) 100(100) STANDBY 4791 2587


In dual-active recovery mode: No

SolCen_VSS_Core#show switch virtual
Switch mode : Virtual Switch
Virtual switch domain number : 100
Local switch number : 1
Local switch operational role: Virtual Switch Active
Peer switch number : 2
Peer switch operational role : Virtual Switch Standby


SolCen_VSS_Core#show proces
SolCen_VSS_Core#show processes cpu sort
SolCen_VSS_Core#show processes cpu sorted
CPU utilization for five seconds: 100%/81%; one minute: 97%; five minutes: 64%
PID TID 5secUtil 1minUtil 5minUtil
16407 6 79.2% 75.3% 41.8%
16407 13 4.4% 2.5% 1.4%
16407 19 4.3% 2.2% 2.2%
16432 5 3.7% 1.5% 0.6%
16407 18 3.4% 2.8% 1.7%
16407 7 1.7% 3.1% 3.9%
16407 3 0.7% 1.8% 1.0%
16407 8 0.5% 0.5% 0.2%
16407 16 0.4% 2.5% 2.2%

And here the VSS configuration:

switch virtual domain 100
 switch mode virtual
 
 redundancy
 main-cpu
  auto-sync running-config
 mode sso
 
 interface Port-channel110
 no switchport
 no ip address
 switch virtual link 1
 mls qos trust cos
 no mls qos channel-consistency
!
interface Port-channel120
 no switchport
 no ip address
 switch virtual link 2
 mls qos trust cos
 no mls qos channel-consistency
 
 interface TenGigabitEthernet1/5/4
 no switchport
 no ip address
 mls qos trust cos
 channel-group 110 mode on
!
interface TenGigabitEthernet1/5/5
 no switchport
 no ip address
 mls qos trust cos
 channel-group 110 mode on
 
 interface TenGigabitEthernet2/5/4
 no switchport
 no ip address
 mls qos trust cos
 channel-group 120 mode on
!
interface TenGigabitEthernet2/5/5
 no switchport
 no ip address
 mls qos trust cos
 channel-group 120 mode on
!

interface GigabitEthernet1/9/47
 no switchport
 no ip address
 dual-active fast-hello
 
 interface GigabitEthernet2/9/47
 no switchport
 no ip address
 dual-active fast-hello

We changed the X2-10GE-CX4 already but nothing happened. Same Issue.

Anybody a idea??

Thanks

regards

Matthias

5 Replies 5

Jose Fonseca
Cisco Employee
Cisco Employee

Hi Matthias, 

Do you think can share the following command:

show proc cpu sorted | ex 0.00

With this command we can confirm which process is generating the high cpu if any. Because this output:  "CPU utilization for five seconds: 100%/81%" tells that the issue is caused due to punted traffic to the CPU. 

This means that 100% of cpu utilization and 81% is due to traffic punted to the cpu for process switch instead of hardware switch. 

Post the output of the requested command and will see. 

Hi Jose,

thanks for your reply.

See below output of the command:

SolCen_VSS_Core#show processes cpu sorted | ex 0.00
CPU utilization for five seconds: 100%/81%; one minute: 100%; five minutes: 100%
PID     TID      5secUtil 1minUtil 5minUtil
16407   6         78.9%   78.6%    78.5%
16407   10         4.9%    2.7%     2.4%
16407   18         3.9%    2.5%     1.1%
16432   3          3.3%    2.7%     2.2%
16407   19         3.1%    2.8%     2.3%
16407   7          1.9%    2.6%     2.6%
16407   3          0.9%    1.8%     1.6%
16407   16         0.9%    1.4%     0.7%
16407   8          0.5%    0.5%     0.5%
16432   5          0.5%    0.5%     0.9%
16433   1          0.3%    0.3%     0.3%
1       10         0.1%    0.3%     0.3%
16407   1          0.1%    0.4%     1.4%
16432   4          0.0%    0.0%     0.0%
16432   8          0.0%    0.5%     0.5%
16432   2          0.0%    0.0%     0.0%
16431   10         0.0%    0.0%     0.0%
16432   1          0.0%    0.0%     0.0%
16432   6          0.0%    0.0%     0.0%
16431   7          0.0%    0.0%     0.0%
16431   6          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
16431   3          0.0%    0.0%     0.0%
16431   1          0.0%    0.0%     0.0%
16431   8          0.0%    0.0%     0.0%
16432   7          0.0%    0.0%     0.0%
16431   9          0.0%    0.0%     0.0%
16430   12         0.0%    0.0%     0.0%
16430   8          0.0%    0.0%     0.0%
16430   7          0.0%    0.0%     0.0%
16430   6          0.0%    0.0%     0.0%
16430   5          0.0%    0.0%     0.0%
16430   4          0.0%    0.0%     0.0%
16430   3          0.0%    0.0%     0.0%
16430   2          0.0%    0.0%     0.0%
16429   6          0.0%    0.0%     0.0%
16429   5          0.0%    0.0%     0.0%
16429   4          0.0%    0.0%     0.0%
16430   11         0.0%    0.0%     0.0%
16429   2          0.0%    0.0%     0.0%
16430   10         0.0%    0.0%     0.0%
16428   7          0.0%    0.0%     0.0%
16428   6          0.0%    0.0%     0.0%
16430   9          0.0%    0.0%     0.0%
16428   4          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
16429   3          0.0%    0.0%     0.0%
16428   2          0.0%    0.0%     0.0%
16428   1          0.0%    0.0%     0.0%
16427   4          0.0%    0.0%     0.0%
16427   3          0.0%    0.0%     0.0%
16427   2          0.0%    0.0%     0.0%
16427   1          0.0%    0.0%     0.0%
16426   5          0.0%    0.0%     0.0%
16426   4          0.0%    0.0%     0.0%
16426   3          0.0%    0.0%     0.0%
16426   2          0.0%    0.0%     0.0%
16425   9          0.0%    0.0%     0.0%
16425   8          0.0%    0.0%     0.0%
16425   7          0.0%    0.0%     0.0%
16425   6          0.0%    0.0%     0.0%
16425   5          0.0%    0.0%     0.0%
16425   3          0.0%    0.0%     0.0%
16425   2          0.0%    0.0%     0.0%
16424   6          0.0%    0.0%     0.0%
16424   5          0.0%    0.0%     0.0%
16424   4          0.0%    0.0%     0.0%
16424   3          0.0%    0.0%     0.0%
16429   1          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
16424   1          0.0%    0.0%     0.0%
16423   12         0.0%    0.0%     0.0%
16428   5          0.0%    0.0%     0.0%
16423   6          0.0%    0.0%     0.0%
16423   5          0.0%    0.0%     0.0%
16423   4          0.0%    0.0%     0.0%
16423   3          0.0%    0.0%     0.0%
16423   2          0.0%    0.0%     0.0%
16422   4          0.0%    0.0%     0.0%
16428   3          0.0%    0.0%     0.0%
16422   2          0.0%    0.0%     0.0%
16433   2          0.0%    0.0%     0.0%
16433   3          0.0%    0.0%     0.0%
16424   2          0.0%    0.0%     0.0%
16421   2          0.0%    0.0%     0.0%
16421   1          0.0%    0.0%     0.0%
16420   4          0.0%    0.0%     0.0%
16420   3          0.0%    0.0%     0.0%
16420   2          0.0%    0.0%     0.0%
16420   1          0.0%    0.0%     0.0%
16419   4          0.0%    0.0%     0.0%
16419   3          0.0%    0.0%     0.0%
16419   2          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
16419   1          0.0%    0.0%     0.0%
16418   4          0.0%    0.0%     0.0%
16418   3          0.0%    0.0%     0.0%
16418   2          0.0%    0.0%     0.0%
16418   1          0.0%    0.0%     0.0%
16417   4          0.0%    0.0%     0.0%
16417   3          0.0%    0.0%     0.0%
16417   2          0.0%    0.0%     0.0%
16417   1          0.0%    0.0%     0.0%
16416   5          0.0%    0.0%     0.0%
16416   4          0.0%    0.0%     0.0%
16416   3          0.0%    0.0%     0.0%
16416   2          0.0%    0.0%     0.0%
16416   1          0.0%    0.0%     0.0%
16415   6          0.0%    0.0%     0.0%
16415   4          0.0%    0.0%     0.0%
16415   3          0.0%    0.0%     0.0%
16415   2          0.0%    0.0%     0.0%
16415   1          0.0%    0.0%     0.0%
16414   4          0.0%    0.0%     0.0%
16414   3          0.0%    0.0%     0.0%
16414   2          0.0%    0.0%     0.0%
16414   1          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
16413   6          0.0%    0.0%     0.0%
16413   5          0.0%    0.0%     0.0%
16413   4          0.0%    0.0%     0.0%
16413   3          0.0%    0.0%     0.0%
16413   2          0.0%    0.0%     0.0%
16413   1          0.0%    0.0%     0.0%
16412   4          0.0%    0.0%     0.0%
16412   3          0.0%    0.0%     0.0%
16412   2          0.0%    0.0%     0.0%
16412   1          0.0%    0.0%     0.0%
16411   4          0.0%    0.0%     0.0%
16423   7          0.0%    0.0%     0.0%
16411   2          0.0%    0.0%     0.0%
16411   1          0.0%    0.0%     0.0%
16410   4          0.0%    0.0%     0.0%
16410   3          0.0%    0.0%     0.0%
16421   3          0.0%    0.0%     0.0%
16410   1          0.0%    0.0%     0.0%
16409   4          0.0%    0.0%     0.0%
16409   3          0.0%    0.0%     0.0%
16409   2          0.0%    0.0%     0.0%
16409   1          0.0%    0.0%     0.0%
16408   5          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
16408   4          0.0%    0.0%     0.0%
16408   3          0.0%    0.0%     0.0%
16408   2          0.0%    0.0%     0.0%
16408   1          0.0%    0.0%     0.0%
16433   4          0.0%    0.0%     0.0%
16433   5          0.0%    0.0%     0.0%
16407   17         0.0%    0.0%     0.0%
16433   6          0.0%    0.0%     0.0%
16407   15         0.0%    0.6%     1.5%
16407   14         0.0%    0.0%     0.0%
16407   13         0.0%    0.9%     1.7%
16407   12         0.0%    0.0%     0.0%
16407   11         0.0%    0.0%     0.0%
16433   7          0.0%    0.0%     0.0%
16407   9          0.0%    0.0%     0.0%
16433   8          0.0%    0.0%     0.0%
16422   3          0.0%    0.0%     0.0%
16422   1          0.0%    0.0%     0.0%
16407   5          0.0%    0.0%     0.0%
16407   4          0.0%    0.0%     0.4%
16421   4          0.0%    0.0%     0.0%
16407   2          0.0%    0.0%     0.0%
16411   3          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
16406   6          0.0%    0.0%     0.0%
16406   5          0.0%    0.0%     0.0%
16406   4          0.0%    0.0%     0.0%
16406   3          0.0%    0.0%     0.0%
16405   5          0.0%    0.0%     0.0%
16405   3          0.0%    0.0%     0.0%
16405   2          0.0%    0.0%     0.0%
16405   1          0.0%    0.0%     0.0%
16404   3          0.0%    0.0%     0.0%
16404   2          0.0%    0.0%     0.0%
16404   1          0.0%    0.0%     0.0%
16403   1          0.0%    0.0%     0.0%
16402   2          0.0%    0.0%     0.0%
16402   1          0.0%    0.0%     0.0%
16386   2          0.0%    0.0%     0.0%
16386   1          0.0%    0.0%     0.0%
12305   18         0.0%    0.0%     0.0%
12305   17         0.0%    0.0%     0.0%
12305   16         0.0%    0.0%     0.0%
12305   15         0.0%    0.0%     0.0%
12305   14         0.0%    0.0%     0.0%
12305   13         0.0%    0.0%     0.0%
12305   12         0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
12305   11         0.0%    0.0%     0.0%
12305   10         0.0%    0.0%     0.0%
12305   9          0.0%    0.0%     0.0%
12305   8          0.0%    0.0%     0.0%
12305   7          0.0%    0.0%     0.0%
12305   6          0.0%    0.0%     0.0%
12305   5          0.0%    0.0%     0.0%
12305   4          0.0%    0.0%     0.0%
12305   3          0.0%    0.0%     0.0%
12305   2          0.0%    0.0%     0.0%
12305   1          0.0%    0.0%     0.0%
12304   8          0.0%    0.0%     0.0%
12304   7          0.0%    0.0%     0.0%
12304   6          0.0%    0.0%     0.0%
12304   5          0.0%    0.0%     0.0%
12304   4          0.0%    0.0%     0.0%
12304   3          0.0%    0.0%     0.0%
12304   2          0.0%    0.0%     0.0%
12304   1          0.0%    0.0%     0.0%
12303   5          0.0%    0.0%     0.0%
12303   3          0.0%    0.0%     0.0%
12303   2          0.0%    0.0%     0.0%
12303   1          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
12302   4          0.0%    0.0%     0.0%
12302   3          0.0%    0.0%     0.0%
12302   2          0.0%    0.0%     0.0%
12302   1          0.0%    0.0%     0.0%
12301   1          0.0%    0.0%     0.0%
12300   5          0.0%    0.0%     0.0%
12300   2          0.0%    0.0%     0.0%
12300   1          0.0%    0.0%     0.0%
12299   3          0.0%    0.0%     0.0%
12299   2          0.0%    0.0%     0.0%
12299   1          0.0%    0.0%     0.0%
12298   4          0.0%    0.0%     0.0%
12298   3          0.0%    0.0%     0.0%
12298   1          0.0%    0.0%     0.0%
12297   1          0.0%    0.0%     0.0%
4104    5          0.0%    0.0%     0.0%
4104    4          0.0%    0.0%     0.0%
4104    2          0.0%    0.0%     0.0%
4104    1          0.0%    0.0%     0.0%
4103    6          0.0%    0.0%     0.0%
4103    5          0.0%    0.0%     0.0%
4103    4          0.0%    0.0%     0.0%
4103    3          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
4103    2          0.0%    0.0%     0.0%
4103    1          0.0%    0.0%     0.0%
4102    6          0.0%    0.0%     0.0%
4102    5          0.0%    0.0%     0.0%
4102    4          0.0%    0.0%     0.0%
4102    3          0.0%    0.0%     0.0%
4102    2          0.0%    0.0%     0.0%
4102    1          0.0%    0.0%     0.0%
5       3          0.0%    0.0%     0.0%
5       2          0.0%    0.0%     0.0%
4       2          0.0%    0.0%     0.0%
4       1          0.0%    0.0%     0.0%
3       1          0.0%    0.0%     0.0%
1       17         0.0%    0.0%     0.0%
1       16         0.0%    0.0%     0.0%
1       15         0.0%    0.0%     0.0%
1       14         0.0%    0.0%     0.0%
1       13         0.0%    0.0%     0.0%
1       12         0.0%    0.0%     0.0%
1       11         0.0%    0.0%     0.0%
16410   2          0.0%    0.0%     0.0%
1       9          0.0%    0.0%     0.0%
1       8          0.0%    0.0%     0.0%
PID     TID      5secUtil 1minUtil 5minUtil
1       7          0.0%    0.0%     0.0%
1       6          0.0%    0.0%     0.0%
1       5          0.0%    0.0%     0.0%
1       4          0.0%    0.0%     0.0%
1       3          0.0%    0.0%     0.0%
1       2          0.0%    0.0%     0.0%
1       1          0.0%    0.0%     0.0%
SolCen_VSS_Core#

Hi @all ,

thanks for your help but we found the issue. It was a bridging loop originating from a Cisco 3750 switch attached to the VSS pair. I dont know why happened this.

I focused just on the power cut.

regards

Matthias

Reza Sharifi
Hall of Fame
Hall of Fame

Matthias,

To isolate this issue further, can you try disconnecting one VSL link one at a time and watch the CPU?

First disconnect 1/5/4 and run "sh proc cpu sorted | ex 0.00" several times.  If the CPU is still high connect 1/5/4 and disconnect 1/5/5 and do the something.

HTH

Hi Reza,

i did it, but CPU is still high. :(

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: