12-09-2016 06:12 AM - edited 03-08-2019 08:30 AM
hi@alll
we have two 6509 switches configured with VSS. Few days ago we had a power cut and both switches was down.
After both switches coming up again but with issues.
installed IOS image is: s72033_rp_ADIPSERVICESK9_WAN_VM 12.2(33)SXI2a
I could find the issue on interface port-channel 110 for VSS Link.
When i disable this interface, the cpu comes down.
Problem is, both switch goes to active and not active/standby.
When i enable this interface again then VSS status come up and i have one active and one standby switch, but CPU is then very high.
see below the show command:
SolCen_VSS_Core#show switch virtual role
Switch Switch Status Preempt Priority Role Session ID
Number Oper(Conf) Oper(Conf) Local Remote
------------------------------------------------------------------
LOCAL 1 UP FALSE(N ) 100(100) ACTIVE 0 0
REMOTE 2 UP FALSE(N ) 100(100) STANDBY 4791 2587
In dual-active recovery mode: No
SolCen_VSS_Core#show switch virtual
Switch mode : Virtual Switch
Virtual switch domain number : 100
Local switch number : 1
Local switch operational role: Virtual Switch Active
Peer switch number : 2
Peer switch operational role : Virtual Switch Standby
SolCen_VSS_Core#show proces
SolCen_VSS_Core#show processes cpu sort
SolCen_VSS_Core#show processes cpu sorted
CPU utilization for five seconds: 100%/81%; one minute: 97%; five minutes: 64%
PID TID 5secUtil 1minUtil 5minUtil
16407 6 79.2% 75.3% 41.8%
16407 13 4.4% 2.5% 1.4%
16407 19 4.3% 2.2% 2.2%
16432 5 3.7% 1.5% 0.6%
16407 18 3.4% 2.8% 1.7%
16407 7 1.7% 3.1% 3.9%
16407 3 0.7% 1.8% 1.0%
16407 8 0.5% 0.5% 0.2%
16407 16 0.4% 2.5% 2.2%
And here the VSS configuration:
switch virtual domain 100
switch mode virtual
redundancy
main-cpu
auto-sync running-config
mode sso
interface Port-channel110
no switchport
no ip address
switch virtual link 1
mls qos trust cos
no mls qos channel-consistency
!
interface Port-channel120
no switchport
no ip address
switch virtual link 2
mls qos trust cos
no mls qos channel-consistency
interface TenGigabitEthernet1/5/4
no switchport
no ip address
mls qos trust cos
channel-group 110 mode on
!
interface TenGigabitEthernet1/5/5
no switchport
no ip address
mls qos trust cos
channel-group 110 mode on
interface TenGigabitEthernet2/5/4
no switchport
no ip address
mls qos trust cos
channel-group 120 mode on
!
interface TenGigabitEthernet2/5/5
no switchport
no ip address
mls qos trust cos
channel-group 120 mode on
!
interface GigabitEthernet1/9/47
no switchport
no ip address
dual-active fast-hello
interface GigabitEthernet2/9/47
no switchport
no ip address
dual-active fast-hello
We changed the X2-10GE-CX4 already but nothing happened. Same Issue.
Anybody a idea??
Thanks
regards
Matthias
12-09-2016 06:36 AM
Hi Matthias,
Do you think can share the following command:
show proc cpu sorted | ex 0.00
With this command we can confirm which process is generating the high cpu if any. Because this output: "CPU utilization for five seconds: 100%/81%" tells that the issue is caused due to punted traffic to the CPU.
This means that 100% of cpu utilization and 81% is due to traffic punted to the cpu for process switch instead of hardware switch.
Post the output of the requested command and will see.
12-09-2016 11:38 AM
Hi Jose,
thanks for your reply.
See below output of the command:
SolCen_VSS_Core#show processes cpu sorted | ex 0.00
CPU utilization for five seconds: 100%/81%; one minute: 100%; five minutes: 100%
PID TID 5secUtil 1minUtil 5minUtil
16407 6 78.9% 78.6% 78.5%
16407 10 4.9% 2.7% 2.4%
16407 18 3.9% 2.5% 1.1%
16432 3 3.3% 2.7% 2.2%
16407 19 3.1% 2.8% 2.3%
16407 7 1.9% 2.6% 2.6%
16407 3 0.9% 1.8% 1.6%
16407 16 0.9% 1.4% 0.7%
16407 8 0.5% 0.5% 0.5%
16432 5 0.5% 0.5% 0.9%
16433 1 0.3% 0.3% 0.3%
1 10 0.1% 0.3% 0.3%
16407 1 0.1% 0.4% 1.4%
16432 4 0.0% 0.0% 0.0%
16432 8 0.0% 0.5% 0.5%
16432 2 0.0% 0.0% 0.0%
16431 10 0.0% 0.0% 0.0%
16432 1 0.0% 0.0% 0.0%
16432 6 0.0% 0.0% 0.0%
16431 7 0.0% 0.0% 0.0%
16431 6 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
16431 3 0.0% 0.0% 0.0%
16431 1 0.0% 0.0% 0.0%
16431 8 0.0% 0.0% 0.0%
16432 7 0.0% 0.0% 0.0%
16431 9 0.0% 0.0% 0.0%
16430 12 0.0% 0.0% 0.0%
16430 8 0.0% 0.0% 0.0%
16430 7 0.0% 0.0% 0.0%
16430 6 0.0% 0.0% 0.0%
16430 5 0.0% 0.0% 0.0%
16430 4 0.0% 0.0% 0.0%
16430 3 0.0% 0.0% 0.0%
16430 2 0.0% 0.0% 0.0%
16429 6 0.0% 0.0% 0.0%
16429 5 0.0% 0.0% 0.0%
16429 4 0.0% 0.0% 0.0%
16430 11 0.0% 0.0% 0.0%
16429 2 0.0% 0.0% 0.0%
16430 10 0.0% 0.0% 0.0%
16428 7 0.0% 0.0% 0.0%
16428 6 0.0% 0.0% 0.0%
16430 9 0.0% 0.0% 0.0%
16428 4 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
16429 3 0.0% 0.0% 0.0%
16428 2 0.0% 0.0% 0.0%
16428 1 0.0% 0.0% 0.0%
16427 4 0.0% 0.0% 0.0%
16427 3 0.0% 0.0% 0.0%
16427 2 0.0% 0.0% 0.0%
16427 1 0.0% 0.0% 0.0%
16426 5 0.0% 0.0% 0.0%
16426 4 0.0% 0.0% 0.0%
16426 3 0.0% 0.0% 0.0%
16426 2 0.0% 0.0% 0.0%
16425 9 0.0% 0.0% 0.0%
16425 8 0.0% 0.0% 0.0%
16425 7 0.0% 0.0% 0.0%
16425 6 0.0% 0.0% 0.0%
16425 5 0.0% 0.0% 0.0%
16425 3 0.0% 0.0% 0.0%
16425 2 0.0% 0.0% 0.0%
16424 6 0.0% 0.0% 0.0%
16424 5 0.0% 0.0% 0.0%
16424 4 0.0% 0.0% 0.0%
16424 3 0.0% 0.0% 0.0%
16429 1 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
16424 1 0.0% 0.0% 0.0%
16423 12 0.0% 0.0% 0.0%
16428 5 0.0% 0.0% 0.0%
16423 6 0.0% 0.0% 0.0%
16423 5 0.0% 0.0% 0.0%
16423 4 0.0% 0.0% 0.0%
16423 3 0.0% 0.0% 0.0%
16423 2 0.0% 0.0% 0.0%
16422 4 0.0% 0.0% 0.0%
16428 3 0.0% 0.0% 0.0%
16422 2 0.0% 0.0% 0.0%
16433 2 0.0% 0.0% 0.0%
16433 3 0.0% 0.0% 0.0%
16424 2 0.0% 0.0% 0.0%
16421 2 0.0% 0.0% 0.0%
16421 1 0.0% 0.0% 0.0%
16420 4 0.0% 0.0% 0.0%
16420 3 0.0% 0.0% 0.0%
16420 2 0.0% 0.0% 0.0%
16420 1 0.0% 0.0% 0.0%
16419 4 0.0% 0.0% 0.0%
16419 3 0.0% 0.0% 0.0%
16419 2 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
16419 1 0.0% 0.0% 0.0%
16418 4 0.0% 0.0% 0.0%
16418 3 0.0% 0.0% 0.0%
16418 2 0.0% 0.0% 0.0%
16418 1 0.0% 0.0% 0.0%
16417 4 0.0% 0.0% 0.0%
16417 3 0.0% 0.0% 0.0%
16417 2 0.0% 0.0% 0.0%
16417 1 0.0% 0.0% 0.0%
16416 5 0.0% 0.0% 0.0%
16416 4 0.0% 0.0% 0.0%
16416 3 0.0% 0.0% 0.0%
16416 2 0.0% 0.0% 0.0%
16416 1 0.0% 0.0% 0.0%
16415 6 0.0% 0.0% 0.0%
16415 4 0.0% 0.0% 0.0%
16415 3 0.0% 0.0% 0.0%
16415 2 0.0% 0.0% 0.0%
16415 1 0.0% 0.0% 0.0%
16414 4 0.0% 0.0% 0.0%
16414 3 0.0% 0.0% 0.0%
16414 2 0.0% 0.0% 0.0%
16414 1 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
16413 6 0.0% 0.0% 0.0%
16413 5 0.0% 0.0% 0.0%
16413 4 0.0% 0.0% 0.0%
16413 3 0.0% 0.0% 0.0%
16413 2 0.0% 0.0% 0.0%
16413 1 0.0% 0.0% 0.0%
16412 4 0.0% 0.0% 0.0%
16412 3 0.0% 0.0% 0.0%
16412 2 0.0% 0.0% 0.0%
16412 1 0.0% 0.0% 0.0%
16411 4 0.0% 0.0% 0.0%
16423 7 0.0% 0.0% 0.0%
16411 2 0.0% 0.0% 0.0%
16411 1 0.0% 0.0% 0.0%
16410 4 0.0% 0.0% 0.0%
16410 3 0.0% 0.0% 0.0%
16421 3 0.0% 0.0% 0.0%
16410 1 0.0% 0.0% 0.0%
16409 4 0.0% 0.0% 0.0%
16409 3 0.0% 0.0% 0.0%
16409 2 0.0% 0.0% 0.0%
16409 1 0.0% 0.0% 0.0%
16408 5 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
16408 4 0.0% 0.0% 0.0%
16408 3 0.0% 0.0% 0.0%
16408 2 0.0% 0.0% 0.0%
16408 1 0.0% 0.0% 0.0%
16433 4 0.0% 0.0% 0.0%
16433 5 0.0% 0.0% 0.0%
16407 17 0.0% 0.0% 0.0%
16433 6 0.0% 0.0% 0.0%
16407 15 0.0% 0.6% 1.5%
16407 14 0.0% 0.0% 0.0%
16407 13 0.0% 0.9% 1.7%
16407 12 0.0% 0.0% 0.0%
16407 11 0.0% 0.0% 0.0%
16433 7 0.0% 0.0% 0.0%
16407 9 0.0% 0.0% 0.0%
16433 8 0.0% 0.0% 0.0%
16422 3 0.0% 0.0% 0.0%
16422 1 0.0% 0.0% 0.0%
16407 5 0.0% 0.0% 0.0%
16407 4 0.0% 0.0% 0.4%
16421 4 0.0% 0.0% 0.0%
16407 2 0.0% 0.0% 0.0%
16411 3 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
16406 6 0.0% 0.0% 0.0%
16406 5 0.0% 0.0% 0.0%
16406 4 0.0% 0.0% 0.0%
16406 3 0.0% 0.0% 0.0%
16405 5 0.0% 0.0% 0.0%
16405 3 0.0% 0.0% 0.0%
16405 2 0.0% 0.0% 0.0%
16405 1 0.0% 0.0% 0.0%
16404 3 0.0% 0.0% 0.0%
16404 2 0.0% 0.0% 0.0%
16404 1 0.0% 0.0% 0.0%
16403 1 0.0% 0.0% 0.0%
16402 2 0.0% 0.0% 0.0%
16402 1 0.0% 0.0% 0.0%
16386 2 0.0% 0.0% 0.0%
16386 1 0.0% 0.0% 0.0%
12305 18 0.0% 0.0% 0.0%
12305 17 0.0% 0.0% 0.0%
12305 16 0.0% 0.0% 0.0%
12305 15 0.0% 0.0% 0.0%
12305 14 0.0% 0.0% 0.0%
12305 13 0.0% 0.0% 0.0%
12305 12 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
12305 11 0.0% 0.0% 0.0%
12305 10 0.0% 0.0% 0.0%
12305 9 0.0% 0.0% 0.0%
12305 8 0.0% 0.0% 0.0%
12305 7 0.0% 0.0% 0.0%
12305 6 0.0% 0.0% 0.0%
12305 5 0.0% 0.0% 0.0%
12305 4 0.0% 0.0% 0.0%
12305 3 0.0% 0.0% 0.0%
12305 2 0.0% 0.0% 0.0%
12305 1 0.0% 0.0% 0.0%
12304 8 0.0% 0.0% 0.0%
12304 7 0.0% 0.0% 0.0%
12304 6 0.0% 0.0% 0.0%
12304 5 0.0% 0.0% 0.0%
12304 4 0.0% 0.0% 0.0%
12304 3 0.0% 0.0% 0.0%
12304 2 0.0% 0.0% 0.0%
12304 1 0.0% 0.0% 0.0%
12303 5 0.0% 0.0% 0.0%
12303 3 0.0% 0.0% 0.0%
12303 2 0.0% 0.0% 0.0%
12303 1 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
12302 4 0.0% 0.0% 0.0%
12302 3 0.0% 0.0% 0.0%
12302 2 0.0% 0.0% 0.0%
12302 1 0.0% 0.0% 0.0%
12301 1 0.0% 0.0% 0.0%
12300 5 0.0% 0.0% 0.0%
12300 2 0.0% 0.0% 0.0%
12300 1 0.0% 0.0% 0.0%
12299 3 0.0% 0.0% 0.0%
12299 2 0.0% 0.0% 0.0%
12299 1 0.0% 0.0% 0.0%
12298 4 0.0% 0.0% 0.0%
12298 3 0.0% 0.0% 0.0%
12298 1 0.0% 0.0% 0.0%
12297 1 0.0% 0.0% 0.0%
4104 5 0.0% 0.0% 0.0%
4104 4 0.0% 0.0% 0.0%
4104 2 0.0% 0.0% 0.0%
4104 1 0.0% 0.0% 0.0%
4103 6 0.0% 0.0% 0.0%
4103 5 0.0% 0.0% 0.0%
4103 4 0.0% 0.0% 0.0%
4103 3 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
4103 2 0.0% 0.0% 0.0%
4103 1 0.0% 0.0% 0.0%
4102 6 0.0% 0.0% 0.0%
4102 5 0.0% 0.0% 0.0%
4102 4 0.0% 0.0% 0.0%
4102 3 0.0% 0.0% 0.0%
4102 2 0.0% 0.0% 0.0%
4102 1 0.0% 0.0% 0.0%
5 3 0.0% 0.0% 0.0%
5 2 0.0% 0.0% 0.0%
4 2 0.0% 0.0% 0.0%
4 1 0.0% 0.0% 0.0%
3 1 0.0% 0.0% 0.0%
1 17 0.0% 0.0% 0.0%
1 16 0.0% 0.0% 0.0%
1 15 0.0% 0.0% 0.0%
1 14 0.0% 0.0% 0.0%
1 13 0.0% 0.0% 0.0%
1 12 0.0% 0.0% 0.0%
1 11 0.0% 0.0% 0.0%
16410 2 0.0% 0.0% 0.0%
1 9 0.0% 0.0% 0.0%
1 8 0.0% 0.0% 0.0%
PID TID 5secUtil 1minUtil 5minUtil
1 7 0.0% 0.0% 0.0%
1 6 0.0% 0.0% 0.0%
1 5 0.0% 0.0% 0.0%
1 4 0.0% 0.0% 0.0%
1 3 0.0% 0.0% 0.0%
1 2 0.0% 0.0% 0.0%
1 1 0.0% 0.0% 0.0%
SolCen_VSS_Core#
12-12-2016 02:15 AM
Hi @all ,
thanks for your help but we found the issue. It was a bridging loop originating from a Cisco 3750 switch attached to the VSS pair. I dont know why happened this.
I focused just on the power cut.
regards
Matthias
12-09-2016 08:00 AM
Matthias,
To isolate this issue further, can you try disconnecting one VSL link one at a time and watch the CPU?
First disconnect 1/5/4 and run "sh proc cpu sorted | ex 0.00" several times. If the CPU is still high connect 1/5/4 and disconnect 1/5/5 and do the something.
HTH
12-09-2016 11:47 AM
Hi Reza,
i did it, but CPU is still high. :(
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide