cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1741
Views
5
Helpful
6
Replies

2960X Stack 100% CPU

jonhill
Level 1
Level 1

We've just started to implement changes on our switches so we can use ISE for authentication and on a switch stack of 6 switches I logged on did int range gig1/0/1 - 48 and copied the below config to the switch.

 

Switch 1 was fine and took approximately 10 minutes to complete, Switch 2 took approximately 15 minutes but when I got to Switch 3 this took in the region of 85 minutes to complete and by the time it had completed CPU was at 100% and didn't drop.

 

Below is the processes and also the last 60 minute CPU history, unfortunately because we started seeing connectivty issues on the switch stack it had to be rebooted which cleared the issue.

 

I'm now very reluctant to carry on rolling out the config changes across the rest of our 2960X stacks just in case the same thing happens again.

 

Since we've had the 2960X's they've been a bit on the slow side when doing int range commands but never had this issue before.

 

The stack is running 15.2(2)E6.

 

Any help or advice would be much appreciated?

 

Thanks

 

Jon

 

CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

178     2016913        8019     251516 57.85% 54.83% 53.11%   0 HRPC hrcli-event

255     2027659     2009154       1009 13.07% 19.28% 19.81%   0 Auth Manager

226     9270915    21965465        422  4.16%  1.87%  2.03%   0 HULC DAI Process

161  3496514514   945477148       3698  3.75%  3.20%  3.41%   0 Hulc LED Process

388   117720163   238809234        492  1.87%  0.44%  0.40%   0 LLDP Protocol

236     4540187    16489663        275  0.58%  0.51%  0.56%   0 IP Host Track Pr

131   289793132    37853037       7655  0.58%  0.43%  0.38%   0 hpm counter proc

233   106330513   383742152        277  0.23%  0.35%  0.34%   0 UDLD

120      238888     1029136        232  0.23%  0.28%  0.27%   0 HRPC hlfm reques

219   130242841   194137883        670  0.23%  0.15%  0.14%   0 Spanning Tree

397       16159       14165       1140  0.17%  0.14%  0.15%   0 SNMP Traps

  15   167833895    70748693       2372  0.17%  0.24%  0.54%   0 ARP Input

 

      111111111111111111111111111111111111111111111111111

      000000000000000000000000000000000000000000000000000999889999

      000000000000000000000000000000000000000000000000000992194944

  100 ###################################################**    *

   90 ####################################################** *#*

   80 ######################################################*###

   70 ##########################################################

   60 ##########################################################

   50 ##########################################################

   40 ##########################################################

   30 ##########################################################

   20 ##########################################################

   10 ##########################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

 

 

 

ip access-group PERMIT-ALL in
authentication open
authentication control-direction in
authentication event fail action next-method
authentication event server dead action authorize voice
authentication event server alive action reinitialize
authentication host-mode multi-domain
authentication order mab dot1x
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication timer inactivity server dynamic
authentication violation replace
mab
dot1x pae authenticator
dot1x timeout tx-period 10
cdp enable
lldp transmit
lldp receive

1 Accepted Solution

Accepted Solutions

Try the following (with your security teams blessing):

!
no aaa authorization commands 1 default group tacacs+ if-authenticated
no aaa authorization commands 15 default group tacacs+ local if-authenticated
!

 

...and see if it improves your situation.

 

cheers,

Seb.

View solution in original post

6 Replies 6

Seb Rupik
VIP Alumni
VIP Alumni

Hi there,

What do your existing aaa commands look llike?

 

I experienced similar behavior on some 4500 chassis a few years ago, range command would bring the switches to their knees. To get around the problem we temporarily removed the aaa authorization methods for the duration of the change window and we were then able to proceed at a good pace.

 

cheers,

Seb.

The existing AAA is below



aaa authentication login default group tacacs+ local

aaa authentication enable default group tacacs+ enable

aaa authentication dot1x default group ise-group

aaa authorization config-commands

aaa authorization exec default group tacacs+ local if-authenticated

aaa authorization commands 1 default group tacacs+ if-authenticated

aaa authorization commands 15 default group tacacs+ local if-authenticated

aaa authorization network default group ise-group

aaa authorization auth-proxy default group ise-group

aaa accounting update newinfo periodic 2880

aaa accounting auth-proxy default start-stop group ise-group

aaa accounting dot1x default start-stop group ise-group

aaa accounting exec default start-stop group tacacs+

aaa accounting commands 1 default start-stop group tacacs+

aaa accounting commands 15 default start-stop group tacacs+

aaa accounting network default start-stop group tacacs+

aaa accounting connection default start-stop group tacacs+

aaa accounting system default start-stop group ise-group


Try the following (with your security teams blessing):

!
no aaa authorization commands 1 default group tacacs+ if-authenticated
no aaa authorization commands 15 default group tacacs+ local if-authenticated
!

 

...and see if it improves your situation.

 

cheers,

Seb.

Seb

That worked a treat and the difference is massive.

I'm assuming the reason those two commands cause the issue with the switch is that every command for every interface is sent to TACACS for the OK for that user so it just gets bogged down particularly if the TACACS server doesn't respond in a timely manner?

Thanks

Jon

Glad to hear you're working at full rate again.

 

Your assumption is correct.

Like I said I have only ever seen it on the 4500 chassis switches previously, now I can add the 2960X to my list!

 

Cheers,

Seb.

Leo Laohoo
Hall of Fame
Hall of Fame
Raise a TAC Case.
It could be CSCvh17444.
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: