12-12-2018 06:57 AM - edited 03-08-2019 04:48 PM
We've just started to implement changes on our switches so we can use ISE for authentication and on a switch stack of 6 switches I logged on did int range gig1/0/1 - 48 and copied the below config to the switch.
Switch 1 was fine and took approximately 10 minutes to complete, Switch 2 took approximately 15 minutes but when I got to Switch 3 this took in the region of 85 minutes to complete and by the time it had completed CPU was at 100% and didn't drop.
Below is the processes and also the last 60 minute CPU history, unfortunately because we started seeing connectivty issues on the switch stack it had to be rebooted which cleared the issue.
I'm now very reluctant to carry on rolling out the config changes across the rest of our 2960X stacks just in case the same thing happens again.
Since we've had the 2960X's they've been a bit on the slow side when doing int range commands but never had this issue before.
The stack is running 15.2(2)E6.
Any help or advice would be much appreciated?
Thanks
Jon
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
178 2016913 8019 251516 57.85% 54.83% 53.11% 0 HRPC hrcli-event
255 2027659 2009154 1009 13.07% 19.28% 19.81% 0 Auth Manager
226 9270915 21965465 422 4.16% 1.87% 2.03% 0 HULC DAI Process
161 3496514514 945477148 3698 3.75% 3.20% 3.41% 0 Hulc LED Process
388 117720163 238809234 492 1.87% 0.44% 0.40% 0 LLDP Protocol
236 4540187 16489663 275 0.58% 0.51% 0.56% 0 IP Host Track Pr
131 289793132 37853037 7655 0.58% 0.43% 0.38% 0 hpm counter proc
233 106330513 383742152 277 0.23% 0.35% 0.34% 0 UDLD
120 238888 1029136 232 0.23% 0.28% 0.27% 0 HRPC hlfm reques
219 130242841 194137883 670 0.23% 0.15% 0.14% 0 Spanning Tree
397 16159 14165 1140 0.17% 0.14% 0.15% 0 SNMP Traps
15 167833895 70748693 2372 0.17% 0.24% 0.54% 0 ARP Input
111111111111111111111111111111111111111111111111111
000000000000000000000000000000000000000000000000000999889999
000000000000000000000000000000000000000000000000000992194944
100 ###################################################** *
90 ####################################################** *#*
80 ######################################################*###
70 ##########################################################
60 ##########################################################
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
ip access-group PERMIT-ALL in
authentication open
authentication control-direction in
authentication event fail action next-method
authentication event server dead action authorize voice
authentication event server alive action reinitialize
authentication host-mode multi-domain
authentication order mab dot1x
authentication priority dot1x mab
authentication port-control auto
authentication periodic
authentication timer reauthenticate server
authentication timer inactivity server dynamic
authentication violation replace
mab
dot1x pae authenticator
dot1x timeout tx-period 10
cdp enable
lldp transmit
lldp receive
Solved! Go to Solution.
12-12-2018 08:16 AM
Try the following (with your security teams blessing):
!
no aaa authorization commands 1 default group tacacs+ if-authenticated
no aaa authorization commands 15 default group tacacs+ local if-authenticated
!
...and see if it improves your situation.
cheers,
Seb.
12-12-2018 07:04 AM
Hi there,
What do your existing aaa commands look llike?
I experienced similar behavior on some 4500 chassis a few years ago, range command would bring the switches to their knees. To get around the problem we temporarily removed the aaa authorization methods for the duration of the change window and we were then able to proceed at a good pace.
cheers,
Seb.
12-12-2018 07:41 AM
12-12-2018 08:16 AM
Try the following (with your security teams blessing):
!
no aaa authorization commands 1 default group tacacs+ if-authenticated
no aaa authorization commands 15 default group tacacs+ local if-authenticated
!
...and see if it improves your situation.
cheers,
Seb.
12-13-2018 12:06 AM
12-13-2018 12:15 AM
Glad to hear you're working at full rate again.
Your assumption is correct.
Like I said I have only ever seen it on the 4500 chassis switches previously, now I can add the 2960X to my list!
Cheers,
Seb.
12-12-2018 10:19 PM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide