Solved: 2960S Stack, 15.0(2)SE3 & TACACS

lekeosi11 · ‎06-07-2013

Hi

I upgraded a Cisco 2960s stack from 15.0(2)SE2 to 15.0(2)SE3 yesterday.

The switch stack is set to use TACACS for authentication.

Since then, I'm no longer able loging to the switch using ssh or http.

I start a SSH session, enter my username and immediatley I get Access Denied (3 times and the switch drops the connection).

I can't see any tacacs packets being sent from the switch to the ACS server.

The release notes for 15.0(2)SE3 do not indicate any issues with Tacacs.

Any ideas?

Thanks

L

Christoph Faber · ‎06-07-2013

Problem is reproducible on several models.

As soon as I add

aaa authentication login default group tacacs+

or

aaa authorization exec default group tacacs+

and have a configured and reachable tacacs server, with the first login attempt the process TPLUS goes up to 100% CPU Load.

Login is not possible, only possibility to reach the server is rsh.

Tacacs-Log show no requests from the server.

If I downgrade to 15.0(2)SE2 all is ok - using the same configuration.

Tested on

WS-C2960G-48TC-L

WS-C3650-8PC-S

WS-C2960G-8TC-L

Update:

Forgot to mention:

Switch is still working properly, there are no other impacts

View solution in original post

pimentel1990 · ‎06-12-2013

I have another work around my coworkers and I used to get back in our 3750 we were testing this upgrade on using SNMP. If it's configured it's possible to do the following:

Prepare a file on a tftp server, call it what you want. Inside we used something like:

aaa authentication login default local

aaa authorization exec default local

username recover password this

end

Push the above configuration to the device by setting the necessary values and then activating. This command should do all that in one, just edit the IP address and th line below it to match the filename we created above on the tftp server.

snmpset -v 2c -c private Device .1.3.6.1.4.1.9.9.96.1.1.1.1.2.50 i 1 \
.1.3.6.1.4.1.9.9.96.1.1.1.1.3.50 i 4 \
.1.3.6.1.4.1.9.9.96.1.1.1.1.4.50 i 1 \
.1.3.6.1.4.1.9.9.96.1.1.1.1.5.50 a "10.0.0.2" \
.1.3.6.1.4.1.9.9.96.1.1.1.1.6.50 s "Router.cfg" \
.1.3.6.1.4.1.9.9.96.1.1.1.1.14.50 i 4 \

Check on the status using the following command, just make sure to change the host and community string:

snmpwalk -v 2c -c private 10.0.0.1 .1.3.6.1.4.1.9.9.96.1.1.1.1.10.50

Possible integer responses to this above command are waiting(1), running(2), successful(3), failed(4). If it returns 1 or 2, just keep trying until it reaches 3 or 4. If it times out, keep trying.

When done, destroy the row:

snmpset -v 2c -c private 10.0.0.1 .1.3.6.1.4.1.9.9.96.1.1.1.1.14.50 i 6

At this point we were able to log in and downgrade without leaving our seats. Of course, this does require SNMP with RW to be configured and it may time out occasionally due to the CPU utilization being high. Hope this helps some others.

View solution in original post

paul driver · ‎06-07-2013

Hello

Are you able to gain physical access to the switch? If so have you tried console access? If that fails I suggest perform a password recovery!

Res
Paul

Sent from Cisco Technical Support iPad App

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Christoph Faber · ‎06-07-2013

Problem is reproducible on several models.

As soon as I add

aaa authentication login default group tacacs+

or

aaa authorization exec default group tacacs+

and have a configured and reachable tacacs server, with the first login attempt the process TPLUS goes up to 100% CPU Load.

Login is not possible, only possibility to reach the server is rsh.

Tacacs-Log show no requests from the server.

If I downgrade to 15.0(2)SE2 all is ok - using the same configuration.

Tested on

WS-C2960G-48TC-L

WS-C3650-8PC-S

WS-C2960G-8TC-L

Update:

Forgot to mention:

Switch is still working properly, there are no other impacts

Leo Laohoo · ‎06-08-2013

Hi Christoph,

Thanks for finding out the offending lines.

I've marked your post as "Endorsed" as a reference to everyone affected.

By the way, this "SURPRISE" bug affects ALL switches that can load this IOS.

Leo Laohoo · ‎06-08-2013

Since then, I'm no longer able loging to the switch using ssh or http.

It's a "SURPRISE" bug with the 15.0(2)SE3. It will affect EVERY switch with have TACACS and runs on 15.0(2)SE3. Any switch rebooting to this IOS version will not able to manage and control the appliance using telnet, SSH or console.

There are two methods to take back control of your switch. And they are:

1. Easy Method:

See if you can find a spare USB thumb drive (not a portable HDD), size can be between 128 mb to 16 Gb);
Format with FAT16;
Copy lower IOS (BIN file extension) into USB stick;
Insert to your 2960S;
Boot into ROMmon (hold down the "Mode" button);
Enter command "flash_init";
In ROMmon, enter the command "boot usbflash0:OLD_IOS.BIN"
Once you've bootup, downgrade your IOS.

2. Slightly Easy Method

Boot into ROMmon;
Enter command "flash_init";
Rename "config.text" into something else, like BLAH.text
Enter command "boot";
When switch has bootup with no config, in enable mode, enter this: copy flash:BLAH.text run
DOWNGRADE the IOS.

Hope this helps.

Please don't forget to rate our useful posts.

WARNING: AVOID IOS version 15.0(2)SE3 at all cost!

Message was edited by: Leo Laohoo ** Removed any posts that relates RADIUS to this bug. RADIUS is not causing the issue.

InayathUlla Sharieff · ‎06-10-2013

Leo PM sent.

Guys,

Check the following bug:

CSCsk28117

Leo Laohoo · ‎06-10-2013

Thank you, Inayath! :)

Sent from Cisco Technical Support Nintendo App

Daniel McDavid · ‎06-10-2013

Have you got a bug ID on this? I did not have the same symptons as the original post. My session just hung after the MOTD.

Leo Laohoo · ‎06-10-2013

Have you got a bug ID on this?

Daniel,

Have a look at Bug ID CSCsk28117. It's similar.

 I did not have the same symptons as the original post. My session just hung after the MOTD.

Check my post above, it'll tell you two methods you can try to get back control of your switch.

Zachary McGibbon · ‎06-11-2013

I've had the same issue on a 3750E-12D and a 3750X-48P.. Downgrading back to 15.0(2)SE2

InayathUlla Sharieff · ‎06-11-2013

Zachary,

Thats good idea as of now.

Regards

Inayath

Leo Laohoo · ‎06-11-2013

Zachary,

This "bug" affects all switches that can load 15.0(2)SE3 and has TACACs authentication. I've created a Cisco TAC Case and will be posting the Bug ID when made available.

michelpe · ‎06-11-2013

This is bug CSCuh43252 which is a duplicate of an internal bug CSCug62154

We are looking into what time frames we can address this and how.

A workaround is to enable single-connection tacacs with single-connection

eg,

tacacs-server host single-connection

or

tacas server

single-connection

You would have to have access to the switch to do so. 15.0(2)SE2 and before are not affected

InayathUlla Sharieff · ‎06-12-2013

Furthermore to add on this, this issue is not seen when using Radius

The bug CSCuh43252 is internal to cisco and not publically viewable.

Workarounds:-

1-One thing to note that the problem only occurs when not using single-connection

Switch(config-server-tacacs)#no single-connection Switch

(config-server-tacacs)#exit

2- When a switch, configured for TACACs authentication, boots up 15.0(2)SE3 no one can gain access to the switch's management IP address via Telnet, SSH or console. Issue is not evident when using RADIUS.

Data traffic STOPS because the CPU jumps to 100%.

The only two methods to regain control of the switch are the following:

1. USB Method

* See if you can find a spare USB thumb drive (not a portable HDD), size can be between 128 mb to 16 Gb);

* Format with FAT16;

* Copy lower IOS (BIN file extension) into USB stick;

* Insert the USB flash drive into your switch;

* Boot into ROMmon (hold down the "Mode" button);

* Enter command "flash_init";

* In ROMmon, enter the command "boot usbflash0:OLD_IOS.BIN";

* Once you've bootup, downgrade your IOS.

2. Mode button Method

* Boot into ROMmon;

* Enter command "flash_init";

* Rename "config.text" into something else, like BLAH.text

* Enter command "boot";

* When switch has bootup with no config, in enable mode, enter this: copy flash:BLAH.text run;

* DOWNGRADE the IOS.

HTH

Regards

Inayath

Leo Laohoo · ‎06-12-2013

Inayath,

Why is 15.0(2)SE3 still available for download?