Cisco IOS 15.2(7)E3 TACACs and SSH Breakage?

[Reuben Farrelly]

I've just upgraded a few 3560CX switches from 15.2(7)E2 to 15.2(7)E3 as part of routine maintenance work today and it looks to have completely broken TACACs.  See the debug below:

Sep 19 09:38:04.265: TPLUS: Queuing AAA Authentication request 433 for processing
Sep 19 09:38:04.268: TPLUS(000001B1) login timer started 1020 sec timeout
Sep 19 09:38:04.268: TPLUS: processing authentication start request id 433
Sep 19 09:38:04.268: TPLUS: Authentication start packet created for 433()
Sep 19 09:38:04.268: TPLUS: Using server 192.168.10.20
Sep 19 09:38:04.268: TPLUS(000001B1)/0/NB_WAIT/DEC67B8: Started 5 sec timeout
Sep 19 09:38:04.272: TPLUS(000001B1)/0/NB_WAIT: socket event 2
Sep 19 09:38:04.272: TAC+: Invalid key
Sep 19 09:38:04.272: TPLUS(000001B1): encryption failed for AAA request
Sep 19 09:38:04.272: TPLUS(000001B1) login timer stopped
Sep 19 09:38:04.272: TPLUS(000001B1)/0/DEC67B8: Processing the reply packet
Sep 19 09:38:04.272: TPLUS: Invalid Client information received as input

This exact same config worked fine on 15.2(7)E2 and still right now works on IOS XE 16.12 and 17.3.  The TACACs key is not invalid and I can see it in plaintext form in the config of one of the switches.  It matches the TACACs key on the server side (tac_plus FWIW).

aaa group server tacacs+ R-TACACS+
  server-private 192.168.10.20 key XXX

Workaround: configure static usernames/passwords.

Has anyone else run into this?  I can't find a bug reference for it in Bug Toolkit but it may just be that one isn't public yet.  Nothing is mentioned in the release notes in regards to fixes/bugs to do with TACACs.

Also worth noting that telnet access appears to be disabled in this release by default, so if you don't have "transport input telnet ssh" configured in a line then you'll lose telnet access after upgrading.  While this is a good security measure, it is really unacceptable that this is not documented in the release notes.

[balaji.bandi]

as per the below messages invalid key, so remove the Radius/tacacs  config, do not paste it from old config, re-type the config and key and let us know how it goes.

Sep 19 09:38:04.272: TAC+: Invalid key
Sep 19 09:38:04.272: TPLUS(000001B1): encryption failed for AAA request

[Reuben Farrelly]

The tacacs key is letter for letter identical.  I've already validated that and copied and pasted it back in in plaintext and cross checked with the devices on other versions of code which are still working.

This worked fine until the upgrade was performed (no config changes at the time), at which point it all broke   So I'm 100% convinced the key itself is not invalid and is not mismatched.

[Richard Burts]

If TACACS worked correctly in an earlier version of code, and if there have been no config changes, and if it does work in the new version of code, and if removing the config for TACACS and manually entering the config and it still does not work then it sure does sound like a bug in the new version of code. Do you have a support contract that would enable you to open a case with Cisco TAC?

[AustinHarsh2908]

I've ran into this same issue. If you configure and enable aes password encryption and type your clear text keys in again the issue goes away. It appears it's something about reading in type 7 passwords that there's an issue with, but only with AAA (NTP type 7 keys work just fine).

Did you ever open a TAC case? If so, did they give you a bug ID by chance?

[tperrier]

Austin, I couldn't make your workaround work. I've tried with and without AES password encryption, retyping the password each time, but there's been no positive effect.

I'll open a TAC case tomorrow.

BTW, if you don't use per-VRF TACACS+, you can still make it work using the default TACACS+ group (commands below). That's what I ended up doing for access switchs (2960-X) which don't use any VRF. But I can't core on the core switch (4507 running 3.11.3aE, which has the same 15.2(7)E3 IOSd).

aaa authentication login default group tacacs+ local-case

tacacs server {name}
  address ipv4  {address}
  key {key}

[Kiley Arena]

RE:  I've ran into this same issue. If you configure and enable aes password encryption and type your clear text keys in again the issue goes away. It appears it's something about reading in type 7 passwords that there's an issue with, but only with AAA (NTP type 7 keys work just fine).

I've got the same issue with one of our devices that was working flawlessly on tacacs+ until the IOS was upgraded.  I just configured aes encryption, retyped my cleartext password and it still does not work. While it no longer tells me type 7 passwords are deprecated and to use stronger encryption, when I do a sh run and look at my aaa group server tacacs+ xxxx, it still shows the type 7 passwords under the server-private x.x.x.x key 7 fakekey123.  The only actual passwords that were changed under the aes encryption are the enable secret and local username secret pwds.

[KevinDebelak2369]

This worked for me! I have an older IOS 15 Catalyst switch that hasn't been upgraded in some time.

I had to add a new TACACS server and it was not working until I ran "password encryption AES" and then ran the command to add the server/key again. 

[tperrier]

I found this bug (not my case): https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvv86851

Also it's referenced as an open caveat in the 2960-X, 2960-CX and 4500 (and probably some other switches) release notes.

[tperrier]

I opened a case. It turns out there's a simple workaround for per-VRF TACACS+:

tacacs server xxxxxxxxxxxx
  address ipv4 a.b.c.d
  key yyyyyyyyyyyyyy
!
aaa group server tacacs+ TAC+
  server name xxxxxxxxxxxx

  ip vrf forwarding zzzzzzz

So basically, declare the TACACS+ server with "tacacs server", then refer to its name in the "aaa group server tacacs+" with "server name", instead of using only the "server-private" command with the IP address.

[Richard Burts]

Thanks for sharing the output of your case with TAC and its solution. I well deserved +5 for this.

[ShaunGreen]

Hi Guys,

Thanks for the useful information.

We had the same issue, although we didn't have the vrf concern.

I followed the useful tips above, but was still having an issue.

tacacs server xxxxxxxxxxxx
  address ipv4 a.b.c.d
  key

When I entered the key in clear text things were still not working. 

I copied and pasted the encrypted key from a device that hadn't been upgraded and BOOM!, everything back on-line.

Thanks again!!

[Eric Hansen]

For anyone stumbling across this thread:  Its june 2021, I have about 30 3560CX switches running 15.2(7)E2, all are working fine talking TACACS to ISE2.7.  Upgraded two of these to E3 and we hit this problem.  You can't add ciphers to ISE manually, even tried setting the ssh ciphers to use on the CX switch itself 'ip ssh client alg encryp aes256-ctr aes256-cbc aes128-ctr aes128-cbc' and set the DH group to 14 'ip ssh dh min size 2048' and still no luck.  For sure aes256 is supported by ISE, its in the official supported ciphers docs for ISE.

Tried many things to no avail, This is a bug in the E3 version.... which is concerning since its the suggested release for the CX 3560 platform.  If you run into this open a case with TAC.

Debug:
11w3d: TPLUS(0000001F)/0/NB_WAIT: socket event 2
11w3d: TAC+: Invalid key
11w3d: TPLUS(0000001F): encryption failed for AAA request

[yensheng_lin]

We are having the same issue.

when issue debug tacacs, it shows invalid key.

I have checked the key and also tried to use plaint text instead of MD5 encrypted key.

But it still doesn't work for our switch.

Below is the model and the IOS

WS-C2960X-48FPS-L 15.2(7)E4 C2960X-UNIVERSALK9-M

I'll raise a tac and see how it goes.