Scenario: TACACS authorization server fails after initial AAA

[Josh Bailey]

Hi all,

Hopefully very simple, but I haven't found anything in the available commands nor in the documentation I've read so far.

Basically, my situation is this:

1. ClearPass 6.6+ server (TACACS/AAA)

2. Cisco 3650 LAN

3. SSH into 3650 - AAA goes smashingly. I issue commands, everything is wonderful.

4. Kill CPPM server

5. Authorization unavailable on Cisco, now I can run ALL commands

I will have a cluster of CPPM servers, so theoretically, this scenario shouldn't be likely. BUT, in the case it happened, how do I fail close? I'm familiar with 802.1x configuration where I can fail back to a default VLAN or ACL, which works great. I just can't seem to find anything similar in this case that would enable me to fall back to a local set of rules/commands.

Thanks for your time,

Josh

[Philip D'Ath]

What would normally happen is you wouldn't even be able to log into the switches, because TACAS+ was unavailable.  If you enable local authentication as the secondary authentication method (which you should do) for an emergency back door, then you absolutely want to be able to run all commands.  After all, it is now an emergency.

You only ended up in this situation because you logged in and then killed TACACS+.

[Josh Bailey]

You only ended up in this situation because you logged in and then killed TACACS+.

But, this is the entire point of my post. This scenario is not only possible, if I were a malicious insider, there are numerous ways I could force this scenario, thus giving me FULL authorization to run any commands I want.

So, given that, my question is: how do you mitigate this issue?

[Richard Burts]

Perhaps we should start by being a bit more specific about what your are configuring for aaa authorization. Are you configuring authorization exec or are you configuring authorization for command execution?

Maybe I am not understanding your post correctly. I assume that you have configured authorization for command execution. You seem to say that if you have logged in and authenticated correctly and then the server stops responding that you are able to execute any command. My experience is otherwise. I have had the experience of being able to login successfully but with authorization for command execution configured and the server not responding that I was not able to execute any privilege level commands.

So can you help me understand what you are asking?

HTH

Rick

[Josh Bailey]

This is the full list of commands currently in use on the test switch:

aaa authentication login default group tacacs+ local
aaa authentication enable default enable
aaa authentication dot1x default group radius
aaa authorization console
aaa authorization config-commands
aaa authorization exec default group tacacs+ local
aaa authorization commands 0 default group tacacs+ none
aaa authorization commands 1 default group tacacs+ if-authenticated
aaa authorization commands 15 default group tacacs+ if-authenticated
aaa authorization network default group radius
aaa accounting dot1x default start-stop group radius
aaa accounting commands 15 default start-stop group tacacs+
aaa accounting connection default start-stop group tacacs+

aaa server radius dynamic-author
 client 10.11.2.200 server-key aruba123
 port 3799
 auth-type all

tacacs server sdc-cppm-01
 address ipv4 10.13.5.95
 key Cisco123
 timeout 5

radius server CPPM
 address ipv4 10.13.5.95 auth-port 1645 acct-port 1646
 key aruba123

Sounds like it's possible the issue could lie in the aaa authorization command set. I've been researching various implementation and best practice documentation, but still have quite a lot of material to go. It'd be great if you could help me cut through that to settings I'm not aware of yet.

I'm more familiar with configuring failure behavior on the port or globally for 802.1x, as an example, so I was expecting it to be the same (or similar) for TACACS+.

Thanks for the response,

Josh

[Richard Burts · Accepted solution]

Josh

Thanks for the clarification. It is good to see that you are indeed configuring authorization for command execution.  The key command is this one

aaa authorization commands 15 default group tacacs+ if-authenticated

And this command does indeed produce the results that you discuss - if you have successfully authenticated and then the aaa server/server group stops responding then you would be able to execute any privilege level command. If you are concerned about that then you should configure it more like this

aaa authorization commands 15 default group tacacs+

HTH

Rick 

[Josh Bailey]

Perfect, thank you Richard. I'll give that a try now and report back.

[Josh Bailey]

OK, mystery solved. This works beautifully now after dropping the "if-authenticated" portion of the command.

Thanks a million, Richard.

Josh

[Josh Bailey]

Bah, spoke too soon!

Here's what happened:

test-switch-cppm#conf t
% Authorization failed.

test-switch-cppm#show run
% Authorization failed.

This was the initial behavior after the switch lost connectivity to the TACACS server. But then, about 60-90 seconds later or so, this happened:

test-switch-cppm#conf t
Tacacs session has expired.Please re-login to continue.

Enter configuration commands, one per line.  End with CNTL/Z.
test-switch-cppm(config)#

So, while it initially helped, all I had to do was wait for the session to timeout, then I was back in business. Is there anything I can do with the timeouts to prevent this? Not sure having NO timeout would be wise for resource issues, but I don't know in this case.

Thanks again,

Josh

[Richard Burts]

Josh

That is a surprise. And we are unexplored territory for me ( I have typically used the if-authenticated on the assumption that if I need local authentication to access the device when TACACS was not working that I wanted to be able to execute commands to recover the situation). The output that you posted shows the message about needing to re-login and then shows you to be in config mode. So I am not clear whether there was some login activity which is not shown? Or what happens between the message about needing to re-login and your continued access?

HTH

Rick

[Josh Bailey]

Richard,

Sorry for any confusion. What happens after the session timeout is the Tacacs message above, THEN config mode. There's nothing in-between, which means I'm back to the same issue: disrupt communication with the TACACs server + session timeout = full authorization.

The only timeouts I have configured are:

tacacs server sdc-cppm-01
 address ipv4 10.13.5.95
 key Cisco123
 timeout 10

This should be for initial tacacs talk, not the session.

Here are the line configurations:

line con 0
 exec-timeout 60 0
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 transport input ssh
line vty 5 15
 transport input ssh

My assumption is that it's going off the exec-timeout listed above at 60 seconds. But, as previously mentioned, while I could remove that and retest, even if it did work, I don't think I'd want to have NO timeout.

Thanks,

Josh

[Richard Burts]

Josh

You certainly do not want to remove the inactivity timeout. And I am convinced that trying that would not change the authorization behavior. And I am pretty sure that it is not involved with any of the vty timeouts. And the exec timeout shown above is 60 minutes and not 60 seconds.

Are you by any chance on the console when you are doing this? Or are you in an SSH session to the vty?

Perhaps you could test again and when your ability to execute  commands resumes would you do show tacacs and post the output? Also might help if you would post the current aaa config.

HTH

Rick

[ShrenikMota0491]

Hi Josh,

were you able to find a solution for Tacacs timeout? I am running into same issue. 

[downieadam]

I am wondering if anyone got to the bottom of this one?