Switch/Router still queries tacacs after local login

[jamsgrove]

Hi Everyone!

I have scoured through the internet and have not been able to find an answer to my exact issue so hoping somone else might have an idea.
I am going through adding tacacs to a range of cisco routers and switches and the issue im having is if the tacacs server is down/unreachable i login using local username and password and then the enable password but any command i type is delayed about 5 seconds as the device tries to check with the tacacs server. This could be improved by reducing the tacacs timeout to say 1 second, unfortunately this may not be a good option as some of these devices are accessed over satellite links.

I will first start by explaining my requirements of the device management then list the configuration i have used:

1. Remote ssh should use tacacs first, if tacacs server is unreachable/down then use local credentials.
2. Console access to only ever use local credentials with enable and never tacacs.
3. Requirement for authorization commands as there are users on the tacacs server with different privilege levels (some that still will need to use enable).
4. Command accounting feature

Please see below for the configuration, this is the same for multiple devices:

WS-C3850-48T switch-stack version 03.03.05SE
WS-C2960CG-8TC-L version 12.2(55r)EX11, RELEASE SOFTWARE (fc1)
CISCO2911/K9 version 15.0(1r)M16, RELEASE SOFTWARE (fc1)

Configuration
----------
username localuser secret 5 &dfknCHANGEDsdfknsdf
enable secret 5 &dfknCHANGEDsdfknsdf
aaa authentication login AUTHENT-CONSOLE local
aaa authentication login AUTHENT-TACACS group TACACS-SERVERS local
aaa authentication username-prompt "Local username: "
aaa authentication password-prompt "Local password: "
aaa authorization exec AUTHOR-CONSOLE local
aaa authorization exec AUTHOR-TACACS group TACACS-SERVERS local
aaa authorization commands 0 COMMANDS-TACACS group TACACS-SERVERS local
aaa authorization commands 1 COMMANDS-TACACS group TACACS-SERVERS local
aaa authorization commands 15 COMMANDS-TACACS group TACACS-SERVERS local
aaa authorization console
aaa accounting exec default start-stop group TACACS-SERVERS
aaa accounting connection default start-stop group TACACS-SERVERS
aaa accounting commands 0 default start-stop group TACACS-SERVERS
aaa accounting commands 1 default start-stop group TACACS-SERVERS
aaa accounting commands 15 default start-stop group TACACS-SERVERS
aaa new-model
aaa session-id common

tacacs server TACACS-SERVER-1
address ipv4 192.168.1.100
key 7 sadfasdfasCHANGEDdfasdfasdf
tacacs-server directed-request

aaa group server tacacs+ TACACS-SERVERS
server name TACACS-SERVER-1

line con 0
exec-timeout 5 0
authorization exec AUTHOR-CONSOLE
logging synchronous
login authentication AUTHENT-CONSOLE
stopbits 1

line vty 0 15
access-class VTY_MGMT in
exec-timeout 15 0
authorization commands 0 COMMANDS-TACACS
authorization commands 1 COMMANDS-TACACS
authorization commands 15 COMMANDS-TACACS
authorization exec AUTHOR-TACACS
logging synchronous
login authentication AUTHENT-TACACS
length 0
transport input ssh
transport output ssh
----------

I have completed some debugging when using ssh and tacacs is down, i get the ~5 second delay before the command output and receive the following debug info:

*Aug 22 04:00:49.407: TAC+: Using default tacacs server-group "TACACS-SERVERS" list.
*Aug 22 04:00:49.407: TAC+: Opening TCP/IP to 192.168.1.100/49 timeout=5
RTR2#
*Aug 22 04:01:00.408: TAC+: TCP/IP open to 192.168.1.100/49 failed -- Connection timed out; remote host not responding
*Aug 22 04:01:00.409: TPLUS: Queuing AAA Accounting request 14 for processing
*Aug 22 04:01:00.409: TPLUS: processing accounting request id 14
*Aug 22 04:01:00.409: TPLUS: Sending AV task_id=12
*Aug 22 04:01:00.409: TPLUS: Sending AV timezone=UTC
*Aug 22 04:01:00.409: TPLUS: Sending AV service=shell
*Aug 22 04:01:00.409: TPLUS: Sending AV priv-lvl=1
*Aug 22 04:01:00.409: TPLUS: Sending AV cmd=show interfaces description <cr>
*Aug 22 04:01:00.409: TPLUS: Accounting request created for 14(localuser)
*Aug 22 04:01:00.409: TPLUS: Using server 192.168.1.100
RTR2#
*Aug 22 04:01:00.410: TPLUS(0000000E)/0/NB_WAIT/F10E7C8: Started 5 sec timeout
RTR2#
*Aug 22 04:01:05.410: TPLUS(0000000E)/0/NB_WAIT/F10E7C8: timed out
*Aug 22 04:01:05.410: TPLUS(0000000E)/0/NB_WAIT/F10E7C8: timed out, clean up
*Aug 22 04:01:05.410: TPLUS(0000000E)/0/F10E7C8: Processing the reply packet
RTR2#

---When using console (and tacacs server is either up or down) everything works as expected with no delays, debug shows only trying to output the accounting for that show command used:

*Aug 22 04:02:24.261: TPLUS: Queuing AAA Accounting request 13 for processing
*Aug 22 04:02:24.262: TPLUS: processing accounting request id 13
*Aug 22 04:02:24.262: TPLUS: Sending AV task_id=15
*Aug 22 04:02:24.262: TPLUS: Sending AV timezone=UTC
*Aug 22 04:02:24.262: TPLUS: Sending AV service=shell
*Aug 22 04:02:24.262: TPLUS: Sending AV priv-lvl=1
*Aug 22 04:02:24.262: TPLUS: Sending AV cmd=show interfaces description <cr>
*Aug 22 04:02:24.262: TPLUS: Accounting request created for 13(localuser)
*Aug 22 04:02:24.262: TPLUS: Using server 192.168.1.100
*Aug 22 04:02:24.263: TPLUS(0000000D)/0/NB_WAIT/E73BDE8: Started 5 sec timeout
RTR2#
*Aug 22 04:02:29.263: TPLUS(0000000D)/0/NB_WAIT/E73BDE8: timed out
*Aug 22 04:02:29.263: TPLUS(0000000D)/0/NB_WAIT/E73BDE8: timed out, clean up
*Aug 22 04:02:29.263: TPLUS(0000000D)/0/E73BDE8: Processing the reply packet

.

Let me know what you think, its a long winded one but im sure theres a simple solution that i just havent found yet!

Many thanks
Jamison

[Georg Pauwen]

Hello,

there used to be a command that lets you specify a single TACACS server, I am not sure if that has been deprecated. That command made sure that the TCP session the TACACS server remained open and did not have to be established each time a request is made. You can also specify a different port.

Here is an example:

tacacs-server host hostname [single-connection] [port integer] [timeout integer] [key string]

tacacs-server host TACACS_1 single-connection port 50 timeout 3 key secret_key

[jamsgrove]

Hi Georg,

Unfortunately that wont work as when the tacacs server is down there is now way to form a session with it. This entire problem is only happening when the tacacs server is down and we ssh to the switch/router, we can access and configure the device fine but every command takes about 5 seconds to work as it still keeps trying to contact the tacacs server.

[paul driver]

Hello

Apply an override to the Tacacs authorisation commands for local database, tacacs has a default 5 second delay on it timeout values, reducing it wont solve this, As you dont have a failback for authorisation it not letting you do anything.

As for the different users with various privilege levels then, if tacacs isn’t working then they cannot gain access unless they have a generic local DB account created with the same privilege levels so to failback on when taccas fails.

res
Paul

[jamsgrove]

Hi Paul,
There is a failback for authorization as each statement has "TACACS-SERVERS" and then "local" if tacacs cannot be reached.
When tacacs server is down i can still login fine as it reverts to needing local username/password. Once enable is entered i can then obviously use any command i like as enable authorizes me to level 15. However! Every command entered still has the 5 second delay before any input as the switch/router still tries to talk to the tacacs server each time (when it shouldnt and i dont want it to if i have already logged in with local and enable).

[paul driver]

Hello
Your correct my apologies wasnt looking correctly,
Can you try local if-authenticated and test

res
Paul

[jamsgrove]

I played around today with the following commands:

aaa authorization commands 0 COMMANDS-TACACS group TACACS-SERVERS local if-authenticated
aaa authorization commands 1 COMMANDS-TACACS group TACACS-SERVERS local if-authenticated
aaa authorization commands 15 COMMANDS-TACACS group TACACS-SERVERS local if-authenticated

It unfortunately made no change, after login the router still kept trying to authorize each command and hang for 5 seconds at a time.

[Georg Pauwen]

Hello,

on a side note (I googled around), can you try and disable ip domain-lookup (no ip domain-lookup) ?

[jamsgrove · Accepted solution]

Hey guys,

After much more testing and playing around the only thing i found worked was to swap the order of authorization. These are the commands different to my above configuration:

aaa authorization commands 0 COMMANDS-TACACS local group TACACS-SERVERS

aaa authorization commands 1 COMMANDS-TACACS local group TACACS-SERVERS

aaa authorization commands 15 COMMANDS-TACACS local group TACACS-SERVERS

Basically this tells the device to check itself for authorization first which is instant and even when the tacacs server is online and you are using tacacs user/password then that is also instant (it doesnt even try to test local which is perfect).

This now works exactly how i was wanting it and the only issue i could see with this is if you have a username/password configured on the device that was similar to one on the tacacs server.

Thanks for everyones help :)

Jamison