AAA console access very slow on L2 switch vs. Router

[RAMAN AZIZIAN]

Hello everyone,

Trying to figure out why does issuing commands on Cisco 3850 via console take long time vs. ISR4431, when TACACS+ is down.

Example:

When I issue sho run on the 3850, it takes about 20 seconds before I see it on the screen, however, issuing the same command on the ISR 4431, it is almost instantaneous (<2 seconds).

Switch is running 16.6.4

I have also changed the method to first use local and then group GROUP_NAME, and I get the same result.

Here's a sample of the AAA config:

aaa authentication login default group GROUP_NAME local
aaa authentication enable default group GROUP_NAME enable
aaa authorization exec default group GROUP_NAME local
aaa authorization exec ISE group GROUP_NAME local
aaa authorization commands 15 default group GROUP_NAME local
aaa authorization config-commands
aaa authorization console
aaa accounting exec default start-stop group GROUP_NAME
aaa accounting commands 15 default start-stop group GROUP_NAME
aaa accounting system default start-stop group GROUP_NAME

Any suggestions/tips would be greatly appreciated.

Thanks,

raman

[paul driver]

Hello

For a test can you disable DNS lookup if enabled

no ip domain-lookup

[RAMAN AZIZIAN]

Hi Paul,

I have verified that command is disabled on the switch. My AAA configs are almost identical on both the switch and the router.

Be glad to look at other commands if needed.

Thanks for taking the time to view this post.

raman

[Reza Sharifi]

Hi Raman,

Does this only happen when you log in via the console?

The reason I ask is that we have some 3850s and when we issue "sh run" it takes about 10 to 15 seconds for the output to show up regardless of how you connect to them (Console, SSH, telnet, ACS, local,  etc..).

HTH

[RAMAN AZIZIAN]

Yes, this only happens when I console in. I can run the same command via SSH with TACACS being accessible, and it is instantaneous. Also, I'm not stacking the switch. I don't see the delay for instance if use the ? to view commands associated with specific commands.

[paul driver]

Hello

---

@RAMAN AZIZIAN wrote:

Yes, this only happens when I console in. I can run the same command via SSH with TACACS being accessible, and it is instantaneous. Also, I'm not stacking the switch. I don't see the delay for instance if use the ? to view commands associated with specific commands.

---

line console 0
transport preferred none
exit

[RAMAN AZIZIAN]

unfortunately that did not work. Also I tried it on 3650 and I get the same behavior.

Curious, if you see the same thing if you have a lab setup?

Thanks,

[paul driver]

Hello

The past two test i have asked you to perform were around dns resolution but this doesn't seem to have helped.

So can you post your line config and also output from debug aaa accounting/authorization  please if applicable.

Lastly if you remove aaa accounting temporally does that make any difference?

[RAMAN AZIZIAN]

Paul,

I will see if I can include the debug output. I can tell you when I do run the debug each the switch does attempt to communicate with the server, which is the behavior of timing out since it is not communicating with the server.

[Richard Burts]

raman

I can only work with the information presented. Your original post described symptoms associated with tacacs being down. I found factors in the provided configuration that explain that behavior. My analysis seems to be supported by your comments about debug output

I can tell you when I do run the debug each the switch does attempt to communicate with the server,

If you have other experience where authorization used local first and then tacacs then please provide documentation of that experience - provide the changed configuration and debug output showing no attempt to communicate with tacacs.

HTH

Rick

[paul driver]

Hello

---

@RAMAN AZIZIAN wrote:

Paul,

 I can tell you when I do run the debug each the switch does attempt to communicate with the server, which is the behavior of timing out since it is not communicating with the server.

---

Do you have tacacs-server timeout period set to other than the default ?
tacacs-server timeout xx

[Richard Burts]

Assuming that the part of the original post showing the aaa configuration accurately reflects what is on this switch there is a pretty easy explanation for the symptoms described. The key to understanding it is the reference that it happens when tacacs is down. Here are the key lines

aaa authorization commands 15 default group GROUP_NAME local
aaa authorization config-commands
aaa authorization console

I will start by pointing to the line that enables authorization for the console. By default the Cisco console does not perform authorization. But here it is enabled. Also note that authorization goes to tacacs first with local as a fall-back. So when he enters the show run command it sends an authorization request to tacacs. He must wait for that to time out (and potentially a retry to timeout) before the command actually executes.

HTH

Rick

[RAMAN AZIZIAN]

Hello Rick,

Thanks for your reply.

I have tried the authentication/authorization method to point to the local first and then group name and I get the same result.

What puzzles me is why the router does not experience the same delay as the switch, and it has the same exact AAA configuration.

The switch is directly connected to the router via .1q, which shouldn't really matter (at least I don't think it should), since the TACACS server (via ISE)  is down, and I am consoling in directly to the switch.

[joepak]

So show run performs slowly only when TACACS is disabled for the switch? And if TACACS is enabled the switch performs show run in a prompt manner?

How is the memory statistics on the switch? And can you confirm/verify that TACACS being enabled allows the switch to run properly?

[RAMAN AZIZIAN]

Memory is not over utilized and yes, if TACACS is back online, it works with no issues.

thanks,

raman