Random devices w/ tacacs authorization slowness in ACS

Jacob-Harris · ‎09-19-2015

Hi Community,

First off, I've got an open TAC case on this and will post if I get a solution, but I've had luck here in the past so here goes!

We're experiencing aaa authorization slowness (experienced by a 5 second delay per command on the CLI of the router/switch) on a few random devices within our environment (ASR, 4500-E, and 2951). Debug's show the 5 second delay, but don't point to an issue. The ACS's are both in good health and don't show any issues in the log. Ping responses between the Router and ACS are 1 ms consistently.

We were able to resolve (temporarily) the issue by disabling ACS 1 in the routers config (just changed the ip address of the acs) leveraging our backup virtual ACS. Command speeds returned to normal when we used ACS 2. We then pointed back to ACS 1 and slowness returned. We also tried deleting the device from ACS and rebuilding it, no luck.

Anyone have any ideas?

Thanks!

Jacob-Harris · ‎10-27-2015

Problem solved. It ended up being a command on the Routers / Switches configured under the TACACS servers that was creating the delay.

After removing the command "single-connection" all is good.

Example:

tacacs server Server01

address ipv4 10.10.10.10

key 7 WXYZ

no single-connection <~~~~ Removes single connection.

Nathan Spitzer · ‎01-25-2016

For posterity:

Single connection was basically a psuedo-standard to allow TCP connection reuse. The idea was that in common TACACS+ configurations there are three separate TCP sessions per logon: one for authentication, one for authorization, and one for accounting. And then possibly two more for every command entered: one authorization and one accounting.

Since starting up TCP sessions takes a relatively large chunk of CPU time as well as possibly significant latency for the three-way handshake to complete by reusing a single TCP connection servers could support much more devices and AAA functions would happen quicker.

Unfortunately Cisco never got it to work right and if you search forTACACS+ single-connection bugs you will find many with notations they will never be fixed. I tried it myself but buggy was an understatement.