[Maybe solved] Repeated prompts for username and password on HTTP(S)

[Gordon Fecyk]

I'm new to IOS but I'm taking ICND 1 and ICND 2 to catch up. I've gone as far as configuring a 3560 switch from "scratch" in a test environment, and am in part one of ICND 2.

Setting this switch up for HTTP administration, I've been prompted repeatedly for my username and password both when using a browser directly and when using Cisco Network Assistant (currently 5.5). This is different from the production switches I'm working with, that will prompt me once only and let me work. At first I suspected this was because I'm trying to use SSL and an in-house certitficate authority to sign the switch's certificate, but this is happening in plain HTTP as well.  If I'm persistent enough, I eventually get the pages or configuration screens I want.

The difference between the production 3560s and my testing one, is the test one uses IOS 12.2(53), the current version.  The production ones use 12.2(25)SEE3 through 12.2(35)SE5 and probably everything in between.  I'm also experimenting with the releases that includes the web-based device manager on my test switch, but the behaviour is similar whether the image has the web-based manager or not.

[Gordon Fecyk]

In an update to this problem, I've reverted the switch I've been testing back to 12.2(35)SE5, which is what the majority of my production 3560s use. It appears to behave in the same fashion: Repeatedly asking for credentials over HTTP(S).

While my non-production switch has a local user database set up, my production switches use two ACS servers.  The relevant bits of the running config are:

aaa new-model
aaa group server tacacs+ mcbhtacacs
server 10.1.2.221
server 10.1.2.222
!
aaa group server radius mcbhradius
server 10.1.2.221 auth-port 1645 acct-port 1646
server 10.1.2.222 auth-port 1645 acct-port 1646
!
aaa authentication login default group mcbhtacacs local
aaa authentication login LOCALAUTH local
aaa authorization exec default group mcbhtacacs
aaa accounting send stop-record authentication failure
aaa accounting exec default start-stop group mcbhtacacs
aaa accounting commands 1 default start-stop group mcbhtacacs
aaa accounting commands 15 default start-stop group mcbhtacacs
aaa accounting system default start-stop group mcbhtacacs
!
aaa session-id common

[...]

tacacs-server host 10.1.2.221 key 7 [censored]

tacacs-server host 10.1.2.222 key 7 [censored]
tacacs-server directed-request
radius-server host 10.1.2.221 auth-port 1645 acct-port 1646 key 7 [censored]
radius-server host 10.1.2.222 auth-port 1645 acct-port 1646 key 7 [censored]
radius-server source-ports 1645-1646

Another switch that does work without repeated prompting has these relevant bits:

aaa new-model
aaa group server tacacs+ mcbhtacacs
server 10.1.2.221
server 10.1.2.222
!
aaa group server radius mcbhradius
server 10.1.2.221 auth-port 1645 acct-port 1646
server 10.1.2.222 auth-port 1645 acct-port 1646
!
aaa authentication login default group mcbhtacacs local
aaa authentication login LOCALAUTH local
aaa authorization exec default group mcbhtacacs local
aaa accounting send stop-record authentication failure
aaa accounting exec default start-stop group mcbhtacacs
aaa accounting commands 1 default start-stop group mcbhtacacs
aaa accounting commands 15 default start-stop group mcbhtacacs
aaa accounting system default start-stop group mcbhtacacs
!
aaa session-id common

[...]

tacacs-server host 10.1.2.221 key 7 [censored]
tacacs-server host 10.1.2.222 key 7 [censored[
tacacs-server directed-request
radius-server host 10.1.2.221 auth-port 1645 acct-port 1646 key 7 [censored]
radius-server host 10.1.2.222 auth-port 1645 acct-port 1646 key 7 [censored]
radius-server source-ports 1645-1646

10.1.2.221 and 10.1.2.222 are the ACS servers, running ACS 4.1 and are linked to a pair of Active Directory domain controllers.

Aside from host names, these two switches appear to have an identical AAA configuration.  So why would the first switch repeatedly ask for credentials over HTTP when the second switch with the same software does not?

[Ganesh Hariharan]

In an update to this problem, I've reverted the switch I've been
testing back to 12.2(35)SE5, which is what the majority of my
production 3560s use. It appears to behave in the same fashion:
Repeatedly asking for credentials over HTTP(S).
While
my non-production switch has a local user database set up, my
production switches use two ACS servers.  The relevant bits of the
running config are:
aaa new-model
aaa group server tacacs+ mcbhtacacs
 server 10.1.2.221
 server 10.1.2.222
!
aaa group server radius mcbhradius
 server 10.1.2.221 auth-port 1645 acct-port 1646
 server 10.1.2.222 auth-port 1645 acct-port 1646
!
aaa authentication login default group mcbhtacacs local
aaa authentication login LOCALAUTH local
aaa authorization exec default group mcbhtacacs
aaa accounting send stop-record authentication failure
aaa accounting exec default start-stop group mcbhtacacs
aaa accounting commands 1 default start-stop group mcbhtacacs
aaa accounting commands 15 default start-stop group mcbhtacacs
aaa accounting system default start-stop group mcbhtacacs
!
aaa session-id common
[...]
tacacs-server host 10.1.2.221 key 7 [censored]
tacacs-server host 10.1.2.222 key 7 [censored]
tacacs-server directed-request
radius-server host 10.1.2.221 auth-port 1645 acct-port 1646 key 7 [censored]
radius-server host 10.1.2.222 auth-port 1645 acct-port 1646 key 7 [censored]
radius-server source-ports 1645-1646
Another switch that does work without repeated prompting has these relevant bits:
aaa new-model
aaa group server tacacs+ mcbhtacacs
 server 10.1.2.221

Hi,

What failed attempts messages says in ACS 4.1 whenever you provide credential to the switch which is prompting for username and passwiord when doing using HTTP.

Ganesh.H

[Gordon Fecyk]

ACS' logs say it's authenticating successfully, and the user account I'm using has Level 15 (privileged exec) access.  If I log on to the switch using SSH using the same account it takes me to privileged exec mode straight away.

Further, the Windows server running ACS has several success audit entries with my AD username.

I'm going to try to take ACS out of the equation and add it back in.  I first tried adding a local username to my production switch giving me this trouble, but that isn't working; the switch is defaulting to ACS authentication.  So I'll take the test switch I know is working with local user accounts, have it use ACS, and see what happens.

This might be a problem across all of my 3560s regardless of IOS version.  I only started trying the web-based device manager included with 12.2 this month, and the repeated authentication problem is more pronounced using that than it is using something like Cisco Network Assistant.  It happens in CNA, but not as often so I might have not paid it any attention until now.

[Gordon Fecyk]

OK so it seems specific to AAA using my ACS servers only.  My testing switch, which worked without problems with a local user database, is repeatedly prompting me for a username and password when I try to authenticate via ACS using TACACS+.

A simplified config that reproduces my problem:

aaa group server tacacs+ mcbhtacacs
server 10.1.2.221
server 10.1.2.222
!
aaa authentication login default group mcbhtacacs
aaa authorization exec default group mcbhtacacs
aaa authorization network default group mcbhtacacs
aaa accounting exec default start-stop group mcbhtacacs
aaa accounting network default start-stop group mcbhtacacs
!
!
!
aaa session-id common
[...]
tacacs-server host 10.1.2.221 key (Censored)
tacacs-server host 10.1.2.222 key (Censored)
tacacs-server directed-request

I created the two tacacs hosts, supplied the key (readable from ACS' console), and followed the instructions in the 12.2(52)SE1 guide for configuring switch-based authentication. That's there the five AAA lines are from.  Originally there were no aaa accounting lines, and the other aaa lines used default local:

aaa authentication login default local
aaa authorization exec default local
aaa authorization network default local

Using default local with some local users created works fine without repeated prompts.  I might be missing something in ACS.

When I see this kind of behaviour on a real web server, I suspect there's a file system permissions problem and I ensure the user in question has access to all of the HTML and graphics documents they're supposed to have.  This makes me wonder if I'm missing some kind of permissions setting to allow ACS-authenticated users access to the web files stored in flash.  The user in question has Level 15; logging on as that user with SSH gives me privileged exec mode straight away.

[Gordon Fecyk]

I moved this thread from LAN switching to AAA because it's an AAA problem, it seems.  Local users and passwords work correctly with the integrated HTTP(S) server on a Catalyst 3560 running IOS 12.2(53)SE1. Attempting to enable TACACS+ authentication on the same switch is resulting in repeated prompts for a username and password over HTTP or HTTPS.

The switch currently has the full web-based device manager installed on it, as the problem is most noticable when this is on the switch.  The basic web-based device manager still has a repeated prompting problem, but it is not as pronounced.  Also, Cisco Network Assistant 5.5 will repeatedly prompt for credentials, but again not as pronounced.

The common denominator here is HTTP and ACS 4.1.

I first copied the configuration from another, already working switch to my testing switch.  This gave me the original problematic result.  I then restored the switch's configuration to a local user list, which worked as I wanted, and then added a bare-bones TACACS+ configuration.  Config examples are in earlier posts in this thread.  In every case, if I manage to get TACACS+ working at all, HTTP will prompt me for credentials repeatedly.

Aside from wiresharing the switch and seeing what's happening, are there any other suggestions? There's no web server log as I know it on the switch, though I suppose I could enable syslog messages and direct them to a syslog server.

[Gordon Fecyk]

One more update. I ended up redoing a lot of stuff on our secondary ACS server to match the primary server's configuration. At first, the secondary server was recording a few "Authen Failed" log entries, but that was only because this server was not instructed to check any external databases (ACS user unknown). Once it was told to look in Active Directory, I started getting multiple passed authentications there. I'm also showing up in Group 15 when I do.

At this point I'm at a loss as to why I'm being prompted multiple times for my username and password when I log on to this switch. I'm going to take this to TAC after I gather more information.

[Gordon Fecyk]

This might not be the best solution, but it did eliminate the repeating prompts. :-/

The winning configuration ended up looking like this:

aaa new-model
!
!
aaa authentication login default group tacacs+ cache tacacs+ local
aaa authorization exec default group tacacs+ cache tacacs+ local
aaa authorization network default group tacacs+ cache tacacs+ local

!
!
!
aaa session-id common
[...]

ip http authentication aaa
[...]

tacacs-server host 10.1.2.221 key 7 [censored]
tacacs-server host 10.1.2.222 key 7 [censored]
tacacs-server directed-request

(also tried no tacacs-server directed-request)
[...]

[end]

The key seemed to be providing a cache option.  I had a AAA server group defined at one point and had cache all set there, but that didn't seem to make a difference. Treating all (2) of my tacacs+ servers as a group and specfying cache tacacs+ seemed to remove the repeated prompting.

I guess this makes sense. With a local user database, the switch had the username and password in memory to check against. With tacacs+ it seemed to constantly refer to the tacacs+ servers. It still does, though apparently less frequently, when I tell it to cache tacacs+ responses. That seems to be the nature of http though; where a username and password are provided on every transaction.

http(s) is still dreadfully slow when authenticating against tacacs+ instead of a local user database. I tried a couple of options, such as no tacacs-server directed-request, which seemed to make no difference.

I suppose using a crypto build and a 1024-bit key pair for SSH and HTTPS isn't helping matters, but these worked reasonably quickly when I was using a local database and not tacacs+. Just to make sure it wasn't a tacacs+ problem I used debug tacacs on the console and looked for failures; a lot of noise from multiple transactions, and nothing obvious but I did find:

May 11 01:05:35.819: TPLUS: Queuing AAA Authorization request 243 for processing

May 11 01:05:35.819: TPLUS: processing authorization request id 243
May 11 01:05:35.819: TPLUS: Inappropriate protocol: 25

And also what looked like a program crash:

May 11 01:06:39.950: %SCHED-7-WATCH: Attempt to enqueue uninitialized watched queue (address 0). -Process= "TPLUS", ipl= 0, pid= 254
-Traceback= 275B200 1D7B4FC 1CDF110 1CDE678 26FFC28 2709368 27095AC 26FF6A4 1B6796C 1B5E3F8

So maybe the full web based device manager is a resource hog on the poor 3560. But the repeated prompting no longer happens on Cisco Network Assistant, and I could load the simpler versions of IPBASE IOS without the device manager and use CNA to manage the switch.

Also on the plus side, I learned how to properly set up a certificate signed by a private CA for https. No more self-signed certs.

[Javier Henderson]

Yes, the cache is what "fixed" this. On the web pages, each object has to be authenticated, so without the cache, you will be prompted to authenticate numerous times.

[Gordon Fecyk]

OK so caching tacacs+ responses helps with authenticating and authorizing http(s) users.  There is still an unusual delay in returning some components of the web-based device manager, and even the occasional exception as I've provided in a tacacs+ debug log.

If I turn on tacacs+ debugging, the console spews out repeated activity.  If I'm authenticating as a tacacs+ user and authorizing as a group 15 user, and the switch is supposed to cache the response, why would the switch be logging repeated connections to the tacacs+ servers?

I've made additional adjustments to a production switch that had the repeated password prompts and I'm now getting the same behaviour as I get on my testing switch.  The production switch sits in a training lab, so I'm not breaking anything critical there. The AAA lines look like this:

aaa new-model
aaa group server tacacs+ mcbhtacacs
server 10.1.2.221
server 10.1.2.222
cache expiry 1
cache authorization profile mcbhcache
cache authentication profile mcbhcache
!
[...]

!
aaa authentication login default group mcbhtacacs cache mcbhcache local
aaa authentication login LOCALAUTH local
aaa authorization exec default group mcbhtacacs cache mcbhcache local
aaa authorization network default group mcbhtacacs cache mcbhcache local
[...]

aaa cache profile mcbhcache
all
!

[...]

ip http authentication aaa login-authentication mcbhtacacs
[...]

!
[...]

tacacs-server host 10.1.2.221 key 7 [censored]

tacacs-server host 10.1.2.222 key 7 [censored]
tacacs-server directed-request

[...]

[end]

The production switch had a distinct tacacs+ server group defined prior to my training, instead of just using all tacacs+ servers as I am on my testing switch. I had to create a distinct cache profile in response, but it just defaults to caching all.

Is there anything I can do to improve the performance of AAA caching, to act more like the local user list I originally had on my testing switch?

[pablo.thoma]

Hi,

I am facing exactly the same problem as described here. All attempts to find a working solution have failed, the caching configuration helps a little bit but the web access to the device manager on the box is still unusably slow and renders multiple connections to the ACS despite the caching.

The "best" configuration so far I found is:

aaa new-model

!

!

aaa authentication login default group tacacs+ cache tacacs+ local

aaa authorization exec default group tacacs+ cache tacacs+ local

!

ip http authentication aaa

The tacacs servers are individually configured, nothing special there.

Can't see this being ACS related, rather it seems to be a buggy or suboptimal handling in the AAA or HTTPD code.

The platform/version is

c3560-ipbasek9-mz.122-53.SE2.bin

I also get the following tracebacks:

Jul  6 11:36:47.126 UTC: %SCHED-7-WATCH: Attempt to enqueue uninitialized watched queue (address 0). -Process= "AAA Server", ipl= 0, pid= 178

-Traceback= 275B200 1D7B4FC 1CDF110 1CDFD28 1CDFE74 1B6796C 1B5E3F8

Any help would be greatly appreciated.

Thanks!

Pablo