cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1197
Views
12
Helpful
11
Replies

Sudden failed authentications for user@domain

david_mayor
Level 1
Level 1

Hello,

We are running 6 ACS 4.1 servers on Windows 2003 Servers. These servers are not the same as the Domain Controllers.

Since many years, we have devices sending their username in the format domain\user and some other use user@domain. Everything was working well in our 6 ACS servers.

Suddenly, this morning, as 06:00:25, on one single server, all the request using user@domain were reported as failed with the follwowing message in the ACS logs: "External DB user invalid or bad password".

We first thought that the DC near the ACS server was the cause of the issue, but we observe that all the other ACS servers could process these user@domain AAA queries without problem. We then rebooted the ACS server and when it went back up, everything was running again like a charm.

We could not find what happened at 06:00:25. There is no Windows Scheduled Tasks at that time, and there is no ACS DB Replication or Backup running at that time neither.

Can someone help us troubleshooting that issue that affected only one single server in an unexpected way ?

Thanks a lot,

David Mayor

11 Replies 11

andamani
Cisco Employee
Cisco Employee

Hi David,

I assume the logging was set to full.

Please collect the package.cab an dpost it here.

Here is the link describing how to collect package.cab

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_server_for_windows/4.1.3/troubleshooting/guide/Ch1.html#wp1041303

Regards,

Anisha

Please find attached the package.cab generated.

For your record, the problems started at 06:00:25 CET (which is the time of the server) the 22nd of February. After the reboot of the server at 09:57, all queries in user@domain have been processed successfully again.

Thank you very much for your support.

Best regards,

David Mayor

Hi David,

Unfortunately the package.cab file did not provide me details or reason behind the authentication failure.

I guess the logging is not set to full.

Please set the logging to full in case the issue occurs in future.

Regards,

Anisha

Hello,

You are right. We let the level to Low in normal operations. And unfortunately, we have not put the level at high during the issue. However, I suspect that if I had put to High during the problem, the services would have been restarted and the problem solved, and we would not have any significant logs as well...

I don't think that letting the debug to Full is recommended for a long period of time.

So according to you, there is nothing or no known bug which can generate such a behavior ? And there is nothing we can do more to investigate more deeply ?

With my best regards,

David M.

Hi David,

Making logging to full is recommended for troubleshooting purpose.

In case the ACS is on windows we need to keep moving the logs to ensure that the drive is not ful.

In case of ACS for appliance we can move the logs to remote agent.

Full logging would give us details of why the authentication stopped. It is very difficult or rather impossible to give you details as to why the authentication stopped.

We need logging at full to find what wrong is going on or what wrong happenned.

Please ensure we have logging full to find out the reason in case the issue occurs in future.

Regards,

Anisha

Hello,

This morning, the problem happened again this morning !

At 06:00:59, the authentication requests in the format user@gmoff failed again. At 7h55, I set the log details at Full and restarted the services from the Web GUI. The restart of the service did not solve the issue.

I let the full debug during about 3 minutes. Then, I restarted the server. When it restarted, the problem was solved. It is exactly the same schedule of last week.

I generated a package.cab. The full debug has been enabled from about 07:55 AM.

Can you please have a look at the package and see if the full debug gives some indications ?

Thanks a lot,

With my best regards,

David Mayor

Hi David,

Please complete the post installations tasks mentioned below:

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_server_for_windows/4.1/installation/guide/windows/postin.html

The error Windows authentication FAILED (error 1326L) usually comes when the services are not run with proper account.

Let me know if this helps or not.

Regards,

Anisha

P.S.: please mark this thread as answered if you feel your query is resolved. Do rate helpful posts.

Hello Anisha,

I understand that with new installation, such post tasks are required. However, our installation is running in such a state for more than 2 or 3 years. And it is only over the past week that such problem happens twice.

We have also observed one more thing: You know that the main problem started few seconds after 6 AM, in both days when it happened. We observed that between 00:02 (midnight + about 2 minutes) and 01:05 AM, the same problems happens also ! But, at 01:05 AM, the problem automatically goes away without any intervention. However, when it happens again at 6 AM, we have to restart the server, because otherwise it would not automatically recover.

Didn't you find anything else than "error Windows authentication FAILED (error 1326L)" on the full log ?

Thanks a lot,

With my very best regards,

David Mayor

Hi David,

The full logging gave just these logs.

The logs just mentioned the error i pasted. I have solved a few cases in which the services when started with appropriate user account has resolved this issue of sudden authentication failure i.e. post installation tasks were performed.

If you want to dig in further, please open a TAC case.

Also i guess you are running ACS 4.1 . i would ask you to upgrade to ACS 4.2.0.124 patch 17.

Regards,

Anisha.

Hello,

This morning again, we faced the same issue. It again started at 00:03 till 01:05 and automatically recovered. But at 6 AM, again it did not work and I had to manually restart the server.

I posted again the package.cab, hoping it will give more information than yesterday.

Thanks a lot,

With my best regards,

David

Hi David,

I cannot find the latest package.cab file. :-(

Regards,

Anisha

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: