topic Re: Sudden failed authentications for user@domain in Network Access Control

Sudden failed authentications for user@domain

david_mayor — Mon, 11 Mar 2019 00:50:58 GMT

Hello,

We are running 6 ACS 4.1 servers on Windows 2003 Servers. These servers are not the same as the Domain Controllers.

Since many years, we have devices sending their username in the format domain\user and some other use user@domain. Everything was working well in our 6 ACS servers.

Suddenly, this morning, as 06:00:25, on one single server, all the request using user@domain were reported as failed with the follwowing message in the ACS logs: "External DB user invalid or bad password".

We first thought that the DC near the ACS server was the cause of the issue, but we observe that all the other ACS servers could process these user@domain AAA queries without problem. We then rebooted the ACS server and when it went back up, everything was running again like a charm.

We could not find what happened at 06:00:25. There is no Windows Scheduled Tasks at that time, and there is no ACS DB Replication or Backup running at that time neither.

Can someone help us troubleshooting that issue that affected only one single server in an unexpected way ?

Thanks a lot,

David Mayor

Re: Sudden failed authentications for user@domain

andamani — Tue, 22 Feb 2011 16:27:31 GMT

Hi David,

I assume the logging was set to full.

Please collect the package.cab an dpost it here.

Here is the link describing how to collect package.cab

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_server_for_windows/4.1.3/troubleshooting/guide/Ch1.html#wp1041303

Regards,

Anisha

Re: Sudden failed authentications for user@domain

david_mayor — Wed, 23 Feb 2011 10:09:32 GMT

Please find attached the package.cab generated.

For your record, the problems started at 06:00:25 CET (which is the time of the server) the 22nd of February. After the reboot of the server at 09:57, all queries in user@domain have been processed successfully again.

Thank you very much for your support.

Best regards,

David Mayor

Re: Sudden failed authentications for user@domain

andamani — Wed, 23 Feb 2011 10:28:55 GMT

Hi David,

Unfortunately the package.cab file did not provide me details or reason behind the authentication failure.

I guess the logging is not set to full.

Please set the logging to full in case the issue occurs in future.

Regards,

Anisha

Re: Sudden failed authentications for user@domain

david_mayor — Wed, 23 Feb 2011 10:43:23 GMT

Hello,

You are right. We let the level to Low in normal operations. And unfortunately, we have not put the level at high during the issue. However, I suspect that if I had put to High during the problem, the services would have been restarted and the problem solved, and we would not have any significant logs as well...

I don't think that letting the debug to Full is recommended for a long period of time.

So according to you, there is nothing or no known bug which can generate such a behavior ? And there is nothing we can do more to investigate more deeply ?

With my best regards,

David M.

Re: Sudden failed authentications for user@domain

andamani — Sat, 26 Feb 2011 05:43:18 GMT

Hi David,

Making logging to full is recommended for troubleshooting purpose.

In case the ACS is on windows we need to keep moving the logs to ensure that the drive is not ful.

In case of ACS for appliance we can move the logs to remote agent.

Full logging would give us details of why the authentication stopped. It is very difficult or rather impossible to give you details as to why the authentication stopped.

We need logging at full to find what wrong is going on or what wrong happenned.

Please ensure we have logging full to find out the reason in case the issue occurs in future.

Regards,

Anisha

Re: Sudden failed authentications for user@domain

david_mayor — Tue, 01 Mar 2011 10:38:18 GMT

Hello,

This morning, the problem happened again this morning !

At 06:00:59, the authentication requests in the format user@gmoff failed again. At 7h55, I set the log details at Full and restarted the services from the Web GUI. The restart of the service did not solve the issue.

I let the full debug during about 3 minutes. Then, I restarted the server. When it restarted, the problem was solved. It is exactly the same schedule of last week.

I generated a package.cab. The full debug has been enabled from about 07:55 AM.

Can you please have a look at the package and see if the full debug gives some indications ?

Thanks a lot,

With my best regards,

David Mayor

Re: Sudden failed authentications for user@domain

andamani — Tue, 01 Mar 2011 12:17:40 GMT

Hi David,

Please complete the post installations tasks mentioned below:

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_server_for_windows/4.1/installation/guide/windows/postin.html

The error Windows authentication FAILED (error 1326L) usually comes when the services are not run with proper account.

Let me know if this helps or not.

Regards,

Anisha

P.S.: please mark this thread as answered if you feel your query is resolved. Do rate helpful posts.

Re: Sudden failed authentications for user@domain

david_mayor — Tue, 01 Mar 2011 14:57:52 GMT

Hello Anisha,

I understand that with new installation, such post tasks are required. However, our installation is running in such a state for more than 2 or 3 years. And it is only over the past week that such problem happens twice.

We have also observed one more thing: You know that the main problem started few seconds after 6 AM, in both days when it happened. We observed that between 00:02 (midnight + about 2 minutes) and 01:05 AM, the same problems happens also ! But, at 01:05 AM, the problem automatically goes away without any intervention. However, when it happens again at 6 AM, we have to restart the server, because otherwise it would not automatically recover.

Didn't you find anything else than "error Windows authentication FAILED (error 1326L)" on the full log ?

Thanks a lot,

With my very best regards,

David Mayor

Re: Sudden failed authentications for user@domain

andamani — Tue, 01 Mar 2011 15:09:33 GMT

Hi David,

The full logging gave just these logs.

The logs just mentioned the error i pasted. I have solved a few cases in which the services when started with appropriate user account has resolved this issue of sudden authentication failure i.e. post installation tasks were performed.

If you want to dig in further, please open a TAC case.

Also i guess you are running ACS 4.1 . i would ask you to upgrade to ACS 4.2.0.124 patch 17.

Regards,

Anisha.

Re: Sudden failed authentications for user@domain

david_mayor — Wed, 02 Mar 2011 07:54:51 GMT

Hello,

This morning again, we faced the same issue. It again started at 00:03 till 01:05 and automatically recovered. But at 6 AM, again it did not work and I had to manually restart the server.

I posted again the package.cab, hoping it will give more information than yesterday.

Thanks a lot,

With my best regards,

David

Re: Sudden failed authentications for user@domain

andamani — Fri, 04 Mar 2011 01:54:51 GMT

Hi David,

I cannot find the latest package.cab file. 😞

Regards,

Anisha