Solved: Re: Problem in ACS v 4.1

ranjit123 · ‎10-15-2010

Dear All,

We had faced a problem in our ACS Server 4.1, it refused all the user connections for 15 mins and we were not able to authenticate through our TACACS username and password during this perdiod. After 15 mins things became normal

Below were the logs generated by the server during this period.

==========================================================================================

Fri Oct 15 17:02:24 2010): Info: GetApplNICConfig GetIfTable size = 11192
(Fri Oct 15 17:02:24 2010): Info: GetApplNICConfig, adpt Idx = 16777220, en adpt Idx = 16777219
(Fri Oct 15 17:02:24 2010): Info: GetApplNICConfig, adpt Idx = 16777219, en adpt Idx = 16777219
(Fri Oct 15 17:02:24 2010): Info: GetApplNICConfig ip < ip address>, mask 255.255.255.240, gateway < ip address>,
(Fri Oct 15 17:09:00 2010): Trying to get current administrator name...
(Fri Oct 15 17:09:01 2010): checking Administrator: admin...
(Fri Oct 15 17:09:01 2010): Administrator admin found
(Fri Oct 15 17:09:01 2010): Trying to get current administrator name...
(Fri Oct 15 17:09:01 2010): checking Administrator: admin...
(Fri Oct 15 17:09:01 2010): Administrator admin found
(Fri Oct 15 17:19:54 2010): Trying to get current administrator name...
(Fri Oct 15 17:19:54 2010): checking Administrator: admin...
(Fri Oct 15 17:19:54 2010): Administrator admin found
(Fri Oct 15 17:19:54 2010): Trying to get current administrator name...
(Fri Oct 15 17:19:54 2010): checking Administrator: admin...
(Fri Oct 15 17:19:54 2010): Administrator admin found

===========================================================================================

Regards,

Ranjit

Tiago Antunes · ‎10-18-2010

As i wrote before:

I would leave the LogLevel to FULL and monitor the ACS so that if it happens again, you can collect the package.cab imediately after the problem occurs and the needed logs will be there.

HTH,
Tiago

--

If this helps you and/or answers your question please mark the question as "answered" and/or rate it, so other users can easily find it.

View solution in original post

Tiago Antunes · ‎10-15-2010

Do you have replication configured?

If yes, can you check if this 15 mins were during the replication process? If yes, it is expected.

Can you share with us the csmon.log file from the C:\Program Files\CiscoSecure ACS v4.2\CSMon\Logs directory?

Cheers,
Tiago

--

If this answers your question please mark the question as "answered" and rate it, so other users can easily find it.

ranjit123 · ‎10-15-2010

Hi,

Thanks for your reply,replications is configured but its duration is 120 mins

please find the csmon.log file attached along with this mail.

ACS was implemented in 2008 and this problem occured the first time since then.

Regards,

Ranjit

Tiago Antunes · ‎10-15-2010

Hi,

Thanks but this is not the file I was asking for.

Can you share with us the "csmon.log" text file from the C:\Program Files\CiscoSecure ACS v4.2\CSMon\Logs directory?

Are you sure no one else configure replication?

Thanks,

Tiago

ranjit123 · ‎10-15-2010

Hi Tiago,

It is a Appliance and LINUX based.

below is the snap shot of the Diagnostic logs avaiable on the box.

Please update me which logs do you want for reference.

Regards,

Ranjit

Tiago Antunes · ‎10-15-2010

Hi Ranjit,

Yes, indeed it is an appliance, however please be aware that it is Windows based even though you don't have access to the OS level.

Ok, so you can collect the package.cab file that you can obtain when you go to System Configuration -> Support -> Collect log file, and collect log files from previous x days making sure you catch the time of the outage.

Thanks,

Tiago

ranjit123 · ‎10-16-2010

Hi!,

Please find the package.cab file attached.

Regards,

Ranjit

Tiago Antunes · ‎10-16-2010

Hi Ranjit,

I see that the timestamp on your initial post isa bit deslocated in relation to the time on the ACS.

On the ACS i see that the authentications stopped between 10/15/2010 16:49:46 and 17:08:09:

...

CSMon 10/15/2010 16:49:46 A 0523 15836 CSTacacs: Failed to authenticate on test account.

CSMon 10/15/2010 16:49:56 I 0718 15836 Auth Failure Retry 1 (Successful auths this cycle 0)

CSMon 10/15/2010 16:50:06 I 0718 15836 Auth Failure Retry 2 (Successful auths this cycle 0)

CSMon 10/15/2010 16:50:16 I 0718 15836 Auth Failure Retry 3 (Successful auths this cycle 0)

CSMon 10/15/2010 16:50:26 I 0718 15836 Auth Failure Retry 4 (Successful auths this cycle 0)

CSMon 10/15/2010 16:50:46 I 0747 15836 Confirmed alert on CSTacacs
CSMon 10/15/2010 16:50:46 E 0748 15836 CSTacacs: Failed to authenticate on test account.

CSMon 10/15/2010 16:50:46 A 0641 43980 CSTacacs: State 6 0 Event Detected Level:4 Message:CSTacacs: Failed to authenticate on test account.

CSMon 10/15/2010 17:06:36 A 0152 43980 Services were all restarted. Attempt 1.

CSMon 10/15/2010 17:08:09 I 0530 15836 CSTacacs: Authenticated
CSMon 10/15/2010 17:08:09 I 0653 43980 CSTacacs: State 0 6 No Problems

...

This tell us that something happened with the tacacs+ service that made the ACS restart the services to resume normal operations.

Unfortunately the TCS logs of the package.cab you sent do not include any logs prior to Oct 16th... Have you collected the package.cab for how many previous days? Please try to collect for previous 3 days, to make sure we get the logs of the 15th Oct.

Thanks,
Tiago

--

If this helps you and/or answers your question please mark the question as "answered" and rate it, so other users can easily find it.

ranjit123 · ‎10-16-2010

Hi!,

Please check the same

Regards,

Ranjit

Tiago Antunes · ‎10-16-2010

Hi Ranjit,

Unfortunately, there is nothing there again...if you open the package.cab yourself, you will see that the file TCS.log contains no logs for the 15th Oct...

Sorry but without them there is no way to know why the tacacs+ service was failing...

Cheers,

Tiago

ranjit123 · ‎10-17-2010

Hi!,

If replication happening i guess it will refuse all connections.

Regards,

Ranjit