ACS to RA server communication issue.

Prasan Venky · ‎05-09-2013

Hi all,

With reference to the post https://supportforums.cisco.com/message/3931622#3931622

I could find the below error on 'ACSRemoteAgent log file earlier.

ACSRemoteAgent 04/04/2012 08:30:34 A 0137 1752 0x0 CSLogAgent launched

ACSRemoteAgent 04/04/2012 08:30:44 E 0194 1752 0x0 CSLogAgent has terminated unexpectedly

ACSRemoteAgent 04/04/2012 08:30:44 A 0137 1752 0x0 CSLogAgent launched

ACSRemoteAgent 04/04/2012 08:30:54 E 0194 1752 0x0 CSLogAgent has terminated unexpectedly

As already said in the above link, we are seeing this issue in all folders auth, admin.We disabled AV also in that PC where RA server installed.

To retrieve package.cab file do we need downtime..? how much ..?

Please help...

Jatin Katyal · ‎05-10-2013

Hi Prasan,

I reviwed the logs you attached and this is what I see. These errors in the cslog suggesting a connectivity problem

between ACS and RA. Are there any firewalls or nat in the way? I would check any switches in the route too. The tcp retransmissions are not good.

ACSRemoteAgent 05/07/2012 12:21:14 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:21:14 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/07/2012 12:21:14 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:21:14 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 10055

ACSRemoteAgent 05/07/2012 12:21:44 E 0237 1860 0x0 Endpoint Library: EndPoint_GetMessage failed: memory allocation error (wanted -2140274429 bytes)

ACSRemoteAgent 05/07/2012 12:21:44 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:21:44 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 8

ACSRemoteAgent 05/07/2012 12:21:59 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:21:59 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 10055

ACSRemoteAgent 05/07/2012 12:23:29 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:23:29 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/07/2012 12:23:59 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:23:59 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/07/2012 12:24:29 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:24:29 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/07/2012 12:24:29 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:24:29 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 10055

ACSRemoteAgent 05/07/2012 12:25:29 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:25:29 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/07/2012 12:26:02 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:26:02 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/07/2012 12:26:02 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/07/2012 12:26:02 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 10055

ACSRemoteAgent 05/08/2012 03:46:39 A 0245 2532 0x0 RPC: Info request received

ACSRemoteAgent 05/08/2012 03:46:39 A 0290 2532 0x0 RPC: Info reply sent

ACSRemoteAgent 05/08/2012 03:46:58 A 0245 0672 0x0 RPC: Info request received

ACSRemoteAgent 05/08/2012 03:46:58 A 0290 0672 0x0 RPC: Info reply sent

ACSRemoteAgent 05/08/2012 13:26:45 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:26:45 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/08/2012 13:26:45 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:26:45 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 10055

ACSRemoteAgent 05/08/2012 13:27:15 E 0237 1860 0x0 Endpoint Library: EndPoint_GetMessage failed: memory allocation error (wanted -2140274429 bytes)

ACSRemoteAgent 05/08/2012 13:27:15 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:27:15 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 8

ACSRemoteAgent 05/08/2012 13:27:30 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:27:30 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 10055

ACSRemoteAgent 05/08/2012 13:29:00 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:29:00 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/08/2012 13:29:30 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:29:30 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/08/2012 13:30:00 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:30:00 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/08/2012 13:30:00 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:30:00 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 10055

ACSRemoteAgent 05/08/2012 13:31:00 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:31:00 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/08/2012 13:31:30 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:31:30 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 0

ACSRemoteAgent 05/08/2012 13:31:30 E 0237 1860 0x0 Endpoint Library: Failed to get SERVICE_DESIRED message during accept phase

ACSRemoteAgent 05/08/2012 13:31:30 E 0158 1860 0x0 EndPoint_Accept failed, Winsock error 10055

I have found few defects regarding intermittent remote logging issue. However, you're running on the latest code and patch. Both the defects have been fixed in the latest code.

CSCta66819 ACS CSLog service stale threads can cause remote logging failure

CSCta61744 Remote agent reaches max connections limit and does not accept new ones

I'd like to check how did you check the version of remote agent server beacuse csagent -v doesn't tell you what patch is installed. I still suggest you to check RA patch on the server.

If ACS SE and the RA are both the same version and the problem still persists, please reboot the RA and the ACS machines, redefine the RA on the ACS and verify the RA icon (Remote Logging icon and/or Windows Authentication icon) appear and enabled in the ACS UI.

If that doesn't help too than get the below listed info:

- Ip address of ACS SE and RA server

- sniffer traces from RA server

- Package.cab file ( service restart make take few moments majorly depends how many days logs you need to fetch. In our case we need current logs, so don't put number of days. Just check collect log files..should not take more than 5 minutes)

Also, in my humble opinion, we should only keep track on single post to have everything in sequence to avoid any confusion.

Jatin Katyal

- Do rate helpful posts -

~Jatin

Prasan Venky · ‎05-13-2013

Thats very kind of you Mr. Jatin. I will make sure to keep all queries in sequence.

ACS and RA patch i have recently upgraded to latest version. As you said, if i put CSAgent.exe -v, i could see only major version but not the patch level. Any other way to find the exact version..?

While upgrading RA, i have replaced the existing files with the new patch files as per readme guideliness.

Between ACS and RA server we have multiple hops (65XX & Nexus). Is that required to connect RA in the same switch where ACS SE is connected..?

Please help.

Jatin Katyal · ‎05-13-2013

Happy to help!!!

Well, if you have applied it yourself than it should be on same patch level however there is no way to check the patch version via any command. Only a person can answer this question who manage that server. Well, it's not required as long as we've good connectivity. However, it's always good to have it in the same network.

We can test something here if your enviornment allows. setup a windows server with in a same network where we have ACS and configure RA on it. I've no idea how feasible it is for you and your enviornment but yes it's an option.

Else, you need to investigate if there is any connectivity issues between ACS and RA over the network.

Jatin Katyal

- Do rate helpful posts -

~Jatin

Prasan Venky · ‎05-13-2013

how much it is meaningful to suggest customer to go for 5.X series to mitigate this issue...?

As per my understanding, no seperate RA server concept there.Any resolved bug in that version ..?

Jatin Katyal · ‎05-13-2013

Moving to ACS 5.x from ACS 4.x is not an upgrade. It's a complete migration as ACS 5.x runs on a different OS and architecture. We need to plan this because it adds additional cost. It would be new product altogether.

If this is the only reason to have ACS 5.x than I would rather not go with it. Even in ACS 5.x, it's recommended that Primary should handle all the authentication/authorization stuff i.e. decision making node and secondary should be acting as a dedicated log collector and help in deployment in case primary goes down.

At this moment, we can troubleshoot the remote logging issue if you wish.

Please keep in mind that eventually you need to deploy ACS 5.x because ACS 4.x is going eos/eol in oct, 2014.

Few ACS 5.x related doc for your future ref:

Migrating from ACS 4.x to ACS 5.4

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_system/5.4/user/guide/migrate.html

Understanding ACS 5.x policy model

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_system/5.4/user/guide/policy_mod.html

log collector/ logging server

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_system/5.4/installation/guide/csacs_deploy.html#wp1113973

Jatin Katyal

- Do rate helpful posts -

~Jatin

Prasan Venky · ‎05-13-2013

Thanks for your reply. Migrating to 5.X series, involving new app. procurement and all. Currently I would like to resolve it.

Further troubleshooting on this, what all steps to be taken on RA and ACS..?

Anyways extracting package.cab file we will do with maintanance window. Our ACS and RA are in production for past four years. So i think if it takes long time to collect all logs, downtime would be required.

Anything else.

Jatin Katyal · ‎05-14-2013

Sure. Please collect the info suggested in the below mentioned post and let me know when you've all this info.

https://supportforums.cisco.com/message/3934089#3934089

Jatin Katyal

- Do rate helpful posts -

~Jatin

Prasan Venky · ‎05-15-2013

Thanks for your help. I will collect the required logs and come back here soon.

Prasan Venky · ‎05-15-2013

Hi Jatin,

We have found the port specified in ACS for RA server was TCP -2004. But in packet capture from RA, we see only 2005 and 2007 ports from ACS to RA server. Any clue?

Thanks in advance

Jatin Katyal · ‎05-15-2013

There are various ports that ACS and RA uses for their communication. Here is a list and explanation for all.

ConfigProviderPort—The TCP port of the ACS SE that is the configuration provider for the remote agent. The appliance listens to this port for communications from the remote agent. The default is 2003.

Port—The TCP port on which the CSAgent service listens. The default value is 2004. An ACS SE using the remote agent first contacts the agent on this port.

Port—The TCP port that CSWinAgent listens to for ACS SE messages. The default value is 2005.

Port—The TCP port that CSLogAgent listens to for ACS SE messages other than accounting records. The default value is 2006.

AccountingPort—The TCP port that CSLogAgent listens to for accounting records from ACS SEs. The default value is 2007.

-------------------------------------------------------------------------

[CSLogAgent]

; This is the log agent's configuration section...

; Name of the agent's executable:

Executable=..\CSLogAgent\CSLogAgent.exe

; To change the local port:

Port=2006

; To change the accounting port:

AccountingPort=2007

-----------------------------------------------------------------------

so in our case we need to check the cslogagent ports that 2006 and 2007.

Jatin Katyal

- Do rate helpful posts -

~Jatin

Prasan Venky · ‎05-15-2013

Jatin thanks for your beautiful explanation. We just tried applying Extended ACL on the switch where RA server connected (to see hits) for the testing. hits were on only TCP port 2005 and 2007. Rest of the ports were without hits. Below is my CSAgent.ini config.

#####

[CSAgent]

; This is the main service's configuration section...

; This service's communication port

; Port=2004

; The configuration provider hostname/IP address:

; ConfigProviderHost=servername

; or

; ConfigProviderHost=X.X.X.X

ConfigProviderHost=X.X.X.X

; The configuration provider's port:

ConfigProviderPort=2003

; You can restrict which clients can use the agent

; manager using the following syntax:

; PermittedClients=x.x.x.x

; List of agents to activate

; Agent=agent1,agent2,...

Agents=CSLogAgent,CSWinAgent

[CSLogAgent]

; This is the log agent's configuration section...

; Name of the agent's executable

Executable=..\bin\CSLogAgent.exe

; This agent's communication port

Port=2006

; Port for accounting traffic

AccountingPort=2007

; You can restrict which clients can use this agent

; using the following syntax:

; PermittedClients=x.x.x.x

[CSWinAgent]

; This is the Windows agent's configuration section...

; Name of the agent's executable

Executable=..\bin\CSWinAgent.exe

; This agent's communication port

Port=2005

####

If hits are not there for the rest of the port, that means the respective logs are not receiving by RA..?

Please suggest.And also if you could provide me a link where AV scanning is to be avoided on RA server, would be grateful.

Thanks in advance

Jatin Katyal · ‎05-15-2013

What AV are you using? There is not much you need to perform. Just click on AV console or settings and look for tab called exclusion, clcik on it and browse for C:\Program Files\Cisco\CiscoSecure ACS Agent, add it in the exclusion list and save the changes.

Jatin Katyal
- Do rate helpful posts -

~Jatin

Prasan Venky · ‎05-15-2013

Thanks. But it would be better if you give the cisco link or document which recommends to exclude. I can provide the same to customer too And also my CSAgent.ini is proper ..?

Jatin Katyal · ‎05-15-2013

I have found one for ACS 4.2 windows but not for RA agent. That step is part of the troubleshooting to see if that is causing any issues. Please visit the below suggested link that says : When you configure antivirus (AV) software and Sybase with ACS, do not include the database file for monitoring by the AV software.

http://www.cisco.com/en/US/docs/net_mgmt/cisco_secure_access_control_server_for_windows/4.2/trouble/guide/Ch1.html#wp1048833

Jatin Katyal
- Do rate helpful posts -

~Jatin