Adding contact from intercluster CUPC server failed

ilicveselin · ‎07-03-2011

Hello everybody,

I am integrating two CUP 8.0 servers via interclustering peers. Both of these two servers works OK with their local CUPC users. Addition of users or IM between users occurs normal within same CUP server. I was able to add Intercluster peers without any errors (on intercluster peer page and on system troubleshooter page al checkmarks are green).

But, when I try to add user from other CUP server to contact list it appears for one second with status "waiting for confirmation" and disapear from list.

I checked following on both servers:

- Services: All network services are running, Feature services running: AXL Web, SIP Proxy, Presence Engine, Sync Agent, XCP Conference Manager, XCP Connection manager, XCP Counter Agregator, XCP Authentication Service.

- Proxy domain configured (same domain on both servers)

- Incoming ACL, Outgoing ACL

- capabilities assignement on CUCM servers

- No duplicate users between CUP servers

- Tried to restart server, some services without success

Christos Georgiadis · ‎07-04-2011

Hi,

To see what is going on there, detailed XCP Router traces should be checked from both CUP nodes

You should be able to see the subscribe request sent from one CUP node via the intercluster trunk to the other side

My guess is that the subscribe request reaches the other node (that needs to be checked though) but it is not ack'ed aand instead you get an unsubscribe back (hence the contact disappears)

Also have a look at this defect and its workaround

CSCtj82283 - Intercluster connection is not available on CUP server

Workaround

----------------------------------------

Run the following CLI command on the publisher of each cluster to verify if isrftrunkmeshpoint is set correctly:

admin:run sql select isrftrunkmeshpoint from processnode where systemnode='f'

If the return result is 'f', then it needs to be corrected using the following command:

admin:run sql update processnode set isrftrunkmeshpoint='t' where systemnode='f'

-----------------------------

HTH,

Christos

ilicveselin · ‎07-05-2011

Hi Christos, thanks for answer. I run this command from CLI and result is 't'.

Also, I collected trace from CUP-1 and CUP-2 servers. In this trace example user from CUP-1 is trying to add user from CUP-2. It seems that CUP-1 not sending subscribe to CUP-2...

From intercluster peer and in system troubleshooter all checkmarks are green. Version of CUP is 8.0.3.10000(4).

Christos Georgiadis · ‎07-05-2011

Hi,

I am not sure if those traces are set to detailed. Especially for CUP-2 there is nothing logged.

I do see some subscribe messages in the logs for CUP-1 (watch out this is not SIP subscribe but it is a subscribe from the jabber protocol)

11:46:27.791 |DELIVER: type:1, host:pod.com, packet:from='marija@pod.com' id='iWtK9-40' to='veselin@pod.com' type='subscribe'/>

There is a redirect to CUP- 2

11:46:28.295 |sdns_plugin redirecting to: jsm-1.cup-2

11:46:28.315 |bouncing a packet to 'jsm-1.cup-2' from 'sdns_plugin-1.cup-1'

and there there is an error coming back from cup-2

from='marija@pod.com' id='iWtK9-40' to='veselin@pod.com' type='subscribe'/><error type='cancel' code='404'> xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/>

Contact is being removed from the roster

11:46:30.119 |_mod_auto_subscribe_send_unsubscribe() from = marija@pod.com; to = veselin@pod.com

11:46:30.964 |Removing user 'veselin@pod.com' from 'marija@pod.com's roster cache.

Couldn't check CUP-2 traces. Maybe there is more info there.

Maybe you can also check the cupsuserlocation table on both servers for those users and compare

run sql select * from cupsenduserlocation where userid='marija'

run sql select * from cupsenduserlocation where userid='veselin'

This table is populated by the intercluster sync agent (ICSA) service

HTH,

Christos

ilicveselin · ‎07-06-2011

Hi Christos,

thanks for answer. To make sure that traces are set to detailed, I performed addition of user in both directions:

- In 11h:04m on CUP-2 server user marija@pod.com tried to add user user veselin@pod.com located at CUP-1 server

- After 3 minutes (11h:07m) user veselin@pod.com on CUP-1 server tried to add marija@pod.com on CUP-2 server

We see that no traces are logged on CUP-1 at 11:04 and on CUP-2 on 11:07.

Also, command

un sql select * from cupsenduserlocation where userid='user'

gives same output on both servers for both users:

The specified table (cupsenduserlocation) is not in the database.

Also, I tried already to restart intercluster agent (multiple times).

Christos Georgiadis · ‎07-06-2011

Hi,

I checked the traces again and indeed although I see a request being generated on CUP-2 at 11:04 to add user veselin to the contact list of user marija I don't see anything generated in the traces on CUP-1. It seems that the request never reaches the CUP-1 and vice versa.

Looks like the XCP Routers from both the servers don't talk with each other

Can we make sure that we meet the pre-requisites as per the deployment guide just in case ?

-----------------------------------------------

•The intercluster peers must each integrate with a different Cisco Unified Communications Manager cluster.

•You must complete the required multi-node configuration in both the home Cisco Unified Presence cluster, and in the remote Cisco Unified Presence cluster:

–Configure the system topology and assign your users as required.

–Activate the services on each Cisco Unified Presence node in the cluster.

•You must turn on the AXL interface on the local Cisco Unified Presence publisher node, and on the remote Cisco Unified Presence publisher node. Cisco Unified Presence creates, by default, an intercluster application user with AXL permissions. To configure an intercluster peer, you will require the username and password for the intercluster application user on the remote Cisco Unified Presence server.

•You must turn on the Sync Agent on the local Cisco Unified Presence publisher node, and on the remote Cisco Unified Presence publisher node. Allow the Sync Agent to complete the user sychronization from Cisco Unified Communications Manager before you configure the intercluster peers.

---------------------------------------------------------------

As for the sql queries apologies I made a typo

it should be cupsuserlocation and NOT cupsenduserlocation

run sql select * from cupsuserlocation where my_lower(userid)='veselin' ---> on CUP-2

do the same for marija on CUP-1

Regards,

Christos

Christos Georgiadis · ‎07-06-2011

On top of the above suggestions, maybe you can grab the xcp router logs after you restart both of them and see if can establish a connection between themselves in the first place.

HTH,

Christos

ilicveselin · ‎07-07-2011

Hi Christos,

thansk for answer, and sorry for late reply. I followed this document:

http://www.cisco.com/en/US/docs/voice_ip_comm/cups/8_0/english/install_upgrade/deployment/guide/dgintercluster.pdf

•The intercluster peers must each integrate with a different Cisco Unified Communications Manager cluster.

A: There are two separate CUCM's, CUP-1 is integrated with CUCM-1, and CUP-2 is integrated with CUCM-2

•You must complete the required multi-node configuration in both the home Cisco Unified Presence cluster, and in the remote Cisco Unified Presence cluster:

–Configure the system topology and assign your users as required.

–Activate the services on each Cisco Unified Presence node in the cluster.

A: For both CUP servers(configuration is same and symetrical except ip addresses and users): All network services are up and running. These feature serveces are started and running:

Cisco AXL Web Service

Cisco UP SIP Proxy

Cisco UP Presence Engine

Cisco UP Sync Agent

Cisco UP XCP Text Conference Manager

Cisco UP XCP Connection Manager

Cisco UP XCP Counter Aggregator

Cisco UP XCP Authentication Service

Also, proxy domain is configured, integration with CUCM's is accomplished via SIP trunk, users are assigned CUP and CUPC capabilities. Incoming and outgoing ACLs are permiting both CUP-1 and CUP-2 on both servers.

•You must turn on the AXL interface on the local Cisco Unified Presence publisher node, and on the remote Cisco Unified Presence publisher node. Cisco Unified Presence creates, by default, an intercluster application user with AXL permissions. To configure an intercluster peer, you will require the username and password for the intercluster application user on the remote Cisco Unified Presence server.

A: AXL service is up on CUCM and CUP servers. One user has AXL API rights and intercluster peer rights, and that user is configured as AXL user at intercluster peer configuration page.

•You must turn on the Sync Agent on the local Cisco Unified Presence publisher node, and on the remote Cisco Unified Presence publisher node. Allow the Sync Agent to complete the user sychronization from Cisco Unified Communications Manager before you configure the intercluster peers.

A: Sync agent is up and running, i tried to restart it on both servers earlier, without success.

Output on CLI:

CUP-1:

admin:run sql select * from cupsuserlocation where my_lower(userid)='marija'

userid firstname lastname srvname node1name node1port node1transport node2name peerid peertag islocal mailid jsmid

================================================================================================================================ ================================================== ================================================== ======= ================================================================================================================================ ========= ============== ========= ==================================== ============= ======= ====== ================================================================================================================================

marija Marija Ilic NULL CUP-2 5060 1 NULL 9bde012d-9c01-fb2c-601a-956e5a36d0d1 1310042883895 f NULL jsm-1.CUP-2

admin:

CUP-2:

admin:run sql select * from cupsuserlocation where my_lower(userid)='veselin'

userid firstname lastname srvname node1name node1port node1transport node2name peerid peertag islocal mailid jsmid

================================================================================================================================ ================================================== ================================================== ======= ================================================================================================================================ ========= ============== ========= ==================================== ============= ======= ====== ================================================================================================================================

veselin Veselin Ilic NULL CUP-1 5060 1 NULL a4bc796b-872d-b21d-4f89-15324484c6b0 1310042861840 f NULL jsm-1.CUP-1

admin:

Tracing results details in attachment: XCP Router service is restarted on CUP-1 at 17:28 and at CUP-2 at 17:35.

BR,

Veselin

Christos Georgiadis · ‎07-08-2011

Hi,

In the XCP Router logs you provided I don't see the routers trying to talk to each other.

I only see in CUP-2 traces the following

17:35:42.214 |Got Buffer (447) to='marija@pod.com/cucsf27757' type='session'>marija@pod.com/cucsf2775710.2.2.11187010.2.1.45222

The above is the only indication that there is a peer but I am pretty sure there should be more than that.

I would try to recreate the intercluster trunk again but I suppose it might be a good idea to open a service request to get this investigated in detail. Something is not right in the way the intercluster trunk is set up as the routers don't appear to be communicating properly.

This is a must, to get the interclustering working

Christos

PS: There is a link to open a SR from within the forums and the whole discussion will be transfered to the SR automatically.

Anyway you can point to this discussion to avoid effort duplication.

ilicveselin · ‎07-12-2011

Hi christos, thansk for your effort.

I have checked once again all configuration, and havent found error.

Recreated intercluster peers on both servers, without success...

Get some intercluster sync agent and XCP router traces again, and cant found what is the cause.

I will check my options about opening SR.

osama_abdo · ‎12-13-2011

Hi ,

Have you solved this issue cuz i'm facing this one on 8.5.2 but the users on the publisher server can' t login or been add to the subscriber server with the same behaviour you have seen .

Many Thanks if you could help me.

DWAM_2 · ‎02-08-2012

Hello,

I match too the same behaviour, same version.

Have you solved the problem ?

Best regards.

osama_abdo · ‎02-08-2012

Plz find TAC reply below:

As discussed, the reason why you were not able to add contacts on the CUPS Buddy list was that both the CUPS servers in your CUPS cluster were on both different VLAN’s and you didn’t have the Multicast DNS configured on the Network level, so if you have servers in different VLAN’s, then you got to make sure that the MDNS is enabled and you wouldn’t face this issue and this is a network level change so what we did was instead of this we changed the setting from MDNS to Router-Router so that we have the Contacts adding between the two XCP Router services of the CUPS server and that fixed out issue.

DWAM_2 · ‎02-08-2012

Thank you for your answer

but we match the behaviour and our presence servers are in the same vlan.

Best regards.

Networks LSE · ‎06-28-2012

Hi,

Just set up an intercluster trunk between two clusters running 8.5.4 and 8.6.1, and seeing something similar. Also can not IM, or view presence status of remote users.

Routing comms type is router-to-router

Anyone had anything similar, and ideas where to look?

Thanks & Regards

Nic