Unable to login to Finesse Desktop on UCCX 10.5

[Gordon Ross]

I've built a new UCCX system, V10.5(1)SU1 and activated finesse from the CLI (utils uccx finesse activate)

I can go to the finesse administration screen fine (https://server:8445/cfadmin)

But when I login to the desktop page (https://server:8445/desktop) I  get the message "This Finesse Server is out of service."

If I go to cfadmin -> Team Resources, I get "The Finesse server is currently Out of Service".

Yet UCCX Serviceability shows Cisco Finesse Tomcat as IN SERVICE

Google hasn't been much help. I seem to get the answer that the CTI settings in Finesse administration are wrong - but in UCCX there is no settings page in Finesse administration. (I think this only applies to Contact Center Enterprise rather than Express)

Any suggestions as to where to start to troubleshoot/fix this?

Thanks,

GTG

[Gergely Szabo]

Hi,

can you check for the status of the Cluster View Daemon?

Did you stop the CAD services?

Did you upload all necessary licenses?

G.

[Gordon Ross]

Can you check for the status of the Cluster View Daemon?

CCX Serviceability is showing it as IN SERVICE.

Did you stop the CAD services?

Not explicitly. I assumed this was all stopped when "utils uccx finesse active" was completed. Also, I can't see anything obvious in CCX Serviceability either.

Did you upload all necessary licenses?

Yep. In appadmin -> System -> License Information -> Display Licenses, I'm seeing:

"Package: Cisco Unified CCX Premium"

GTG

[Abu Hadee]

Did you reboot the nodes after activating Finesse? If you have already rebooted, try following

1. Check which node is master if you have HA.

2. Open the following https://<IP of Master node>:8445/finesse/api/SystemInfo

If you out of service, you may need to restart Finesse Tomcat Service. utils service restart Cisco Finesse Tomcat

Once the Tomcat service is restarted, try login in again. The Service Status in Serviceability page is misleading. That is just check if Tomcat service is running or not. But if Tomcat is running but Finesse is not able to talk to UCCX, then it will remain as out of service.

Let me know how it goes.

Abu

[Gordon Ross]

Check which node is master if you have HA.

uccx-a01 - the one I'm accessing via the web browser

Open the following https://<IP of Master node>:8445/finesse/api/SystemInfo

<SystemInfo>

 <currentTimestamp>2014-10-21T12:16:03.593Z</currentTimestamp>

 <deploymentType>UCCX</deploymentType>

 <primaryNode>

  <host>uccx-a01.example.com</host>

 </primaryNode>

 <secondaryNode>

  <host>uccx-b01.example.com</host>

 </secondaryNode>

 <status>OUT_OF_SERVICE</status>

 <timezoneOffset>60</timezoneOffset>

 <uri>/finesse/api/SystemInfo</uri>

 <xmppDomain>uccx-a01.example.com</xmppDomain>

 <xmppPubSubDomain>pubsub.uccx-a01.example.com</xmppPubSubDomain>

</SystemInfo>

If you out of service, you may need to restart Finesse Tomcat Service. utils service restart Cisco Finesse Tomcat

Done that - plus a full reboot too. Still no joy :-(

Thanks,

GTG

[Abu Hadee]

Can you check the same info on the secondary UCCX Server as well? Also, I see the name as uccx-a01.example.com, I know Finesse does auto populate for uccx, but are these valid FQDN? The Finesse will try to setup CTI connection to these addresses. If the secondary is also shows out of service, the next step for you to look at finesse logs to see if it is making any attempt to establish CTI connection to UCCX.

[Gordon Ross]

Checked the secondary, and that's the same too. The FQDN was changed to protect the guilty ;-)

Finesse/logs/catalina.out has various complaints in it:

"Oct 20, 2014 3:26:50 PM org.apache.shindig.gadgets.AbstractSpecFactory$SpecUpdater run
INFO: Failed to update undefined. Using cached version.
ResourceBundleTag.doTag(), !!!ERROR: could not retrieve bundle from rbMgr for basename=Messages, locale=, sessionId=E4657818552217EBC580B2DC8F75FDA0 on page=finesse-desktop/src/main/webapp/modules/header.jsp
ResourceBundleTag.doTag(), !!!ERROR: could not retrieve bundle from rbMgr for basename=Messages, locale=, sessionId=E4657818552217EBC580B2DC8F75FDA0 on page=phoenix-gadget-container/src/main/webapp/container/body.jsp
ResourceBundleTag.doTag(), !!!ERROR: could not retrieve bundle from rbMgr for basename=Messages, locale=, sessionId=E4657818552217EBC580B2DC8F75FDA0 on page=finesse-desktop/src/main/webapp/modules/footer.jsp
Oct 20, 2014 3:31:59 PM org.apache.shindig.gadgets.AbstractSpecFactory$SpecUpdater run
INFO: Failed to update undefined. Using cached version.
a

...

Oct 20, 2014 3:47:53 PM com.sun.jersey.spi.container.ContainerResponse traceException
SEVERE: Mapped exception to response: 503 (Service Unavailable)
com.cisco.ccbu.finesse.api.exception.ServerUnavailableException

...

Oct 20, 2014 3:56:18 PM org.apache.shindig.gadgets.render.DefaultServiceFetcher retrieveServices
SEVERE: HTTP Error 404 fetching service methods from endpoint https://uccx-a01.example.com:8445/rpc

"

Generally, it's not happy :-(

GTG

[Abu Hadee]

The above errors are for the gadget related. Not when Finesse is trying to establish the connection to CTI. Can you upload the complete log file with timestamps when you restarted? UCCX MIVR logs need to be checked as well if the services are up and running. If you can reboot the server again, wait for couple of minute to make sure all the services are coming up (at least in Serviceability page). Then collect both UCCX and Finesse logs.

[Gordon Ross]

I restarted Finesse, and took a look at the logs. Last time I looked, I think I missed some of the logs, as I found this in desktop\logs\webservices\Error*

"%CCBU_pool-11-thread-1-3-CTI_OPEN_FAILURE: %[ERROR_DESCRIPTION=Open Req returned failure][error_code=92][error_message=CSInboundMsgProcessor: The server is standby, not Master]: CTI Server rejected open request ; There is a configuration mismatch between this server and the CTI server "

Now here, I'm little confused. Exactly what is complaining? Is it Finesse not being able to talk to another part of UCCX, or UCCX not being able to talk to CUCM? The CTI Ports in CCMadmin all show as registered.

GTG

[Abu Hadee]

This may be expected to be seen on the standby as it will not accept the CTI connection from the finesse. I think we are going in Circle. Lets us concentrate only one server. Can you shutdown secondary server. Then reboot or restart services of the primary and check the status and the logs on the primary?

[Gordon Ross]

Secondary shut down.
Primary rebooted.

Desktop-webservices log file is about the same as before:

0000000280: 172.18.0.12: Oct 22 2014 11:25:40.494 +0100: %CCBU_Log Purge Timer-6-LOG_PURGER_EVENT: %[event_info=file name: Desktop-webservices.2014-10-22T10-12-07.748.startup.log.zip, file length: 130751][event_name=Zipped log file]: Log Purger Event
0000000281: 172.18.0.12: Oct 22 2014 11:25:40.537 +0100: %CCBU_Log Purge Timer-6-LOG_PURGER_EVENT: %[event_info=file name: Error-Desktop-webservices.2014-10-22T10-12-07.986.startup.log.zip, file length: 12472][event_name=Zipped log file]: Log Purger Event
0000000024: 172.18.0.12: Oct 22 2014 11:27:21.954 +0100: %CCBU_pool-11-thread-1-3-CTI_OPEN_FAILURE: %[ERROR_DESCRIPTION=Peripheral is offline][error_code=-1][error_message=Peripheral is offline]: CTI Server rejected open request ; There is a configuration mismatch between this server and the CTI server
0000000025: 172.18.0.12: Oct 22 2014 11:27:21.955 +0100: %CCBU_Reader-6-COMPONENT_STATE_CHANGE: %[component_name=CTI_ADAPTER][component_new_state=DISCONNECTED][component_old_state=CONNECTING_1]: Component's state has changed
0000000026: 172.18.0.12: Oct 22 2014 11:27:21.958 +0100: %CCBU_Reader-6-COMPONENT_STOP: %[component_name=CTI_WRITER]: Component is stopping
0000000003: 172.18.0.12: Oct 22 2014 11:27:21.958 +0100: %CCBU_Reader-6-STATE_CHANGE: %[current_state=Out of Service][previous_state=Starting][reason_string=Dependency not at full service][ssid=AWS1][sstype=AWS]: Subsystem state changed.
0000000282: 172.18.0.12: Oct 22 2014 11:27:21.958 +0100: %CCBU_Reader-6-STATE_CHANGE_NOTIFICATION: %[current_state=STATE_OUT_OF_SERVICE][listener_id=putting state change request on queue for processing: listeners are [com.cisco.ccbu.finesse.subsystem.AWSSubsystem@1a6237]][previous_state=STATE_STARTING][state_change_id=AWS1]: Sending a state change notification to a state change listener
0000000283: 172.18.0.12: Oct 22 2014 11:27:21.959 +0100: %CCBU_Reader-6-SUBSYSTEM_STATE_CHECK: %[current_state=STATE_OUT_OF_SERVICE][ssid=AWS1]: Subsystem not in service during readiness check
0000000284: 172.18.0.12: Oct 22 2014 11:27:21.960 +0100: %CCBU_SubsystemStateChange Listener Notifier-6-STATE_CHANGE_NOTIFICATION: %[current_state=STATE_OUT_OF_SERVICE][listener_id=AWSSubsystem][previous_state=STATE_STARTING][state_change_id=AWS1]: Sending a state change notification to a state change listener
0000000285: 172.18.0.12: Oct 22 2014 11:27:21.961 +0100: %CCBU_SubsystemStateChange Listener Notifier-6-STATE_CHANGE_NOTIFICATION: %[current_state=null][listener_id=Waiting 30 seconds for 1 state changes to complete.][previous_state=null][state_change_id=AWS1]: Sending a state change notification to a state change listener
0000000027: 172.18.0.12: Oct 22 2014 11:27:21.964 +0100: %CCBU_Reader-6-COMPONENT_STATE_CHANGE: %[component_name=CTI_ADAPTER][component_new_state=CONNECTING_1][component_old_state=DISCONNECTED]: Component's state has changed
0000000286: 172.18.0.12: Oct 22 2014 11:27:21.968 +0100: %CCBU_SubsystemStateChange Listener Notifier-6-STATE_CHANGE_NOTIFICATION: %[current_state=null][listener_id=Done waiting for state change notification to complete][previous_state=null][state_change_id=AWS1]: Sending a state change notification to a state change listener
0000000028: 172.18.0.12: Oct 22 2014 11:27:22.025 +0100: %CCBU_Reader-6-QUEUE STATISTICS POLLING STOPPED:  Stopped polling for queue statistics
0000000029: 172.18.0.12: Oct 22 2014 11:27:22.025 +0100: %CCBU_Reader-6-AGENT_CACHE_CLEARED:  Agent cache has been cleared
0000000030: 172.18.0.12: Oct 22 2014 11:27:22.025 +0100: %CCBU_Reader-6-CALL_CACHE_CLEARED:  Call cache has been cleared
0000000031: 172.18.0.12: Oct 22 2014 11:27:22.025 +0100: %CCBU_Reader-6-TEAM_CACHE_CLEARED:  Team cache has been cleared
0000000032: 172.18.0.12: Oct 22 2014 11:27:22.026 +0100: %CCBU_Reader-3-INTERNAL_ADAPTER_ERROR: %[ERROR_DESCRIPTION=0][exception=***************Socket Error while reading Message from CTI Reader ***************][noname=Connection reset]: Generic Internal Adapter Error
0000000033: 172.18.0.12: Oct 22 2014 11:27:22.027 +0100: %CCBU_pool-11-thread-1-4-CTI_CONNECTION_RETRYING:  System is trying to connect to CTI server ; Please wait as the system retries
0000000034: 172.18.0.12: Oct 22 2014 11:27:22.029 +0100: %CCBU_pool-11-thread-1-6-CONNECT_TO_RESOURCE: %[resource_name=CTI_SERVER][noname=localhost][noname=12028]: Connecting to resource
0000000035: 172.18.0.12: Oct 22 2014 11:27:22.031 +0100: %CCBU_pool-11-thread-1-6-COMPONENT_INIT: %[component_name=CTI_WRITER]: Component is being initialized
0000000036: 172.18.0.12: Oct 22 2014 11:27:22.032 +0100: %CCBU_pool-11-thread-1-6-MESSAGE_TO_CTI_SERVER: %[cti_message=Invoke id :2 , perpheralId: 0, versionNumber : 16, idleTimeout : 120, servicesRequested: 0x94019e, callMsgMask: 0x6fffff, configMsgMask: 0xf, clientId: Finesse, clientPass: passwd, clientSignature: null, agentExtension: null, agentId: null, agentInstrument: null][cti_message_name=OpenReq]: Message going to the backend cti server
0000000037: 172.18.0.12: Oct 22 2014 11:27:22.036 +0100: %CCBU_CTIMessageEventExecutor-0-6-MESSAGE_FROM_CTI_SERVER: %[cti_message_name=OpenConf][cti_sequence_id=2]: Message coming to Finesse from backend cti server
0000000038: 172.18.0.12: Oct 22 2014 11:27:22.036 +0100: %CCBU_pool-14-thread-2-6-MESSAGE_FROM_CTI_SERVER: %[cti_message_name=OpenConf][cti_sequence_id=0]: Message coming to Finesse from backend cti server
0000000039: 172.18.0.12: Oct 22 2014 11:27:22.042 +0100: %CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=CTIOpenConf [servicesGranted=9699742, peripheralOnline=false, agentState=9, agentExtension=null, agentID=null, agentInstrument=null, agentID_Long=null, numPeripherals=1, peripheralIDList=[0], multilineAgentControlList=[1]]CTIMessageBean [invokeID=2, msgID=4, timeTracker={"id":"OpenConf","CTI_MSG_NOTIFIED":1413973642042,"CTI_MSG_RECEIVED":1413973642037}, msgName=OpenConf, deploymentType=CCX]][cti_response_time=5]: Decoded Message to Finesse from backend cti server
0000000040: 172.18.0.12: Oct 22 2014 11:29:52.045 +0100: %CCBU_pool-11-thread-1-3-CTI_OPEN_FAILURE: %[ERROR_DESCRIPTION=Peripheral is offline][error_code=-1][error_message=Peripheral is offline]: CTI Server rejected open request ; There is a configuration mismatch between this server and the CTI server
0000000041: 172.18.0.12: Oct 22 2014 11:29:52.045 +0100: %CCBU_Reader-6-COMPONENT_STATE_CHANGE: %[component_name=CTI_ADAPTER][component_new_state=DISCONNECTED][component_old_state=CONNECTING_1]: Component's state has changed
0000000042: 172.18.0.12: Oct 22 2014 11:29:52.047 +0100: %CCBU_Reader-6-COMPONENT_STOP: %[component_name=CTI_WRITER]: Component is stopping
0000000004: 172.18.0.12: Oct 22 2014 11:29:52.047 +0100: %CCBU_Reader-6-STATE_CHANGE: %[current_state=Out of Service][previous_state=Out of Service][reason_string=Dependency not at full service][ssid=AWS1][sstype=AWS]: Subsystem state changed.
0000000043: 172.18.0.12: Oct 22 2014 11:29:52.047 +0100: %CCBU_Reader-6-COMPONENT_STATE_CHANGE: %[component_name=CTI_ADAPTER][component_new_state=CONNECTING_1][component_old_state=DISCONNECTED]: Component's state has changed
0000000044: 172.18.0.12: Oct 22 2014 11:29:52.047 +0100: %CCBU_Reader-6-QUEUE STATISTICS POLLING STOPPED:  Stopped polling for queue statistics
0000000045: 172.18.0.12: Oct 22 2014 11:29:52.047 +0100: %CCBU_Reader-6-AGENT_CACHE_CLEARED:  Agent cache has been cleared
0000000046: 172.18.0.12: Oct 22 2014 11:29:52.048 +0100: %CCBU_Reader-6-CALL_CACHE_CLEARED:  Call cache has been cleared
0000000047: 172.18.0.12: Oct 22 2014 11:29:52.048 +0100: %CCBU_Reader-6-TEAM_CACHE_CLEARED:  Team cache has been cleared

And Error-Desktop-webservice is the same too:

0000000204: 172.18.0.12: Oct 22 2014 11:24:40.587 +0100: %CCBU_Infrastructure_SS_com.cisco.ccbu.finesse.subsystem.DBSubsystem_boot_thread-3-SERVER_NOT_READY: %[reason_string=subsystems not ready]: Server State is NOT_READY. ; This is caused by some other condition which caused the server to be unable to continue processing.  Check the rest of the log for other associated errors.  
0000000024: 172.18.0.12: Oct 22 2014 11:27:21.954 +0100: %CCBU_pool-11-thread-1-3-CTI_OPEN_FAILURE: %[ERROR_DESCRIPTION=Peripheral is offline][error_code=-1][error_message=Peripheral is offline]: CTI Server rejected open request ; There is a configuration mismatch between this server and the CTI server
0000000032: 172.18.0.12: Oct 22 2014 11:27:22.026 +0100: %CCBU_Reader-3-INTERNAL_ADAPTER_ERROR: %[ERROR_DESCRIPTION=0][exception=***************Socket Error while reading Message from CTI Reader ***************][noname=Connection reset]: Generic Internal Adapter Error
0000000033: 172.18.0.12: Oct 22 2014 11:27:22.027 +0100: %CCBU_pool-11-thread-1-4-CTI_CONNECTION_RETRYING:  System is trying to connect to CTI server ; Please wait as the system retries

GTG

[timhughsmith]

Hey mate,

Just had this reported for one of my installs that has been working on Finesse happily for a couple of months now.

Similar, but we had Finesse activated with no issues.

I'm going to log a TAC for mine. I'll let you know the outcome.

EDIT - Actually wondering if mine is just because it's failed over.

Cheers,

Tim

[jasonlnielsen]

This should be working as designed from what I am aware of. I was on the phone with Cisco TAC yesterday and showed them what has happening via a WebEx.

Because my HA server was M(Master) and my Pub was S(Standby), the engineer said that was correct.

You could restart your UCCX Engine on your HA to fail it back to your Primary server and then log in again.

It is misleading to an end user when you log in to a "bookmarked" URL. It will redirect you to the Master server for login and state that it is Out of service. An agent does not know "which" server to log into for the Master to not experience this issue. If you were to log into the Active Master server you would not see that error and be redirected because you are on the Master server.

I am definitely interested in the answer that you get from Cisco TAC as I was told this is working as designed.

Unless this is documented somewhere that you can log into either server for Finesse, no matter which server is Master, I would like to see that.

This is my Primary Server:

<SystemInfo>
<currentTimestamp>2014-10-29T13:31:54.642Z</currentTimestamp><deploymentType>UCCX</deploymentType>
<primaryNode>
<host>xx.xxxx.com</host></primaryNode>
<secondaryNode>
<host>xx.xxxx.com</host>
</secondaryNode><status>OUT_OF_SERVICE</status>
<timezoneOffset>-300</timezoneOffset>
<uri>/finesse/api/SystemInfo</uri>
<xmppDomain>xx.xxxx.com</xmppDomain><xmppPubSubDomain>pubsub.xx.xxxx.com</xmppPubSubDomain></SystemInfo>

This is the Secondary Node(HA)

<SystemInfo>
<currentTimestamp>2014-10-29T13:34:21.754Z</currentTimestamp><deploymentType>UCCX</deploymentType>
<primaryNode>
<host>xx.xxxx.com</host></primaryNode>
<secondaryNode>
<host>xx.xxxx.com</host></secondaryNode>
<status>IN_SERVICE</status>
<timezoneOffset>-300</timezoneOffset>
<uri>/finesse/api/SystemInfo</uri>
<xmppDomain>xx.xxxx.com</xmppDomain><xmppPubSubDomain>pubsub.xx.xxxx.com</xmppPubSubDomain></SystemInfo>

Please rate if helpful! Thank you.

[Ian Freely - Shared Account]

Hey mate,

Same answer as yours, I also agree it's working as designed. This is different to OP's original issue.

It threw me because some of the live reporting was out, and we hadn't been alerted to the failover, so I thought it was something bigger.

Cheers,

Tim

[m-lehmann]

Home run.  This is exactly what my problem was.  By exactly I mean...

Because my HA server was M(Master) and my Pub was S(Standby)

and

It is misleading to an end user when you log in to a "bookmarked" URL. It will redirect you to the Master server for login and state that it is Out of service. An agent does not know "which" server to log into for the Master to not experience this issue. If you were to log into the Active Master server you would not see that error and be redirected because you are on the Master server.