Services Missing After UCCX SU3 Upgrade

[ejlackey]

Hello,
I used the following guide to upgrade UCCX from 12.5(1) SU1 to 12.5(1) SU3.

https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/crs/express_12_5_1_su3/install/guide/uccx_b_1251su3_install-and-upgrade-guide/uccx_m_1251su2_unified-ccx-upgrade.html

Installed these cop files and upgraded with this image.
ucos.keymanagement.v02.cop.sgn
ciscouccx.1251.SU1.1251SU3PREUPGRADE.59.cop.sgn
UCSInstall_UCCX_12_5_1_UCOS_12.5.1.11003-511.sgn.iso

After the upgrade was completed, not all of the services are activated, notably the CCX Engine and Notification Services.
We have Smart Licensing.
Database replication was fine, no corruption.
I have the install logs but not sure which one(s) are pertinent.

admin:utils service list

Requesting service status, please wait...
System SSH [STARTED]
Cluster Manager [STARTED]
Name Service Cache [STARTED]
Entropy Monitoring Daemon [STARTED]
Cisco SCSI Watchdog [STARTED]
Service Manager [STARTED]
Service Manager is running
Getting list of all services
A Cisco DB[STARTED]
A Cisco DB Replicator[STARTED]
Cisco AMC Service[STARTED]
Cisco Audit Event Service[STARTED]
Cisco CDP[STARTED]
Cisco CDP Agent[STARTED]
Cisco CallManager Serviceability[STOPPED] Component is not running
Cisco Certificate Change Notification[STARTED]
Cisco Certificate Expiry Monitor[STARTED]
Cisco DRF Local[STARTED]
Cisco DRF Master[STARTED]
Cisco Database Layer Monitor[STARTED]
Cisco Finesse Tomcat[STARTING]
Cisco Identity Service[STARTING]
Cisco Log Partition Monitoring Tool[STARTED]
Cisco RIS Data Collector[STARTED]
Cisco RTMT Reporter Servlet[STOPPED] Component is not running
Cisco Serviceability Reporter[STARTED]
Cisco Syslog Agent[STARTED]
Cisco Tomcat[STARTED]
Cisco Tomcat Stats Servlet[STOPPED] Component is not running
Cisco Trace Collection Service[STARTED]
Cisco Trace Collection Servlet[STOPPED] Component is not running
Cisco Unified CCX Administration[STOPPED] Component is not running
Cisco Unified CCX CVD Dependent Webapp[STOPPED] Component is not running
Cisco Unified CCX Cluster View Daemon[STARTED]
Cisco Unified CCX Configuration API[STOPPED] Component is not running
Cisco Unified CCX DB Perfmon Counter Service[STARTED]
Cisco Unified CCX Database[STARTED]
Cisco Unified CCX Perfmon Counter Service[STARTED]
Cisco Unified CCX SNMP Java Adapter[STARTED]
Cisco Unified CCX Serviceability[STOPPED] Component is not running
Cisco Unified CCX Voice Subagent[STARTED]
Cisco Unified CCX WebServices[STOPPED] Component is not running
Cisco Unified Intelligence Center Reporting Service[STARTING]
Cisco Unified Serviceability RTMT[STOPPED] Component is not running
Cisco Web Proxy Service[STARTING]
Host Resources Agent[STARTED]
MIB2 Agent[STARTED]
Platform Administrative Web Service[STOPPED] Component is not running
Platform Communication Web Service[STOPPED] Component is not running
SNMP Master Agent[STARTED]
SOAP -Log Collection APIs[STOPPED] Component is not running
SOAP -Performance Monitoring APIs[STOPPED] Component is not running
SOAP -Real-Time Service APIs[STOPPED] Component is not running
System Application Agent[STARTING]
Cisco Cloud Connect Container Manager[STOPPED] Service Not Activated
Cisco DirSync[STOPPED] Service Not Activated
Cisco Unified CCX Engine[STOPPED] Service Not Activated
Cisco Unified CCX Notification Service[STOPPED] Service Not Activated
Cisco Unified CCX Socket.IO Service[STOPPED] Service Not Activated
Cisco Unified Intelligence Center Serviceability Service[STOPPED] Service Not Activated
Docker Engine[STOPPED] Service Not Activated
Primary Node =true

[david.macias]

I’ve had a few customers upgrade and none had seen this before. I would go straight to TAC for help. When you get it solved please provide the answer here. I’m super curious.

david

[rikardkrvaric]

I've had "bad luck" each and every time i've upgraded anything UCCX 12.5.

First I went from 12.0 to 12.5 SU1...
- CUIC not working - known defect
- Finesse agents can suddenly see every agent from every team - known defect apparently
- Agent's Jabber disconnects frequently (and I don't know why, because CUCM was not touched.) - ongoing TAC case since MARCH.
Upgraded 12.5 SU1 to 12.5 SU2...
- Live reports suddenly don't work in Finesse - apparently a known issue, resolved quickly with TAC case.

I am terrified to upgrade to SU4!

[ejlackey · Accepted solution]

Update.

I performed the upgrade a second time and switched versions.  This time I had more time to evaluate and eventually the services and systems settled down, synchronized, and are working.  This seemed to take multiple hours, as I left it overnight, and by morning it had stabilized.  Probably not an ideal situation for most, but this is a lab system for me with a small database.

However, after the update, it came up in evaluation mode and I had to re-register the Smart License.  In so doing, I ran into this bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwf38499

The test to the Direct Transport URL was failing.  The resolution was to install this cop file: https://software.cisco.com/download/home/286325233/type/286314176/release/12.5(1)_SU03_ES01

I was then able to register successfully, although it was still showing evaluation mode. All services are working during this time in evaluation mode.  I again left it overnight, and by morning everything had synchronized and Smart License account was updated accordingly.

[johns10t]

Had the same issue - services were in the same state after the SU1 -> SU3 upgrade. We were also facing an issue with Tomcat not starting properly on the Publisher. TAC found the Tomcat issue to be related to a problem with the Tomcat-ECDSA cert (which was still valid) and regenerated it. After doing this, they recommended applying the ES01, rebooted and then services were started on the Publisher. Subscriber failed to switch versions on the first try, so we rebooted it and tried again which was successful though we saw the same issue with services not activated. Applied the ES01, rebooted and services eventually came up. The upgrade wrapped up after about 18 hours. Not a smooth upgrade. 

[dandrools]

I have seen variations of this issue several times now, where some services are either stuck in a starting state, or listed as [STOPPED] Service Not Activated when they are core services that should absolutely be activated, such as Cisco Unified CCX Engine.  In the one case where tomcat was the service stuck in a starting state it was resolved by correcting the tomcat-ECDSA cert, same as the poster above.  In all other cases I have seen, I have proceeded with installing the latest ES as if nothing was wrong, and have seen one of two things happen:

1. After installing the ES and rebooting, all services started up normally and everything was fine.

2. While the ES was installing, all of the services that had failed to start suddenly began to start up and normalize.  It's as if the ES was healing the system in real-time. 

I'm sure that someday I'll find a situation that does something else but for now, if you are upgrading your UCCX to 12.5(1) SU3 and see services not starting after switching versions, my recommendation is to simply proceed with the ES installation and push your way through the problem.

[j.souchaud]

Hello,

I had the same issue here where services, notably Cisco Unified CCX Engine were stuck into [STOPPED] Service Not Activated after a successful switch-version to 12.5(1)SU3 from a two 12.5(1)ES03 servers in a HA cluster context.

I finally succeeded to fix this issue following @dandrools advice to apply the latest ES which in my case is ES06.

After yesterday evening first attempt failure being stuck for hours waiting for this service and others to start, even after a system restart, I'm glad I found this article.

Thanks you @ejlackey and others ! You make my evening sweeter now

Context :

2x UCCX 12.5(1)ES03 in HA cluster

Steps followed according to upgrade guide https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/crs/express_12_5_1_su3/install/guide/uccx_b_1251su3_install-and-upgrade-guide/uccx_m_1251su2_unified-ccx-upgrade.html#task_29EC75356F1D192F110CA937C79619C6 :

1- Applied ucos.keymanagement.v01.cop and ciscouccx.1251.1251SU3PREUPGRADE.4.cop files

2- Upgraded publisher to 12.5(1)SU3

3- Upgraded subscriber to 12.5(1)SU3

4- switch-version publisher and being stuck there