CUCM 11.0 Subscriber Rebuild

[RL5901]

Our organization has a 5-node CUCM cluster (EOL/EOS version 11.0). One of the Subscriber nodes failed to recover when the UCS lost power during an electrical outage at the site where it is installed (UPS ran out). This is the output from CLI when it came back online...

Command Line Interface is starting up, please wait ...
java.io.FileNotFoundException: /var/log/active/platform/log/cli.bin (Read-only file system)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at com.cisco.iptplatform.fappend.ciscoRollingFileAppender.restoreIndex(ciscoRollingFileAppender.java:100)
at com.cisco.iptplatform.fappend.ciscoRollingFileAppender.setFile(ciscoRollingFileAppender.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:245)
at org.apache.log4j.config.PropertySetter.setProperty(PropertySetter.java:204)
at org.apache.log4j.xml.DOMConfigurator.setParameter(DOMConfigurator.java:691)
at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:265)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:176)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.java:191)
at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurator.java:523)
at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:492)
at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1001)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:867)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:755)
at org.apache.log4j.xml.DOMConfigurator.configure(DOMConfigurator.java:891)
at sdMain.initialize(sdMain.java:509)
at sdMain.main(sdMain.java:676)
java.lang.NullPointerException
at com.cisco.iptplatform.fappend.ciscoRollingFileAppender.updateIndex(ciscoRollingFileAppender.java:117)
at com.cisco.iptplatform.fappend.ciscoRollingFileAppender.nextFileName(ciscoRollingFileAppender.java:92)
at com.cisco.iptplatform.fappend.ciscoRollingFileAppender.append(ciscoRollingFileAppender.java:74)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.debug(Category.java:260)
at com.cisco.iptplatform.cli.CliSettings.getInstance(CliSettings.java:116)
at sdMain.initialize(sdMain.java:521)
at sdMain.main(sdMain.java:676)
log4j:ERROR No output stream or file set for the appender named [CLI_LOG].

After researching I came to the conclusion that this Subscriber node will require a rebuild as there is no Recovery ISO file available from the Cisco download site anymore for our version. I can only find documentation for 10.5 and 11.5 for the rebuild procedure, but oddly not for 11.0. This is the document I found...

Replacing A Single Server or Cluster for Cisco Unified Communications Manager, Release 11.5(1)

Will this documentation work for replacing/rebuilding the lost Subscriber node 11.0?

Our organization's CUCM cluster is on system version: 11.0.1.22900-14 and I very recently gained access to a network share drive that has the UCSInstall_UCOS_11.0.1.22900-14.sgn file. 

I made our organization aware that CUCM 11.0 has been EOL/EOS since July 31, 2020, and was finally successful in convincing leadership to approve a project to upgrade (pushed hard for this for 2 years). We have a project underway to install new UCS servers and upgrade all of our UC systems (CUCM, CUC, CER) to version 14 (which will require a step upgrade). The UCS install is being worked on but there may be a month before we can get to the upgrades due to other scheduling issues with our limited internal team. We could wait until the UCS servers are installed to build the Subscriber node on the new UCS servers but I'm not sure if waiting is better or if building it new in the existing cluster (project work) is better. 

1. Will the above documentation provide the needed steps (and cautions) needed to rebuild the broken Subscriber on our current version (11.0)? 
2. Is there another way to bring that broken Subscriber back into service without the rebuild?
3.  What are the pros and cons of rebuilding the broken Subscriber before the new cluster is built? 
4. What are the risks of rebuilding the broken Subscriber node in the current cluster (version 11.0)?
5. What are the risks of waiting to build it on the new UCS servers as a Subscriber node in the version 14 cluster? 
6. Any advice or direction is appreciated. 

[Jonathan Schulenberg · Accepted solution]

Yes, that 11.5 doc is fine for 11.0.

I suggest downloading the 11.5 recovery ISO and attempting a fsck; it’s possible the journal can replay the final write transactions to repair the file system.

Cisco employees also have access to an internal software library of all versions. Ask your Cisco account team for the 11.0 ISOs. The one you have probably isn’t bootable; the file name usually starts with Bootable_ when it is.

[RL5901]

Thank you, Jonathan. I was under the impression that the Recovery ISO files are specific to the version. This gives me hope that we can recover it.

I will reach out to my Cisco account team and ask if they have the bootable 11.0 iso for our system version. Thank you!