RTMT SHOWING NODE NOT REACHABLE WHEN IT IS

[Jon Clark1]

PROBLEM DESCRIPTION

Client is installing a new cucm subscriber node at a branch site.

All CUCM's can ping each other.

RTMT is showing the node unreachable(at the bottom in red) from all but 1 other CUCM.

CUCM Pub is showing all the phones as "unregistered" that are registered to this new node

If you push reset from CUCM Pub it will reset one of the "unregistered" phones

All nodes can ping eachother

show network cluster shows "unauthenticated" between the new node and all other nodes except the 1 CUCM Node it likes

a packet capture reveals that NEW node is not accepting/sending any packets on port 8002.

If you goto CUCM CLI and issue the command : show network ipprefs all

We see "disabled" for port 8002 under H-Status where as we see "enabled" on other nodes that are working properly.

For example?

admin:show network ipprefs all
Application IPProtocol PortValue HashLimit (max:rate) H-Status ConnLimit C-Status Type XlatedPort Status Description
------------ ------------ ------------ ------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------

ccm tcp 8002 2000:15/second disabled - disabled public - enabled CCM SDL Link

Any idea if this is related to the problem and/or how to enable it?

[Anthony Holloway · Accepted solution]

Wait, so you have nodes which are not authenticated in the show network cluster command? That sounds like they are not using the same cluster security password as the publisher, perhaps. I'm guessing the databases are out of sync though. For example, what does your db status show you? You might want to verify that your cluster security passwords are the same on all nodes, or just force them to a new value. You can do the former with the set password user security trick, where you type in the password you think it is, but then type in banana for the new password. If you get the current password wrong, it will tell you that, but if you got it right, you'll get a complaint that banana isn't a strong password. For the latter, you need console access, an ISO, and the pwrecovery user with pwreset password, and a reboot of all nodes.

[Jon Clark1]

Thank you for this reply. Though I posted the solution I am going to use your trick!

Thank you again for replying. Very grateful !

-JC

[Jon Clark1]

Great trick Anthony.  Below is a test where I'm guessing I used the correct cluster security password?

"BAD PASSWORD: it does not contain enough DIFFERENT characters" was the result. See below.

admin:set password user security
Please enter the old password: ******
Please enter the new password: ******
Reenter new password to confirm: ******
WARNING:
The Disaster Recovery System is dependent on this security password you are attempting to change.
If you need to use any of the older backup archive to restore this system, you need to remember the
older security password. To avoid this scenario, we recommend you to conduct a DRS Backup of your
system/cluster immediately after this password change.
Please make sure that the security password on the publisher is changed first.
The security password needs to be the same on all cluster nodes,
or the publisher and subscriber(s) will not communicate.
After changing the security password on a cluster node, please restart that node.

Continue (y/n)?Y

Please wait...

BAD PASSWORD: it does not contain enough DIFFERENT characters
admin:

[Jon Clark1 · Accepted solution]

Turns out the customer's firewall was blocking 2 ports that disrupted cluster communication between some of the servers thus having RTMT show "Node not reachable" when it actually was via ping.

Traffic on ports 8500 (Intracluster replication of system data by IPSec Cluster Manager) and 2555 (Real-time Information Services (RIS) database server) between CUCM01 and the other nodes to the new CUCM node being installed was being denied by a Firewall rule. The 2555 makes sense as it’s real time information that wasn’t being updated. 

So lesson learned. If you run into replication issues or show network cluster showing "unauthenticated" try checking if ports 8500 and/or ports 2555 are blocked. If they are open then try Anthony's set password user security trick to test the cluster security password on the new node.