CUCM 7.1: RTMT is reporting ServerDown, but it's actually up

[Giovanni Ceci]

Hey everyone,

I've just ran into a sudden problem. RTMT (signed in with Publisher IP) is reporting that Subscriber is down. But I can access both servers just fine! All devices seem to be working properly as no one has reported any issues. Pinging works fine between the two with both IP and hostname. It seems that Publisher can't talk to Subscriber in some capacity. I've restarted the RTMT service on both servers, but still it's reporting this. I've attached a screenshot of RTMT.

Any ideas on what could be the problem could be here?

Thanks,

neocec

[chweeks]

Just to be on the safe side, I would take a look at these things.

1. Go into the Unified Reporting Tool (top right corner dropdown, bottome choice), and run a Database Status report. Do that by selecting the Database status report on the left, then on the right select, "Generate New Report", this will look like a page with a graph logo over the top. That report will give you a good idea if the dtabase replication between these nodes is doing ok. This is not the be-all-end all, but is a good indicator of problems.

2. I would go into each node, via CLI, and check things with the command, "utils service list page" and make sure all the services are started that should be started.

3. I would also run the command, in CLI, 'utils diagnose test' and make sure that come back clean.

- Check your system and application logs on each node for stange entries.

- Are you using the RTMT tool from a seperate network that where the nodes reside? Is there a VPN or firewall between your client machine (where rtmt is) and the nodes?

These are just ideas to either find some underlying issue, or further ensure you that things are in fact fine and RTMT is wacked.

Cheers,

Chris

[Giovanni Ceci]

Hey Chris,

Thanks for replying. Those are all good things to check. However, later on when I checked RTMT, it showed some SDLLinkOutOfService errors. Looking further into that error, there are only two possible causes: A network failure occurred or a subscriber node is down, and I know the latter was not the case. So I believe some network issue triggered a disconnect between the nodes, and I'll have to check out the logs and traces.

Thanks for your help!

neocec

[keithknowles]

Have you run a long ping? By that I mean let the pings run for a minute or two to see if there is a point that several packets drop and then it returns to normal? I have seen similar behavior with an intermittent network problem.

Sent from Cisco Technical Support iPhone App

[chweeks]

You can add the 'count' flag at the end:

For example. This will ping 1,000,000 times to the destination.

admin:utils network ping 10.88.7.250 count 1000000
PING 10.88.7.250 (10.88.7.250) 56(84) bytes of data.
64 bytes from 10.88.7.250: icmp_seq=0 ttl=60 time=0.718 ms
64 bytes from 10.88.7.250: icmp_seq=1 ttl=60 time=0.507 ms

Just be advised, ping is a very poor tool for determing that the health of a connection between two CUCM nodes. Yes it proves layer 1,2,3 connectivity, but thats about it. Pinging something is one thing, having all the other necessary ports for CUCM nodes to communicate through opened through the network is a totally different things all together.

Are there any firewalls, ACL's, or VPN's between these two nodes?