NTP issue on CUCM 12.0.1

rwesel · ‎04-27-2021

Hello,

I have an issue on our CUCM for which I cannot find a solution.

NTP on CUCM publisher show still unsynchronised.

NTP servers are Solaris and seems to be NTPv4.
NTP servers are reachable.
Other devices (Switches, Linux, ...) can synchronise with these NTP servers.
Hints from Cisco document "NTP Troubleshooting on Cisco Unified Communications Manager" didn't help.
We have one subscriber that is synced with the publisher.

Does anyone have any idea how I can solve the problem?

Thank you

admin:utils ntp status 
ntpd (pid 9284) is running...

remote        refid  st t when poll reach delay   offset    jitter
==============================================================================
192.168.1.21  .DCFa. 1  u 38   64   3     1.791   -19576.   17723.3
192.168.1.22  .PPS.  1  u 37   64   3     2.984   -19864.   17713.5

unsynchronised
polling server every 8 s

Current time in UTC is : Tue Apr 27 07:34:27 UTC 2021
Current time in Europe/Berlin is : Tue Apr 27 09:34:27 CEST 2021

admin:utils network capture port 123
Executing command with options:
size=128 count=1000 interface=eth0
src=dest= port=123 
ip= 
10:00:02.525354 IP cucm01.54177 > 192.168.1.21.ntp: NTPv4, Client, length 48
10:00:02.526615 IP 192.168.1.21.ntp > cucm01.54177: NTPv4, Server, length 48


10:00:07.597767 IP cucm01.ntp > 192.168.1.22.ntp: NTPv4, Client, length 48
10:00:07.601732 IP 192.168.1.22.ntp > cucm01.ntp: NTPv4, Server, length 48

admin:file view activelog syslog/messages.1
...
Apr 25 03:29:05 cucm01 user 2 platform: The local NTP client is off by more than the acceptable threshold of 3 seconds from its remote NTP system peer. The normal remedy is for NTP Watch Dog to automatically restart NTP. However, an unusual number of automatic NTP restarts have already occurred on this node. No additional automatic NTP restarts will be done until NTP time synchronization stabilizes. This is likely due to an excessive number of VMware Virtual Machine migrations or Storage VMotions. Please consult your VMware Infrastructure Support Team.
Apr 25 03:30:05 cucm01 user 4 platform: The local NTP client is off by more than the acceptable threshold of 3 seconds from NTP server 192.168.1.21. Restarting NTP ...
Apr 25 03:30:05 cucm01 user 6 ilog_impl: Restarting NTP as time offset is -145.010660 msec.
Apr 25 03:30:06 cucm01 daemon 4 init: ntpd pre-stop process (3978) terminated with status 2
Apr 25 03:30:06 cucm01 daemon 4 init: ntpd main process (29599) killed by TERM signal
Apr 25 03:30:06 cucm01 user 6 ilog_impl: NTP servers list: 192.168.1.21 192.168.1.22
Apr 25 03:30:06 cucm01 user 6 ilog_impl: ntpdate query response: server 192.168.1.21, stratum 1, offset -145.560090, delay 0.02640#01225 Apr 03:30:06 ntpdate[4196]: step time server 192.168.1.21 offset -145.560090 sec (rc=0).
Apr 25 03:30:06 cucm01 user 6 ilog_impl: ntpdate query response: server 192.168.1.22, stratum 1, offset -145.666639, delay 0.02815#01225 Apr 03:30:06 ntpdate[4212]: step time server 192.168.1.22 offset -145.666639 sec (rc=0).
Apr 25 03:30:07 cucm01 user 6 ilog_impl: NTP server '192.168.1.21' has minimum stratum (1) and delay (0.02640).
Apr 25 03:27:41 cucm01 user 6 ilog_impl: ntpdate 192.168.1.21 response: 25 Apr 03:27:41 ntpdate[4231]: step time server 192.168.1.21 offset -145.718609 sec
Apr 25 03:27:41 cucm01 user 4 platform: A large time correction of 145.718609 seconds by NTP server 192.168.1.21 has occurred. NTP Watch Dog will automatically restart NTP if required. No manual action is required.
Apr 25 03:27:41 cucm01 user 6 ilog_impl: ntpdate 192.168.1.21 response: 25 Apr 03:27:41 ntpdate[4252]: adjust time server 192.168.1.21 offset -0.080398 sec
Apr 25 03:27:41 cucm01 user 6 ilog_impl: ntpdate 192.168.1.21 response: 25 Apr 03:27:41 ntpdate[4271]: adjust time server 192.168.1.21 offset -0.142787 sec
Apr 25 03:27:41 cucm01 user 6 ilog_impl: NTP time snchronization successful.
Apr 25 03:27:42 cucm01 daemon 5 ntpd: ntpd 4.2.6p5@1.2349-o Mon Feb 6 07:22:46 UTC 2017 (1)
Apr 25 03:27:42 cucm01 daemon 5 ntpd: proto: precision = 1.117 usec
...

admin:show version active 
Active Master Version: 12.0.1.10000-10
Active Version Installed Software Options: 
cmterm-ata191.12-0-1-0301-002.k3.cop
cm-locale-de_DE-12.0.1.2000-1.cop

Nithin Eluvathingal · ‎04-27-2021

CUCM doesn’t sync with some ntp servers. So what I normally do, sync any Cisco switch or router with enterprise ntp and make this device as a ntp sever for CUCM.

rwesel · ‎04-27-2021

Thank you,

these NTP servers are successful used by all of our hosts, machines, switches etc. It is strange that the CUCM should not be able to use this...

If I would consider to change the NTP source to a Switch or Router I would have to remove the existing servers.

But when I try to do this I get a warning:

"WARNING: Changing this setting will invalidate software license
on this server. The license will have to be re-hosted."

I'm not sure about all the steps I would have to do then.

Stefan

Nithin Eluvathingal · ‎04-27-2021

From your first message I see active version 12.0, AFAIK, changing the NTP on CUCM 12 doesn't have any issues with the license. Second thing you don't need to remove the old, add the new server and it will sync.

This is my lab gateway configurations.

JLR-CORE#sh run | sec ntp

ntp master 2

ntp server time.google.com

admin:utils ntp status

ntpd (pid 13747) is running...

remote refid st t when poll reach delay offset jitter

==============================================================================

*10.60.51.1 216.239.35.8 2 u 462 1024 377 1.686 1.283 2.606

synchronised to NTP server (10.60.51.1) at stratum 3

time correct to within 100 ms

polling server every 1024 s

TONY SMITH · ‎04-28-2021

@rwesel wrote:

But when I try to do this I get a warning:

"WARNING: Changing this setting will invalidate software license
on this server. The license will have to be re-hosted."

That message is incorrect, changing NTP reference doesn't affect licensing, I think that's been the case since version 9. The licences aren't even installed on the CUCM servers since then, they're installed on ELM/PLM.

Nithin Eluvathingal · ‎04-28-2021

What version of CUCM you use ? I never seen such warning with new versions.

Restart NPT service and wait few mins and see if it sync. make suer that CUCm can reach the IP.

utils ntp restart

From my lab UCM version 12.X

admin:utils ntp server delete

1: 10.60.51.1

2: 192.168.1.1

a: all

q: quit

Choice: 2

Restart NTP (y/n): y

192.168.1.1 will be deleted from the list of configured NTP servers.

Continue (y/n)?y

192.168.1.1 : deleted successfully.

Restarting NTP on the server.

TONY SMITH · ‎04-30-2021

@Nithin Eluvathingal wrote:

What version of CUCM you use ? I never seen such warning with new versions. .

It was 12.0 as well, 12.0.1.22900-14 to be exact.

rwesel · ‎04-30-2021

As I wrote in my first post:

admin:show version active 
Active Master Version: 12.0.1.10000-10
Active Version Installed Software Options: 
cmterm-ata191.12-0-1-0301-002.k3.cop
cm-locale-de_DE-12.0.1.2000-1.cop

Because of the license warning I don't dare to delete the Stratum1 NTP server. But maybe I should be brave and trust the statement that with version 12 there is no effect on the license status.

Nithin Eluvathingal · ‎04-30-2021

The details I shared on my previous reply was from CUCM 12.5. AFAIk there is no license issues if you remove the NTP on CUCM 12. these are smart licenses so no issues.

TONY SMITH · ‎04-30-2021

@rwesel wrote:

Because of the license warning I don't dare to delete the Stratum1 NTP server. But maybe I should be brave and trust the statement that with version 12 there is no effect on the license status.

It will be OK. I've done it on 12.0 servers where that message was (incorrectly) displayed. Changing NTP doesn't affect licensing on any version later than 8.x.

rwesel · ‎04-30-2021

Ok, now I removed all other NTP servers and left only Cisc IOS Router (192.168.1.1) with ntp master 1 confgured.

No success - CUCM still says unsychronised after ~15min. The clock of CUCM is about 2-3min out of sync. Maybe it is not so much, but I would prefer the correct time.

Is there a way to debug why CUCM will not sync with NTP server? (logfile, debug tool, ...?)

admin:utils ntp status
ntpd (pid 18623) is running...

remote refid st t when poll reach delay offset jitter
==============================================================================
192.168.1.1 192.168.1.22 2 u 64 64 17 1.590 -59892. 51617.0


unsynchronised
polling server every 8 s

Current time in UTC is : Fri Apr 30 12:02:18 UTC 2021
Current time in Europe/Berlin is : Fri Apr 30 14:02:18 CEST 2021

Thanks
Stefan

TomMar1 · ‎04-27-2021

You do not have to remove those NTP references. Just add a router or switch as Nithin suggests. UCM will decide which one is best, if the newly added routers/switches seem to be working better, you can than remove your current NTP references. As mentioned UCM NTP is fickle.

rwesel · ‎04-28-2021

Unfortunately it is not working.

1) I configured my router as NTP source

ntp master 1

2) I added router as NTP server in CUCM

admin:utils ntp server add 192.168.1.1

3) Because the router itself has the company NTP server (192.168.1.21) as the NTP source, the CUCM sees it as stratum 2 and thus will not use it.

admin:utils ntp status 
ntpd (pid 9284) is running...

remote       refid        st t when poll reach delay offset jitter
==============================================================================
192.168.1.21 .DCFa.       1 u 38 64 3 1.791 -19576. 17723.3
192.168.1.1  192.168.1.21 2 u 37 64 3 2.984 -19864. 17713.5

unsynchronised
polling server every 8 s

4) I think deleting the company NTP server in CUCM would fix it. But if i try this, a warning comes.

admin:utils ntp server delete 
1: 192.168.1.21
2: 192.168.1.1
a: all
q: quit

Choice: 1

WARNING: Changing this setting will invalidate software license 
on this server. The license will have to be re-hosted.

TONY SMITH · ‎04-28-2021

How far out is the time on the server? It may take a while to synchronise after you changed NTP. Does adding the server stop/restart the NTP service? If not then maybe do that manually.

mn4 · ‎10-27-2021

Hello rwesel,

having the same problem, was it then fixed?

If yes can you share details?

Thank you

MN