CUCM 12.5 NTP Unsynchronized

[bwolf]

Hello All,

CUCM System version: 12.5.1.13900-152   - 1 Publisher and 3 subscribers

I'm having an issue where NTP is unsynchronized. I can sit and use utils ntp restart on the publisher all day long and it never synchronizes.

remote refid st t when poll reach delay offset jitter
==============================================================================
+<NTP IP ADDRESS>                    u 15 64 1 1.023 -1823.8 1653.58
+<NTP IP ADDRESS>                    u 14 64 1 40.296 -1865.4 1568.27
+<NTP IP ADDRESS>                    u 13 64 1 0.907 -1910.8 1698.47
*<NTP IP ADDRESS>                    u 45 64 1 42.508 -519.46 287.555

unsynchronised
polling server every 64 s

You can see lots of jitter and offset. this status is right after I had just restarted NTP

We're using known good internal Linux NTPs (CUCM is the only system in our network having the NTP issue) that we have been using for years with no issues.  port 123 is open, no issues there.

I have a case open with TAC, they are recommending to uncheck the "Synchronize Time with Host" option on the VM in VMWare (ESXi version 7.0).  Which I haven't been able to do yet because our company is in a change freeze. 

I just wanted to check here to see if anyone else has experienced this and if they had any suggestions to get NTP resynchronized.  I also wanted to see if anyone who had this issue was able to resolve it by unchecking the "Synchronize Time with Host" option in VMWare. Most info I can find says to do traces (we've done that with TAC) and just keep trying to restart NTP, but this does not seem to help with getting NTP synchronized.

This issue is causing a DBRepliecationFailure to occur every hour.  The issue seems to have started after I upgraded from 11.5 to 12.5 (SU1) and continues to happen after we upgraded to 12.5 (SU4), which is the most current version of CUCM available.

When it first started happening, it seemed like restarting the CUCM cluster would resolve it temporarily.  About one week after the restart, the issue would occur again.  But the issue came up again two days ago and now restarting the cluster isn't even helping as the issue persists pretty much immediately after the cluster restart.

Any info would be greatly appreciated

Thanks!

[mn4]

Hello bwolf,

our customer is encountering the same issue. 

Was in your case the issue definitely solved as per your post on ‎07-13-2021 09:48 AM here above?

Never occurred again since that time?

Thanks

MN

[bwolf]

Hey mn4,

We haven't encountered the NTP issue since I upgraded the VMWare VM Hardware to version 19 and restarted the CUCM cluster.  Our ESXi hosts were upgraded from version 6 to version 7.0 U2 by our Unix team, that is when CUCM started having the NTP issue. Once I upgraded the VM Hardware to version 19, the NTP issue went away.

[bcon6928]

Do you know the Build Version of ESXi 7 you are running? I know mine was fine until I went to 18538813. The previous build of 7.0 U2 was fine. 

[mn4]

Update on this post:

The fix based on VM Hardware upgrade to version 19 appearently had some positive effect as after that CUCM nodes got back synchronized.

However after a few days I noticed that the CUCM was unsynchronized again and it would flap sync/unsync.

Cisco TAC confirmed that Windows NTP is not the best option for the CUCM and highlighted that our customer Windows NTP server is version 3, while CUCM works on NTP v4.

Above conditions are not guaranteed by Cisco (it may work or it may not).

We ended having the customer using public NTP servers.

Thanks.

MN

[bcon6928]

Update on the issue. Without much help from Cisco I had the issue come up again Monday. After digging through the vm events I found that the boxes with the NTP issue had v-motioned to a new host. Once I moved them back and restarted the NTP service we were back to synchronized. I know Cisco does not support v-motion so I created some DRS rules to keep voice on specific hosts. So all in all upgrading the hardware to match the host and creating DRS rules fixed it so far for us. What I'm trying to understand now is why moving to a new host would only affect the CUCM NTP service on the VM. If the CUCM VM needs to be moved to a new host a reboot of the VM will fix the NTP service as well. I am going to upgrade the VM tools in my next maintenance window and see if this helps as well. I hope this information helps as this has been a very frustrating issue for me with little help from VMware or Cisco. 

[Roger Kallberg]

Vmotion is AFAIK supported, but not DRS. Having rules to turn off DRS is a good thing as it's not supported.

[EYC]

Do we know if this same issue will occur in CUC and CCX v12.5? Thanks

[bcon6928]

I personally never had the issue come up on my UCCX. I'm running 12.X. I did have it on CUCM, Unity, and IM&P. I have not had the issue come up since locking them to specific hosts. I do think upgrading the hardware did help but I would not move past 7.0 U1 on ESXi due to Cisco not supporting. I am running 7.0 U2 and as long as the VM's don't move NTP works fine. I am using a Cisco Router for NTP. I hope this helps. Cisco was little help if any on this issue.

[EYC]

What is the VMtools version you all were running? Did u all do a refresh?

[bcon6928]

I upgraded all of mine to 10359. 

[EYC]

I have pretty much used Esxi v7.0u2 VMTools 10359, HW Tools v19 on CUCM v12.5su5 unchecked ntp sync. It will sync after a reboot but it will go off every couple of days and causing SIP endpoints to unregister flapping between cucm nodes and saying CUCM reset TCP connections in the status logs on the phone UI. I have nothing else to test and tac is pretty much useless