UC Version 15 Post Upgrade NTP Issue

[UCAdmin405]

Has anyone had any issues with NTP after upgrading to version 15? I just upgraded my test CER cluster from v14su3 to v15su1. I had an issue with the database not starting on the primary node after switching versions. But a reboot of the server fixed (started) the database. However, after switching the version and the second reboot I'm still getting the WARNING: NTP is Unreachable message (the NTP server is up and running and reachable). Running the 'utils ntp status' command returns "ntp is stopped". Running the 'utils ntp start' command returns "NTP is already running", and the restart command fails. The kicker is the 'utils diagnose test' command returns a pass for all the NTP related tests. I engaged TAC but I'm waiting for a response. While I'm waiting I went ahead and upgraded my PCD server from v14 to v15 as well. I'm experiencing the same issue with the exact same results I'm getting in CER. So to me it seems to be something with version 15 and the change from centOS to AlmaLinux since the same thing is happening both on CER and PCD? I'm at a loss and I haven't found anything in documentation regarding NTP issues. I figured I'd try my luck here and see if the community has seen anything like this before?

[Roger Kallberg]

We did almost the exact same update of CER last week, the difference is that we went to 15SU1a. We haven’t experienced any issues whatsoever. On PCD we did the upgrade to FCS release of 15 and then upgraded to SU1 when it was released, but that was months ago. Haven’t seen any issues on that either.

What version of ESXi are you running and what version of the hardware compatibility are you using?

[UCAdmin405]

I'm not sure what you mean by, "what version of the hardware compatibility are you using?". We are running ESXi 7.0 U3 on a BE7K M4 box. On the original post I did leave off the 'a' from the version number. The upgrade I did was from v14su3 to v15su1a. The PCD was running the FCS v14 and it was upgraded straight to v15su1. I should have clarified that in the original post. When I did the upgrades from the CLI and as I was watching log messages it did hang for quite awhile with an output message about NTP. I didn't think at the time to make any note of it. I should be hearing back from TAC tomorrow. It's just weird the same thing happened on both CER and PCD. Glad to know you were able to complete the same upgrade without issues but leaves me still scratching my head. Are you running newer hardware and/or ESXi in your environment?

[Roger Kallberg]

Originally we were on the same version of ESXi as you, but since the DC team have updated the hosts to v8u1 I think, not sure on the U version. Our hardware is likely a bit newer than yours and we’re not on Cisco hardware anymore, we moved to HPE hardware for our UC infrastructure last year.

In ESXi you can use different compatibility versions on the VMs. These are seen as a number on the VMs, for example compatibility version 19 that equals ESXi 7u3, or 21 which equals 8u1. 19 is the one where using, even if we’re on a newer ESXi version on the host.

I’m currently on vacation, so can’t give you a screenshot of where you’d see the version or upgrade it, but I’m sure you’ll figure it out.

[Roger Kallberg]

This document describes the process for how to update the compatibility version on the VM. Upgrade Virtual Machine Compatibility by Using the VMware Host Client 

[UCAdmin405]

TAC suggested deleting and re-adding the ntp server. To do that I had to add another ntp server which was successful. However, when adding back the original ntp server it failed. Adding the additional ntp server did fix the issue with the inconsistent 'utils ntp' commands though and I was able to stop/start/restart the ntp service. It also resolved the ntp unreachable warning message. The ntp server running in this test environment is a raspberry pi running ntpd. It is the time source for everything in the test environment and works great.

I was able to dig through some of the CER logs once the ntp service actually started after adding the new ntp server. When trying to add the original ntp server, the reachability check did return this, "response for NTP server <x.x.x.x> failed to include stratum information. Response: 506 Cannot talk to dameon" error in the logs. Nothing has changed on the ntp server and the "temporary" ntp server I configured is actually using the original ntp server as its source. The original ntp server is also the source for the other 3 UC applications (all running 12.5) which are synchronized and function with it. It's a bizarre outcome which looks to be isolated to v15 not liking the raspberry pi ntpd service for some reason. @Roger Kallberg, thank you for your responses but it appears my "ntp upgrade issue" is self inflicted. Though it is out of the scope of this forum I believe. I will try to continue to troubleshoot it and see if I can reestablish connectivity between CER and the pi as well as seeing what TAC says about my latest findings.

[Roger Kallberg · Accepted solution]

Try using Chrony as your time server instead of the very old ntpd and make sure that you have this in the configuration.

local stratum 2  

Cisco UC systems are notably picky about the stratum level its talks to and if the time server doesn’t give information on the level, like the logs seems to indicate that yours doesn’t, it won’t be happy.

[Roger Kallberg · Accepted solution]

This article outlines how to setup Chrony as a time server in Raspberry-Pi. https://conorrobinson.ie/raspberry-pi-ntp-server-part-2/

[UCAdmin405]

Switching to chrony worked! The upgraded v15 server allowed me to configured the original ntp server after switching it to use chrony. I also rebuilt my old v14 cluster and restored the backups. Proceeding with the v15 upgrade again there were no issues at all (database or ntp). TAC also confirmed that there is an issue with version 15 and it's communication with the ntpd service in there lab. They got the same results I did with there upgrade. They have escalated this internally to the developers and will work on updating documentation stating the incompatibility until it can be addressed.  @Roger Kallberg I appreciate the help and especially the link for setting up chrony. It was a great resource.