11-18-2010 03:15 AM - edited 03-16-2019 01:59 AM
Hi,
My CUPS server is complaining that the CM publisher's NTP isn't working. The publisher is a CUCM-BE and a show ip preferences shows that NTP is blocked by the firewall. It's synced fine to our internal NTP servers but won't let anything else sync to it which is tricky because CUPS insists on syncing to it. Any ideas how to enable NTP on the server?
Thanks,
Joe
11-23-2010 09:33 AM
Hello Joe
Can you check the status of the ntp with this command in the CUCMBE:
utils ntp status
Also check the output on the CUCMBE of the command:
utils firewall list
It should include a line saying:
ACCEPT udp -- anywhere anywhere udp dpt:ntp
Please send the output of those commands to see how it looks like.
Try also if you have available the command: utils ntp start
( i dont have a CUCMBE that i can test, i only have CUCM servers)
Thanks
Fernando
11-23-2010 12:25 PM
Well, my first question would be the following:
Did you configure your CUCMBE to sync to an external NTP server - if so, is that server truly EXTERNAL (i.e., on the internet) and something that may well be blocked by your firewall? Or, rather did you sync to a server on a different subnet that may be inadvertently blocked by a firewall?
The first is an issue that has come up many times and is not a recommended configuration IMHO. For many reasons...which I won't go into. The second is something that can be easily rectified. What I would do is configure a local gateway (for example 2811) as an NTP server and set that as the NTP source for the CUCMBE. See if you can sync to that as a source - internally, and ideally within a trusted network.
Hailey
Please rate helpful posts!
11-23-2010 03:45 PM
admin:utils ntp status
ntpd (pid 30559) is running...
remote refid st t when poll reach delay offset jitter
==============================================================================
127.127.1.0 LOCAL(0) 10 l 20 64 377 0.000 0.000 0.004
*10.222.69.248 203.12.160.2 3 u 125 128 377 1.077 8.685 1.726
10.222.67.1 .INIT. 16 u - 1024 0 0.000 0.000 4000.00
synchronised to NTP server (10.222.69.248) at stratum 4
time correct to within 119 ms
polling server every 128 s
Current time in UTC is : Tue Nov 23 23:35:55 UTC 2010
Current time in Australia/Queensland is : Wed Nov 24 09:35:55 EST 2010
admin:utils firewall ipv4 list
Table: mangle
Not happy publishing my firewall rules although they're default.
Anyway - standard IPTables and no entries for port 123 or NTP in the config.
I'd figured that the firewall rules were the cause - the question is can we tweak them? My guess was CUCMBE is standalone so doesn't want to talk NTP but CUPS insists on learning NTP from the CM.
Regards,
Joe Bennett
Network Consultant
Bridge Point Communications
11-23-2010 05:00 PM
Joe,
Could you check/post the output of "utils ntp status" from the CUPS server console? In regards to the firewall list on the CUCM, there should be a line for udp port 123. If you have a Linux or Mac host, you could check to see if you can reach NTP on the CUCM server by using the ntpq command.
For example:
sh-3.2# ntpq -c peers 10.3.4.20
remote refid st t when poll reach delay offset jitter
==============================================================================
*10.3.2.1 204.9.54.119 2 u 45 64 377 2.341 -81.569 29.076
In the example above, 10.3.4.20 is my CUCM publisher node.
If you only have a Windows PC/server then you can probably find a NTPQ windows binary on the internet. The older CallManager servers had a NTP package called XNTP, which I used on my Windows boxes in the lab.
If you can successfully run a NTP query (ntpq) on the CUCM from a remote station, then the CUCM firewall is not blocking you.
You could also try to troubleshoot this using the network capture facility on the CUCM and CUPS servers. For example:
admin:utils network capture port 123 count 1000 size 128 host ip 192.168.1.4
Executing command with options:
size=128 count=1000 interface=eth0
src= dest= port=123
ip=192.168.1.4
In this example, I am running the capture on my CUCM and the 192.168.1.4 address is my macbook running the ntpq command. However, you could set this up on both the CUCM-BE and CUPS server.
On CUCM:
admin:utils network capture file ntpcheckFromCUCM count 100000 size all port 123
On CUPS:
admin:utils network capture file ntpcheckFromCUPS count 100000 size all port 123
You could also filter by IP addresses if you wanted to see other information (like an ICMP port unreachable?). In this latter case, I would recommend filtering by IP address on the CUPS server (host ip
Anyway, after you execute the command on both servers. You may want to open up a NEW SSH session to the CUPS server and execute a NTP restart (utils ntp restart) or start (utils ntp start). After a minute, you will want to enter Ctrl-C on the CUCM-BE and CUPS SSH sessions where you initiatied the capture.
You can then download the capture files:
On CUCM:
admin: file get activelog platform/cli/ntpcheckFromCUCM.cap
On CUPS:
admin: file get activelog platform/cli/ntpcheckFromCUPS.cap
You can then use Wireshark or some other protocol analyzer to review the sniffer traces.
Now, you don't necessarily have to capture the output for processing off box, if you go back to the example where I was capturing traffic on port 123 to/from host 192.168.1.4 (above) then you could at least determine packets were being exchanged. In our example, I ran a ntpq -c peers command from 192.168.1.4 right after setting up the capture on the CUCM host. Here is the output:
admin:utils network capture port 123 count 1000 size 128 host ip 192.168.1.4
Executing command with options:
size=128 count=1000 interface=eth0
src= dest= port=123
ip=192.168.1.4
19:44:52.953623 IP 192.168.1.4.61591 > iecucm01.cnclab.com.ntp: NTPv2, Reserved, length 12
19:44:52.954080 IP iecucm01.cnclab.com.ntp > 192.168.1.4.61591: NTPv2, Reserved, length 16
19:44:52.957733 IP 192.168.1.4.61591 > iecucm01.cnclab.com.ntp: NTPv2, Reserved, length 12
19:44:52.957910 IP iecucm01.cnclab.com.ntp > 192.168.1.4.61591: NTPv2, Reserved, length 480
19:44:52.957975 IP iecucm01.cnclab.com.ntp > 192.168.1.4.61591: NTPv2, Reserved, length 144
You can tell two things from this output. First, the NTP packets from 192.168.1.4 successfully reached the CUCM server. This definitively tells us the network isn't blocking. Second, you can see that the CUCM server (iecucm01) responds on the NTP port. This definitevely tells us that the CUCM firewall isn't blocking anything. Of course, I suspect that if the host firewall was blocking NTP we may even find that the first NTP packet is not seen. I would have to test that but the result is not important.
At this point, you should have enough data to determine where the communication is breaking.
Note, that if you needed to test the host based firewall (i.e. remove it from the equation), then you could try to disable the firewall (utils firewall disable). I haven't done this myself, so read up on this command before you start your testing.
HTH.
Regards,
Bill
Please rate helpful posts.
Please remember to rate helpful responses and identify
11-23-2010 06:06 PM
Presence server
admin:utils ntp status
ntpd (pid 8458) is running...
remote refid st t when poll reach delay offset jitter
==============================================================================
*127.127.1.0 LOCAL(0) 10 l 52 64 377 0.000 0.000 0.001
10.222.66.234 .INIT. 16 u - 1024 0 0.000 0.000 4000.00
synchronised to local net at stratum 11
time correct to within 12 ms
polling server every 64 s
Current time in UTC is : Wed Nov 24 01:53:17 UTC 2010
Current time in Australia/Brisbane is : Wed Nov 24 11:53:17 EST 2010
======================================================================
joebennett@toad:~$ ntpq -c peers cucmbe
cucmbe: timed out, nothing received
***Request timed out
============================================
Ssh to CUCMBE;
admin:utils firewall ipv4 disable
Warning: Disabling the internal firewall can cause disruption in network
services. In particular redirected traffic such as HTTP and TFTP will be
disrupted which can affect phone registrations.
Do you want to continue?
Enter (yes/no)? yes
firewall (iptables) is disabled
firewall (iptables) will be enabled at Wed Nov 24, 2010 12:07:30
admin:exit
================================================
SSH to presence
admin:utils ntp restart
Restarting NTP
admin:utils ntp status
ntpd (pid 29716) is running...
remote refid st t when poll reach delay offset jitter
==============================================================================
127.127.1.0 LOCAL(0) 10 l 2 64 1 0.000 0.000 0.001
10.222.66.234 10.222.69.248 4 u 1 64 1 1.187 0.822 0.001
unsynchronised
time server re-starting
polling server every 64 s
Current time in UTC is : Wed Nov 24 02:02:59 UTC 2010
Current time in Australia/Brisbane is : Wed Nov 24 12:02:59 EST 2010
========================================================
Works fine with no firewall - how do I add a rule in IPTables with no shell access?
Regards,
Joe Bennett
Network Consultant
Bridge Point Communications
11-23-2010 06:30 PM
joe.bennett wrote:
================================================
SSH to presence
admin:utils ntp restart
Restarting NTP
admin:utils ntp status
ntpd (pid 29716) is running...
remote refid st t when poll reach delay offset jitter
==============================================================================
127.127.1.0 LOCAL(0) 10 l 2 64 1 0.000 0.000 0.001
10.222.66.234 10.222.69.248 4 u 1 64 1 1.187 0.822 0.001
unsynchronised
time server re-starting
polling server every 64 s
Current time in UTC is : Wed Nov 24 02:02:59 UTC 2010
Current time in Australia/Brisbane is : Wed Nov 24 12:02:59 EST 2010
========================================================
Works fine with no firewall - how do I add a rule in IPTables with no shell access?
Joe,
Your output still shows unsynchronized. Did the NTP eventually establish sync? If it did, then maybe you are running into this bug:
NTP port is not opened on Publisher firewall after changing server role | |
Symptom: Installation of secondary node fails with following error message "NTP server inaccessible, system will halt" Conditions: It is observed on VOS based products ( like Communication Manager, Unity Connection) where the installation of the Publisher server was started of as secondary node and later changed as primary ( first node in the cluster ) during the installation procedure. Workaround: Following commands can be used to verify whether Publisher ( primary node ) is accepting the NTP traffic or not. show network ipprefs all show firewall ipv4 list If NTP is disabled and udp/123 port is blocked in the firewall, we can execute the following command on Publisher to update the firewall configuration /usr/local/bin/base_scripts/ipprefs -T ntp --enable |
Which means something got fowled up during the install. You would need to contact Tac to execute the workaround.
HTH.
Regards,
Bill
Please rate helpful posts.
Please remember to rate helpful responses and identify
11-23-2010 06:44 PM
admin:utils ntp status
ntpd (pid 29716) is running...
remote refid st t when poll reach delay offset jitter
==============================================================================
127.127.1.0 LOCAL(0) 10 l 2 64 377 0.000 0.000 0.001
*10.222.66.234 10.222.69.248 4 u 42 64 377 0.204 20.483 0.009
synchronised to NTP server (10.222.66.234) at stratum 5
time correct to within 198 ms
polling server every 64 s
Current time in UTC is : Wed Nov 24 02:39:22 UTC 2010
Current time in Australia/Brisbane is : Wed Nov 24 12:39:22 EST 2010
The bug looks interesting except that the cause is different (can't start a CUCMBE install as a sub). It looks like shell access is the answer either way.
Thanks for your help. I'd never poked around with the firewall and NTP from the CLI so those commands were useful.
Regards,
Joe Bennett
Network Consultant
Bridge Point Communications
11-23-2010 08:21 PM
OK. Glad to be of some nominal help.
Regards,
Bill
Please remember to rate helpful responses and identify
11-24-2010 05:24 AM
Hello
In CUCM BE there is no concept of Publisher or Subcribers. It is a standalone server.
And that might be the cause of the problem.
I checked some of the standalone servers in my lab (CUCM 5.1.3, CUCM 6.1.4)
All of them have the firewall locked for the NTP traffic. (As there is no other server configured for clustering).
I will try with CUCM 7.X and CUCM 8.X now.
Which version of CUCMBE do you have installed?
Do yo have your CUPS configured in the CUCM under Server -> Application Server?
Thanks
Fernando
11-24-2010 07:10 AM
Fernando,
Good feedback, thanks (+5).
I agree with the lack of pub/sub in CUCM-BE. I thought about the FW filter as it relates to NTP and CUCM-BE but I haven't worked with CUCM-BE in well over a year and I wasn't sure. I dismissed it because I believe CUCM-BE and CUPS are supposed to be compatible and in CUPS 7.0(4) (??) and later, I am pretty sure that CUPS will assume it recovers NTP from the CUCM cluster it is associated with. IOW, you no longer assign a NTP server via the CUPS GUI (maybe that has changed in 8.x). Anyway, if what you say is true (and I don't doubt you) then this could still be logged as a bug and the mitigation would likely be the same.
Regards,
Bill
Please remember to rate helpful responses and identify
11-24-2010 01:58 PM
Hi Will
I totally agree with you.
CUPS should be able to work with the CUCM-BE.
Basically CUPS should be added in the Application Server list in the CUCM.
Then CUPS should try to authenticate with the CUCM (Cluster Manager service is responsible for that).
If the CUCM authenticate the CUPS (meaning that CUCM has CUPS listed in Application Server config and that the security password matches) then it will open the firewall.
At this moment i would expect the NTP line to be added in the firewall rules.
(btw the CUCM stand-alone servers version 7.1.3 and 8.0.3 i tested had the rule added by default, contrary to CUCM 5.x and 6.x)
So i believe that Joe's problem could be that his CUPS server is not authenticated by the Cluster Manager and hence FW still is closed.
Otherwise as you said it is a big bug, but it should be affecting more customers (or i would expect it at least) and i havent seen any SR like that or any defect mention it.
@Joe:
Please get the output of the command: show network cluster from your CUCM
It should list the CUCM and the CUPS.
What versions of CUCM and CUPS are you using?
Also another problem is that in my lab i dont have any CUCM-BE, i only have CUCM single-node and multinode clusters so i cannot really test same scenario as Joe.
Tomorrow i will try to add a CUPS to one of my CUCM 6.x or 5.x that had the FW closed for NTP and test if the ClusterManager service opens the FW once the CUPS is validated. (I would expect that).
Thanks
Fernando
11-24-2010 03:08 PM
It's definitely in the application server list and the firewall has rules for it, jut not NTP.
Table: filter
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:4040 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:1500 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:1515 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:8001 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:1090 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:1099 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:2555 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:2556 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpts:2551:2552 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:1502 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:1503 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpts:4900:4904 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:19003 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:20055 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:22000 flags:0x02/0x02
ACCEPT udp -- 10.222.66.235 0.0.0.0/0 udp dpt:22000
ACCEPT udp -- 10.222.66.235 0.0.0.0/0 udp dpt:22001
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpt:22001 flags:0x02/0x02
ACCEPT tcp -- 10.222.66.235 0.0.0.0/0 tcp dpts:20500:20502 flags:0x02/0x02
admin:show network cluster
10.222.66.234 cucmbe.bpc.local cucmbe Publisher authenticated
10.222.66.235 presence.bpc.local presence Subscriber authenticated using UDP since Wed Nov 24 12:03:18 2010
CUCM-BE is
Active Master Version: 8.5.0.96091-8
Presence is
Active Master Version: 8.0.2.10000-30
Regards,
Joe Bennett
Network Consultant
Bridge Point Communications
12-02-2010 02:36 PM
Hi Joe
I could not reproduce the issue using normal CUCM standalone servers (All have the NTP rule in the FW).
I am trying to install the CUCM-BE in one of my VMs and will try to reproduce the issue.
I will keep you posted.
In any case, if it is urgent you can open a TAC SR and we will handle it.
(If you open it during european hours, send me a private message with the SR number and i will try to grab it).
Thanks
Fernando
12-03-2010 05:44 AM
Hi Joe
I managed to install a CUCM-BE 8.5 and a CUPS 8.0.1
I found that the CUCM-BE as you said has the ntpd disabled and the firewall is closed.
During the installation of CUPS i was prompted to enter an NTP source, which could be any server. I didnt point it to the CUCM-BE i pointed to a real NTP.
Post install when i introduced the CUCM in the CUPS is when the NTP source changed to CUCM.
Apparently this is enforced automatically post-install.
Also at the same it is normal that the CUCM-BE doesnt have the NTP active, as it is not expecting anybody
There is one defect:
CSCtf06250 Can not specify NTP server in OS administration
That it is supposed to be fixed in CUPS 8.0.2 which is the one you have, so you should be able to change the NTP source.
But i believe this is a defect on the CUCM side, firewall is not opening the NTP ports. Although the CUPS is CLM validated.
At this point quickest way would be open a TAC SR so they can get root and open the NTP ports.
Regards
Fernando
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide