NTP not updating

[Charlie Jones]

Last night we upgraded one of our offices 3560 (WS-C3560-24PS) to 12.2(55) SE6.   With these switches, NTP is no longer working.   The NTP server is a router at this office.  Switches at this office that were not upgraded (3750) are getting NTP normally.  Below is a copy of the show ntp association and show ntp status. I have removed the ntp lines from the switch configuration and re-added them, and that has not changed anything.

Any suggestions?

      address         ref clock     st  when  poll reach  delay  offset    disp

~172.28.119.1     0.0.0.0          16     -    64    0     0.0    0.00  16000.

* master (synced), # master (unsynced), + selected, - candidate, ~ configured

Clock is unsynchronized, stratum 16, no reference clock

nominal freq is 119.2092 Hz, actual freq is 119.2092 Hz, precision is 2**17

reference time is 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

clock offset is 0.0000 msec, root delay is 0.00 msec

root dispersion is 0.00 msec, peer dispersion is 0.00 msec

I enabled NTP debugging for events, packets, and clock synchro.

.Jan  8 12:40:20.821 CST:  org D496E694.D23A9C1E (12:40:20.821 CST Tue Jan 8 2013)

.Jan  8 12:40:20.821 CST:  rec D496EA23.CD71F8DD (12:55:31.802 CST Tue Jan 8 2013)

.Jan  8 12:40:20.821 CST:  xmt D496EA23.CD72AD94 (12:55:31.802 CST Tue Jan 8 2013)

.Jan  8 12:40:20.821 CST:  inp D496E694.D2F5B99A (12:40:20.824 CST Tue Jan 8 2013)

.Jan  8 12:41:24.826 CST: NTP: xmit packet to 172.28.119.1:

.Jan  8 12:41:24.826 CST:  leap 3, mode 3, version 3, stratum 0, ppoll 64

.Jan  8 12:41:24.826 CST:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

.Jan  8 12:41:24.826 CST:  ref 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:41:24.826 CST:  org 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:41:24.826 CST:  rec 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:41:24.826 CST:  xmt D496E6D4.D380FC51 (12:41:24.826 CST Tue Jan 8 2013)

.Jan  8 12:41:24.826 CST: NTP: rcv packet from 172.28.119.1 to 10.19.0.111 on Vlan1:

.Jan  8 12:41:24.826 CST:  leap 0, mode 4, version 3, stratum 4, ppoll 64

.Jan  8 12:41:24.826 CST:  rtdel 0B6E (44.647), rtdsp 1A4B (102.707), refid AC1C9605 (172.28.150.5)

.Jan  8 12:41:24.826 CST:  ref D496EA59.756DBDF0 (12:56:25.458 CST Tue Jan 8 2013)

.Jan  8 12:41:24.826 CST:  org D496E6D4.D380FC51 (12:41:24.826 CST Tue Jan 8 2013)

.Jan  8 12:41:24.826 CST:  rec D496EA63.CEE98503 (12:56:35.808 CST Tue Jan 8 2013)

.Jan  8 12:41:24.826 CST:  xmt D496EA63.CEEA3978 (12:56:35.808 CST Tue Jan 8 2013)

.Jan  8 12:41:24.826 CST:  inp D496E6D4.D440210D (12:41:24.829 CST Tue Jan 8 2013)

.Jan  8 12:42:28.822 CST: NTP: xmit packet to 172.28.119.1:

.Jan  8 12:42:28.822 CST:  leap 3, mode 3, version 3, stratum 0, ppoll 64

.Jan  8 12:42:28.822 CST:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

.Jan  8 12:42:28.822 CST:  ref 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:42:28.822 CST:  org 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:42:28.822 CST:  rec 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:42:28.822 CST:  xmt D496E714.D3885867 (12:42:28.826 CST Tue Jan 8 2013)

.Jan  8 12:42:28.822 CST: NTP: rcv packet from 172.28.119.1 to 10.19.0.111 on Vlan1:

.Jan  8 12:42:28.822 CST:  leap 0, mode 4, version 3, stratum 4, ppoll 64

.Jan  8 12:42:28.822 CST:  rtdel 0B6F (44.662), rtdsp 1B72 (107.208), refid AC1C9605 (172.28.150.5)

.Jan  8 12:42:28.822 CST:  ref D496EA99.75D55A7C (12:57:29.460 CST Tue Jan 8 2013)

.Jan  8 12:42:28.822 CST:  org D496E714.D3885867 (12:42:28.826 CST Tue Jan 8 2013)

.Jan  8 12:42:28.822 CST:  rec D496EAA3.CF1E056B (12:57:39.809 CST Tue Jan 8 2013)

.Jan  8 12:42:28.822 CST:  xmt D496EAA3.CF1EBAA7 (12:57:39.809 CST Tue Jan 8 2013)

.Jan  8 12:42:28.822 CST:  inp D496E714.D43215B2 (12:42:28.828 CST Tue Jan 8 2013)

.Jan  8 12:43:32.819 CST: NTP: xmit packet to 172.28.119.1:

.Jan  8 12:43:32.819 CST:  leap 3, mode 3, version 3, stratum 0, ppoll 64

.Jan  8 12:43:32.819 CST:  rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)

.Jan  8 12:43:32.819 CST:  ref 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:43:32.819 CST:  org 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:43:32.819 CST:  rec 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

.Jan  8 12:43:32.819 CST:  xmt D496E754.D30205DD (12:43:32.824 CST Tue Jan 8 2013)

.Jan  8 12:43:32.819 CST: NTP: rcv packet from 172.28.119.1 to 10.19.0.111 on Vlan1:

.Jan  8 12:43:32.819 CST:  leap 0, mode 4, version 3, stratum 4, ppoll 64

.Jan  8 12:43:32.819 CST:  rtdel 0B73 (44.724), rtdsp 1AC6 (104.584), refid AC1C9605 (172.28.150.5)

.Jan  8 12:43:32.819 CST:  ref D496EAD9.7633439A (12:58:33.461 CST Tue Jan 8 2013)

.Jan  8 12:43:32.819 CST:  org D496E754.D30205DD (12:43:32.824 CST Tue Jan 8 2013)

.Jan  8 12:43:32.819 CST:  rec D496EAE3.CEC7C08E (12:58:43.807 CST Tue Jan 8 2013)

.Jan  8 12:43:32.819 CST:  xmt D496EAE3.CEC87126 (12:58:43.807 CST Tue Jan 8 2013)

.Jan  8 12:43:32.819 CST:  inp D496E754.D387299F (12:43:32.826 CST Tue Jan 8 2013)

[ALIAOF_]

What is your ntp config looks like?  Should be something like

ntp server x.x.x.x prefer

And you might have to define a source too like...

ntp source "interface"

[Charlie Jones]

Thanks for replying.  We have the ntp server x.x.x.x prefer but we did not have the ntp source command because we didn't need that prior to the upgrade.  I have added that line so we'll see what happens.

[paul driver]

Hello charlie,

It can take upto 15 mins to sync but i assume this time period has past prior to posting here -Can you post the switch config if applicable.

res

Paul

[Leo Laohoo]

NTP, on IOS version  12.2(55)SE6, takes about 5 minutes to sync.  On IOS 15.0(1)SE and 15.0(2)SE, however, takes 10 minutes.  Don't know why.  It just does.

Charlie:  Try another source.  Can you ping the NTP IP address?  I've got >300 switches running 12.2(55)SE6 and I have not seen a thing like this.  Our NTP "source" is our core 6500 switch.  What do you have?  A MS PDC? 

[Charlie Jones]

This is the NTP portion of the config:

ntp source Vlan1

ntp server x.x.x.x prefer.

Yes, I can ping the NTP IP.  The NTP source is a router that is at this same office.   I tried sourcing to another NTP server, and I get the same thing.  

[ALIAOF_]

Check the NTP version on the router "show ntp association detail" and same on the switch make sure they match.  Also something else setup time on the switch manually and then setup NTP again.  Some times if the clock is to far off it won't sync.

[Leo Laohoo]

Yes, I can ping the NTP IP.  The NTP source is a router that is at this same office.

is the router's NTP sync to a valid source?

[paul driver]

Charile -

1) peer to the ntp master source interface

2) make sure the ntp master source interface ip is accessible

3) The clock timezone requires to be set the same as the ntp server- - (eg: clock timezone GMT -0)

res

Paul

[Charlie Jones]

Hopefully I don't miss anyone's questions.  Mohammed, I think this will answer your questions:

Here is the output for "show ntp associations detail" from the router and the switch.

Router

172.28.150.5 configured, our_master, sane, valid, stratum 3
ref ID 151.153.1.2, time D497EEE7.8F83818D (07:28:07.560 CST Wed Jan 9 2013)
our mode client, peer mode server, our poll intvl 1024, peer poll intvl 64
root delay 40.25 msec, root disp 94.96, reach 377, sync dist 117.264
delay 4.36 msec, offset -0.0230 msec, dispersion 0.09
precision 2**24, version 3
org time D497F0B9.FF671C26 (07:35:53.997 CST Wed Jan 9 2013)
rcv time D497F0B9.FFF671D5 (07:35:53.999 CST Wed Jan 9 2013)
xmt time D497F099.067E0033 (07:35:21.025 CST Wed Jan 9 2013)
filtdelay =     4.36    4.68    4.33    4.24    4.36    4.38    4.27    4.43
filtoffset =   -0.02   -0.17    0.01   -0.02   -0.01   -0.03   -0.03   -0.16
filterror =     0.02    0.99    1.97    2.94    3.92    4.90    5.87    6.85

172.28.199.20 configured, selected, sane, valid, stratum 4
ref ID 172.28.150.5, time D497EED4.9020C628 (07:27:48.563 CST Wed Jan 9 2013)
our mode client, peer mode server, our poll intvl 1024, peer poll intvl 1024
root delay 58.21 msec, root disp 176.12, reach 377, sync dist 212.311
delay 13.44 msec, offset -5.6867 msec, dispersion 0.37
precision 2**6, version 3
org time D497F0B4.06E978E8 (07:35:48.027 CST Wed Jan 9 2013)
rcv time D497F0B4.0A16C081 (07:35:48.039 CST Wed Jan 9 2013)
xmt time D497F0B4.06A58E08 (07:35:48.025 CST Wed Jan 9 2013)
filtdelay =    13.44   13.43   13.32   13.40   13.37   13.40   13.40   13.43
filtoffset =   -5.69   -6.20   -5.65   -6.17   -5.67   -6.18   -5.70   -6.21
filterror =     0.02    0.99    1.97    2.94    3.92    4.90    5.87    6.85

Switch

172.28.119.1 configured, insane, invalid, unsynced, stratum 16

ref ID 0.0.0.0, time 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64

root delay 0.00 msec, root disp 0.00, reach 0, sync dist 573.318

delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00

precision 2**5, version 3

org time 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

rcv time 00000000.00000000 (18:00:00.000 CST Thu Dec 31 1899)

xmt time D497EDD4.D2ED88FD (07:23:32.823 CST Wed Jan 9 2013)

filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

To answer the question about setting the time manually.  After the IOS upgrade, all of these 3560 switches reset their clocks as if they were new switches (Feb 28, 1993 was the date I believe).   The switch I have been working on I manually changed the time and date to be current less than 15 minutes.  I was hoping that by getting it to the same date and close to the actual time, it would pick up the extra time, and it hasn't.

I have also removed the ntp information from the switch and re-added, and it does not update.

-------------------

Paul,

I checked the time zone settings on the router and the switch, and those match including the offset (CST -6).  I also verified the source interface is accessible in that I can ping it from this switch (the source interface is a loopback interface on the router).  This is the configuration I have on the switch for NTP followed by the configuration on the router.  I noticed on the switch, there is no ntp clock-period line in the configuration.  My assumption is that is automatically set rather than something I need to configure, is that correct?

ntp source Vlan1

ntp server 172.28.119.1 prefer

ntp clock-period 17180271

ntp source Loopback0

ntp server 172.28.150.5

ntp server 172.28.199.20

[Charlie Jones]

I have an update.  I decided to change the ntp source on this switch to point to our NTP source back at our datacenter rather than to the router.   When I did that, the time updated.  

Since I was thinking the problem was with the router, I took a switch that was not upgraded at this site, and I rolled the clock back 10 minutes to see if it would update using the router as its source.   After about 10-15 minutes, the time updated.

I feel like I am back to square one.  Im glad to see that something is working, I can't find why upgrading and reloading the switch would cause this problem.   I'd understand it if I couldn't ping the NTP source or not get to it, but I can. 

We upgraded and reloaded 50 switches, and the reloads all occcurred at the same time. Could that have caused a problem?  Just throwing it out there.

[paul driver]

Hello charlie,

Just noticed you comment - I decided to change the ntp source on this switch to point to our NTP source back at our datacenter rather than to the router

I was on the asumption the router you was trying to sync with was the ntp master - apologies. Then maybe the router requires the ntp peer statement so other clients can syc with it.

ntp peer xxxx( this will be you datacenter ntp master) - My understanding of this statement is that it alows other clients to peer from itself.

Worth a shot!

res

Paul

[ALIAOF_]

Just curious have you tried to setup the time manually and just like one or two min off and then setup ntp?

[Richard Burts]

I am experiencing a very similar problem on a 3750 switch running 12.2(55). We were running 12.2(25) originally when we saw the problem and upgraded code. This did not solve the issue. The switch was configured with ntp server pointing to a router at the same location. The router is configured with ntp server pointing to an Internet time source. The router is correctly in sync with its server. The router is responding to requests from the switch. But most of the time the switch is not in sync with the router.

I have checked the obvious things: the switch config correctly specifies the router as ntp server, there is IP connectivity between the switch and the router, the router is correctly learning NTP time from an Internet time server, the switch time is set manually to be right. But most of the time the switch does not sync with the router. I have found that configuring a second time server seems to be effective in getting the switch to sync with a time server.

I have a case open with TAC on this. If we find the source of the problem I will update the forum.

HTH

Rick

[Charlie Jones]

More odd behavior from these switches....

I had completely looked over this in the switch configuration, but on all of the upgraded switches, the ip default-gateway was missing.   Unfortunately, I added the statement to the configuration and it hasn't made a difference.   I've tried the following on different switches that NTP is not working on:

Added the DG

Update clock time manually w/in 10 minutes

Added the DG

removed any ntp settings

updated the clock within 10 minutes of actual time

Added ntp settings

Added the DG

Left the time as it was (1993 date)

Removed NTP settings

Added DG

set clock to real time

Added NTP server

For all of these, none of them show as synchronizing with the NTP server when I do a "show ntp status" or "show ntp association"