Re: ntp masters do not sync

katerina.dardoufa · ‎02-28-2011

Hello,

I have two ntp masters that receive their clock from the internet. Both seem to have lost their sync, as can be seen from the following output.

Sw-MK-C6509-R-IS-MHX-A1#sh ntp associations detail
127.127.7.1 configured, our_master, sane, valid, stratum 7
ref ID 127.127.7.1, time D11617E3.E23FD5C2 (14:47:31.883 EET Mon Feb 28 2011)
our mode active, peer mode passive, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 377, sync dist 0.504
delay 0.00 msec, offset 0.0000 msec, dispersion 0.02
precision 2**18, version 3
org time D11617E3.E23FD5C2 (14:47:31.883 EET Mon Feb 28 2011)
rcv time D11617E3.E23FD5C2 (14:47:31.883 EET Mon Feb 28 2011)
xmt time D11617E3.E23F852B (14:47:31.883 EET Mon Feb 28 2011)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =     0.02    0.99    1.97    2.94    3.92    4.90    5.87    6.85
Reference clock status: Running normally
Timecode:

129.132.98.11 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 1024, peer poll intvl 1024
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 196460.159
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
rcv time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
xmt time D1161718.E1F1D41A (14:44:08.882 EET Mon Feb 28 2011)
filtdelay = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filtoffset = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filterror = 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

193.93.167.241 configured, insane, invalid, stratum 2
ref ID 193.93.167.239, time D10EE2C0.DC84F4FE (03:34:56.861 EET Wed Feb 23 2011)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 128
root delay 0.37 msec, root disp 19.87, reach 0, sync dist 7220.215
delay 2.38 msec, offset -4.7321 msec, dispersion 16000.00
precision 2**20, version 3
org time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
rcv time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
xmt time D11617D7.E23AF02F (14:47:19.883 EET Mon Feb 28 2011)
filtdelay = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filtoffset = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filterror = 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

130.149.17.8 configured, insane, invalid, stratum 1
ref ID .PPS., time D10EE50D.BEA734B7 (03:44:45.744 EET Wed Feb 23 2011)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 128
root delay 0.00 msec, root disp 0.43, reach 0, sync dist 7233.063
delay 67.31 msec, offset -0.1017 msec, dispersion 16000.00
precision 2**27, version 3
org time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
rcv time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
xmt time D11617DD.E23E371A (14:47:25.883 EET Mon Feb 28 2011)
filtdelay = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filtoffset = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filterror = 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

193.218.254.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 1024, peer poll intvl 1024
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 196460.159
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
rcv time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
xmt time D1161718.E1F7909D (14:44:08.882 EET Mon Feb 28 2011)
filtdelay = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filtoffset = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
filterror = 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

ntp source Loopback1
ntp master
ntp server 129.132.98.11
ntp server 193.93.167.241
ntp server 130.149.17.8 prefer
ntp server 193.218.254.1

***************************************************

Sw-KK-C6509-R4A1#sh ntp associations detail
127.127.7.1 configured, our_master, sane, valid, stratum 7
ref ID 127.127.7.1, time D11613D8.D91FA03C (14:30:16.848 EET Mon Feb 28 2011)
our mode active, peer mode passive, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.00, reach 377, sync dist 0.031
delay 0.00 msec, offset 0.0000 msec, dispersion 0.02
precision 2**18, version 3
org time D11613D8.D91FA03C (14:30:16.848 EET Mon Feb 28 2011)
rcv time D11613D8.D91FA03C (14:30:16.848 EET Mon Feb 28 2011)
xmt time D11613D8.D91F3829 (14:30:16.848 EET Mon Feb 28 2011)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror =     0.02    0.99    1.97    2.94    3.92    4.90    5.87    6.85
Reference clock status: Running normally
Timecode:

130.149.17.8 configured, insane, invalid, stratum 1
ref ID .PPS., time D1159C3E.BEC4724F (05:59:58.745 EET Mon Feb 28 2011)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 128
root delay 0.00 msec, root disp 0.38, reach 0, sync dist 1250.244
delay 63.72 msec, offset -0.2481 msec, dispersion 16000.00
precision 2**27, version 3
org time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
rcv time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
xmt time D11613C7.D91C06A6 (14:29:59.848 EET Mon Feb 28 2011)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror = 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

193.218.254.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 1024, peer poll intvl 1024
root delay 0.00 msec, root disp 0.00, reach 0, sync dist 8229.263
delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00
precision 2**5, version 3
org time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
rcv time 00000000.00000000 (02:00:00.000 EET Mon Jan 1 1900)
xmt time D116155F.D969700E (14:36:47.849 EET Mon Feb 28 2011)
filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
filterror = 16000.0 16000.

ntp source Loopback1
ntp master
ntp server 130.149.17.8 prefer
ntp server 193.218.254.1

******************************************************************************

Usually both received their sync from 130.149.17.8. On 22/2 I accidentally removed and re-entered the "ntp master" command from Sw-KK-C6509-R4A1. Could that have something to do with the devices losing sync (Sw-MK-C6509-R-IS-MHX-A1 lost sync on 23/2 and Sw-KK-C6509-R4A1 on 28/2)??? (maybe this is a silly question, but I read somewhere that when issuing the ntp master command the device should be rebooted!).

I also opened "debug ntp select" on Sw-MK-C6509-R-IS-MHX-A1 and what I got was:

Feb 28 14:35:47 EET: NTP: nlist 1, allow 0, found 0, low -0.000015, high 0.000015
Feb 28 14:35:47 EET: NTP: candidate 127.127.7.1 cdist 112.000015 error 0.000015
Feb 28 14:35:47 EET: NTP: survivor 127.127.7.1 offset 0.000000, cdist 112.00002
Feb 28 14:36:12 EET: NTP: nlist 1, allow 0, found 0, low -0.000397, high 0.000397
Feb 28 14:36:12 EET: NTP: candidate 127.127.7.1 cdist 112.000397 error 0.000397
Feb 28 14:36:12 EET: NTP: survivor 127.127.7.1 offset 0.000000, cdist 112.00040
Feb 28 14:36:51 EET: NTP: nlist 1, allow 0, found 0, low -0.000015, high 0.000015
Feb 28 14:36:51 EET: NTP: candidate 127.127.7.1 cdist 112.000015 error 0.000015
Feb 28 14:36:51 EET: NTP: survivor 127.127.7.1 offset 0.000000, cdist 112.00002
Feb 28 14:37:16 EET: NTP: nlist 1, allow 0, found 0, low -0.000397, high 0.000397
Feb 28 14:37:16 EET: NTP: candidate 127.127.7.1 cdist 112.000397 error 0.000397
Feb 28 14:37:16 EET: NTP: survivor 127.127.7.1 offset 0.000000, cdist 112.00040
Feb 28 14:37:55 EET: NTP: nlist 1, allow 0, found 0, low -0.000015, high 0.000015
Feb 28 14:37:55 EET: NTP: candidate 127.127.7.1 cdist 112.000015 error 0.000015
Feb 28 14:37:55 EET: NTP: survivor 127.127.7.1 offset 0.000000, cdist 112.00002
Feb 28 14:38:20 EET: NTP: nlist 1, allow 0, found 0, low -0.000397, high 0.000397
Feb 28 14:38:20 EET: NTP: candidate 127.127.7.1 cdist 112.000397 error 0.000397
Feb 28 14:38:20 EET: NTP: survivor 127.127.7.1 offset 0.000000, cdist 112.00040
Feb 28 14:38:59 EET: NTP: nlist 1, allow 0, found 0, low -0.000015, high 0.000015

no attempt is done to select another master! I know that configuration on F/W towards the internet has not changed, so both devices should be able to connect to 130.149.17.8!

How can I trigger the devices to select another master? Debug ntp sync and refclock gave no output, but I only left the debug open for 5 minutes. How often does a master device try to sync???

Any help would be highly appreciated

Katerina

katerina.dardoufa · ‎03-01-2011

Ok... After thoroughly searching for an answer I came accross this really interesting post in the Cisco Learning Network.

https://learningnetwork.cisco.com/message/41918

It seems that in order for the ntp masters to regain sync, one needs to remove and re-enter the IPs of the public servers from which they receive their clock.

After doing exactly that, both my masters are now working fine. My question still is why they sometimes lose sync. The attached post states :

1) "if a router is synchronized to an outside time source via NTP, it is a good idea to periodically update the calendar with the time learned from NTP. Otherwise, the calendar will tend to gradually lose or gain time. I don't think that there is command for resynchronizatio"

2) use the 'ntp update-calendar' command in order to update the component calendar periodically from the time source."

Is this correct? If I use "ntp update-calendar" command on the masters will it solve the problem?

Thanks is advance,

Katerina

Leo Laohoo · ‎03-01-2011

ntp master

Take this line off your config. This line basically tells everyone, including the low-stratum-number clocks to "trust me". Cisco doesn't like recommending this line unless it's really, really necessary.

katerina.dardoufa · ‎03-01-2011

The two 6509 act as the ntp masters for all the rest network devices, as well as the Domain servers. If I remove the command how will all these network devices sync their clocks? Do you propose to install a ntp server (actual server) and have all all devices configured with this as their master???

Leo Laohoo · ‎03-02-2011

If I remove the command how will all these network devices sync their clocks?

It won't.

Do you propose to install a ntp server (actual server) and have all all devices configured with this as their master???

Yes and no.

If you have the finance, then yes. Get a "true" NTP server(s). A "true" NTP server synchronizes to the GPS. There are vendors who claim that they are a "true" NTP as long as you can "give the appliance access to the internet". This is NOT an NTP server.

If you don't have the finance, then poke a hole in the firewall to allow a few devices to synchronize with internet-based NTP servers.

Either way, it works equally as well.