NTP synch issues when upgrading to IOS 12.4(24)T1

[kst.amand]

After upgrading to 12.4(24)T1 from 12.4(15)T3 we experienced time synch issues.

Topology is 2 IOS Routers as source for NTP at different Remote WAN sites over MPLS.

Configuration is;

ntp source loopback0

ntp server x.x.x.x prefer

ntp server x.x.x.x

Prior to the upgrade ntp assoc detail would look like this;

router#sh ntp assoc det

x.x.x.x configured, selected, sane, valid, stratum 4

ref ID y.y.y.y, time CE4B7F70.D5A2BC66 (07:58:40.834 edt Fri Sep 4 2009)

our mode client, peer mode server, our poll intvl 64, peer poll intvl 64

root delay 7.26 msec, root disp 2.99, reach 377, sync dist 14.374

delay 15.08 msec, offset 1998.0521 msec, dispersion 0.21

precision 2**18, version 3

org time CE4B80B9.B1F95635 (08:04:09.695 edt Fri Sep 4 2009)

rcv time CE4B80B7.B4671B16 (08:04:07.704 edt Fri Sep 4 2009)

xmt time CE4B80B7.B08A7AA9 (08:04:07.689 edt Fri Sep 4 2009)

filtdelay = 15.08 15.27 17.52 15.21 15.29 15.09 18.81 15.09

filtoffset = 1998.05 1998.01 1999.29 1998.05 1998.14 1998.06 1999.89 1997.65

filterror = 0.02 0.99 1.97 2.94 3.92 4.90 5.87 6.85

a.a.a.a configured, our_master, sane, valid, stratum 4

ref ID b.b.b.b, time CE4B8051.4CDCFBC4 (08:02:25.300 edt Fri Sep 4 2009)

our mode client, peer mode server, our poll intvl 64, peer poll intvl 64

root delay 6.93 msec, root disp 2.56, reach 377, sync dist 16.922

delay 19.55 msec, offset -1.0158 msec, dispersion 0.70

precision 2**18, version 3

org time CE4B809B.B27CD6FB (08:03:39.697 edt Fri Sep 4 2009)

rcv time CE4B809B.B54027DA (08:03:39.708 edt Fri Sep 4 2009)

xmt time CE4B809B.B03DF2ED (08:03:39.688 edt Fri Sep 4 2009)

filtdelay = 19.55 21.30 20.05 27.33 28.35 19.12 20.68 20.68

filtoffset = -1.02 -0.13 -0.80 2.83 -4.60 -1.28 -1.23 -1.45

filterror = 0.02 0.99 1.97 2.94 3.92 4.90 5.87 6.85

After the upgrade, it will work for a short while and then fail. the ntp assoc det looks like this after it fails;

router#sh ntp assoc det

x.x.x.x configured, sane, invalid, stratum 4

ref ID y.y.y.y , time CE4B83B2.51C2635A (08:16:50.319 edt Fri Sep 4 2009)

our mode client, peer mode server, our poll intvl 64, peer poll intvl 128

root delay 0.20 msec, root disp 202.57, reach 177, sync dist 0.60

delay 0.00 msec, offset -2033.6347 msec, dispersion 64.99

precision 2**18, version 4

org time CE4B83C8.E5B10FC8 (08:17:12.897 edt Fri Sep 4 2009)

rec time CE4B83CA.F3DB1883 (08:17:14.952 edt Fri Sep 4 2009)

xmt time CE4B83CA.E8BEE419 (08:17:14.909 edt Fri Sep 4 2009)

filtdelay = 0.04 0.04 0.04 0.04 0.04 0.04 0.04 0.00

filtoffset = -2.03 -2.03 -2.03 -2.03 -2.04 -2.04 -2.04 0.00

filterror = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16.00

minpoll = 6, maxpoll = 10

a.a.a.a configured, sane, invalid, stratum 4

ref ID b.b.b.b , time CE4B8170.D889053F (08:07:12.845 edt Fri Sep 4 2009)

our mode client, peer mode server, our poll intvl 128, peer poll intvl 128

root delay 0.00 msec, root disp 1.83, reach 177, sync dist 0.10

delay 0.00 msec, offset -35.1710 msec, dispersion 65.72

precision 2**18, version 4

org time CE4B83A7.E3A8FC5F (08:16:39.889 edt Fri Sep 4 2009)

rec time CE4B83A7.F07E9267 (08:16:39.939 edt Fri Sep 4 2009)

xmt time CE4B83A7.E8D4B335 (08:16:39.909 edt Fri Sep 4 2009)

filtdelay = 0.02 0.03 0.03 0.03 0.03 0.03 0.02 0.00

filtoffset = -0.03 -0.03 -0.03 -0.04 -0.04 -0.04 -0.04 0.00

filterror = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16.00

minpoll = 6, maxpoll = 10

sh ntp assoc

address ref clock st when poll reach delay offset disp

x~x.x.x.x y.y.y.y 4 52 64 377 0.000 -2033.6 4.267

x~a.a.a.a b.b.b.b 4 22 64 377 0.000 -31.080 3.204

* sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

Am I missing something?

[Edison Ortiz]

You may be affected by this bug:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsx21595

If you really source from the loopback, I recommend going with T2 or 12.4(15)T9.

__

Edison.

[kst.amand]

12.4.(24)T1 was needed to correct a different problem - missing userid on failed telnet logins. That one we need for audit purposes.

I was able to confirm with ntp debug info that the packets are being sourced from the loopback - so it looks like ntp source has not been dropped from the config.

Any other thoughts?

[Edison Ortiz]

I was able to duplicate you situation and it seems due to the new NTP version 4 incorporated in T1 the behavior has changed. It takes around 15 minutes for the time to sync after the router has been reloaded. With NTP version 3, the process was a lot faster.

How long has it been since you last check the NTP status?

Do you have the bugid for missing userid on failed attempts? I believe that was corrected with T9.

__

Edison.

[kst.amand]

I have gone up to 2 days and haven't seen the NTP get synched. For this one particular router I have psted the output in for, it has been over 2 hours now and still no time synch

As to the bugid, I do not have it. I do know that at 12.4(24)T and I had the problem. SSH connections would show the userid, but not Telnet.

I hope this helps.

[Edison Ortiz]

Can you remove the NTP config and create the config again on this router - then wait 15 to 20 minutes for sync?

Can you also use another source interface and see if the problem is with the loopback being the source?

[kst.amand]

Same results - still no time synch after almost 30 minutes.

[Edison Ortiz]

I recommend opening a TAC case. Perhaps you have a new NTP bug on that release. Your config looks fine.

__

Edison.