ASR 9006 NTP sync issue

[Thijs Van Severen]

Hi all

i'm having NTP sync issues on a ASR 9006

the log shows this :

RP/0/RSP0/CPU0:Apr 29 20:15:14.786 : ntpd[256]: %IP-IP_NTP-5-HP_CONN_RECOVERED : High priority NTP peer connection recovered - Stratum 15->1. 
LC/0/2/CPU0:Apr 29 21:01:51.965 : ntpdc[203]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : DLRSC node : The clock was stepped and needs to be resynced 
LC/0/0/CPU0:Apr 29 21:03:07.460 : ntpdc[203]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : DLRSC node : The clock was stepped and needs to be resynced 
LC/0/1/CPU0:Apr 29 21:03:41.777 : ntpdc[209]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : DLRSC node : The clock was stepped and needs to be resynced 
RP/0/RSP0/CPU0:Apr 29 21:06:36.796 : ntpd[256]: %IP-IP_NTP-5-SYNC_LOSS : Synchronization lost : 10.81.254.202 : The clock was stepped and needs to be resynced 
RP/0/RSP0/CPU0:Apr 29 21:06:36.796 : ntpd[256]: %IP-IP_NTP-5-HP_CONN_LOST : High priority NTP peer connection lost - Stratum 1->15. 
RP/0/RSP0/CPU0:Apr 29 21:06:36.796 : ntpd[256]: %IP-IP_NTP-5-ALL_CONN_LOST : All NTP peer connections failed.

these are the ntp associations :
      address         ref clock     st  when  poll reach  delay  offset    disp
 ~10.81.254.202    .USER.           16  42083  1024    0    0.00   0.000   15937
 ~10.64.58.51      .USER.           16     -  1024    0    0.00   0.000   15937
 ~10.64.58.50      .USER.           16   19d  1024    0    0.00   0.000   15937
 ~144.254.15.122   .USER.           16     -  1024    0    0.00   0.000   15937
 ~72.163.32.43     .USER.           16  42025  1024    0    0.00   0.000   15937

ntp status :

Thu Apr 30 08:02:41.810 CET
Clock is unsynchronized, stratum 16, no reference clock
nominal freq is 1000.1530 Hz, actual freq is 1000.1530 Hz, precision is 2**10
reference time is D8EBB311.FDD077F0 (20:44:17.991 CET Wed Apr 29 2015)
clock offset is 302.073 msec, root delay is 0.000 msec
root dispersion is 589.38 msec, peer dispersion is 0.00 msec
loopfilter state is 'NSET' (Never set), drift is 0.0000000000 s/s
system poll interval is 64, last update was 40704 sec ago

the NTP servers i'm using are ok (other systems can sync to them)

i have seen this 1 or 2 times before, but the sync was restored after a while

this time the sync issue remains

this is my ntp config :

ntp
 server 10.64.58.50
 server 10.64.58.51
 server 10.81.254.202
 server 72.163.32.43
 server 144.254.15.122
 master 2
 access-group ipv4 peer PeerNTP
 access-group ipv4 serve-only NoExtNTP
!
!
ipv4 access-list PeerNTP
 25 permit ipv4 host 10.81.254.202 any
 26 permit ipv4 host 144.254.15.122 any
 27 permit ipv4 host 72.163.32.43 any
 30 permit ipv4 host 10.64.58.50 any
 40 permit ipv4 host 10.64.58.51 any
!         
ipv4 access-list NoExtNTP
 10 permit ipv4 10.50.200.0 0.0.3.255 any
 20 permit ipv4 10.50.232.0 0.0.3.255 any
 25 permit ipv4 host 10.81.254.202 any
 26 permit ipv4 host 144.254.15.122 any
 27 permit ipv4 host 72.163.32.43 any
 30 permit ipv4 host 10.64.58.50 any
 40 permit ipv4 host 10.64.58.51 any

i found this bug report : https://tools.cisco.com/bugsearch/bug/CSCsz22456/?referring_site=bugquickviewredir

but this is an old report and it does not apply to the ASR, but the output on the log is very similar

this makes me wonder if this is a known issue, and if i should upgrade to fix it

all tips are welcome

Grtz

Thijs

i'm currently running Cisco IOS XR Software, Version 4.2.3

[Thijs Van Severen]

below is the output of 'show ntp associations detail'

the output claims that all configured servers are "configured, insane, invalid, unsynced, stratum 16"

??

at least 2 of the servers are ok (i have other devices that sync to these servers and are ok)

10.81.254.202 configured, insane, invalid, unsynced, stratum 16
ref ID .USER., time D8EBB80D.64FF309B (21:05:33.394 CET Wed Apr 29 2015)
our mode client, peer mode server, our poll intvl 1024, peer poll intvl 1024
root delay 0.0000 msec, root disp 320.43, reach 0, sync dist 15948.3754
delay 0.00 msec, offset 0.000 msec, dispersion 15937.500
precision 2**20, version 4
org time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
rcv time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
xmt time D8EC966A.87A37C12 (12:54:18.529 CET Thu Apr 30 2015)
filtdelay =    0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filtoffset =   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

10.64.58.51 configured, insane, invalid, unsynced, stratum 16
ref ID .USER., time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
our mode client, peer mode unspec, our poll intvl 1024, peer poll intvl 1024
root delay 0.0000 msec, root disp 0.00, reach 0, sync dist 15948.3550
delay 0.00 msec, offset 0.000 msec, dispersion 15937.500
precision 2**10, version 4
org time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
rcv time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
xmt time D8EC9656.7E2EFC64 (12:53:58.492 CET Thu Apr 30 2015)
filtdelay =    0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filtoffset =   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

10.64.58.50 configured, insane, invalid, unsynced, stratum 16
ref ID .USER., time D8D29C7D.27FBC664 (20:01:17.156 CET Fri Apr 10 2015)
our mode client, peer mode server, our poll intvl 1024, peer poll intvl 1024
root delay 0.0000 msec, root disp 320.43, reach 0, sync dist 15948.3304
delay 0.00 msec, offset 0.000 msec, dispersion 15937.500
precision 2**20, version 4
org time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
rcv time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
xmt time D8EC966D.890E8F39 (12:54:21.535 CET Thu Apr 30 2015)
filtdelay =    0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filtoffset =   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

144.254.15.122 configured, insane, invalid, unsynced, stratum 16
ref ID .USER., time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
our mode client, peer mode unspec, our poll intvl 1024, peer poll intvl 1024
root delay 0.0000 msec, root disp 0.00, reach 0, sync dist 15948.2350
delay 0.00 msec, offset 0.000 msec, dispersion 15937.500
precision 2**10, version 4
org time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
rcv time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
xmt time D8EC965E.81F72F76 (12:54:06.507 CET Thu Apr 30 2015)
filtdelay =    0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filtoffset =   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0
          
72.163.32.43 configured, insane, invalid, unsynced, stratum 16
ref ID .USER., time D8EBB841.3A0D0B4F (21:06:25.226 CET Wed Apr 29 2015)
our mode client, peer mode server, our poll intvl 1024, peer poll intvl 1024
root delay 0.0000 msec, root disp 381.47, reach 0, sync dist 15948.9165
delay 0.00 msec, offset 0.000 msec, dispersion 15937.500
precision 2**20, version 4
org time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
rcv time 00000000.00000000 (01:00:00.000 CET Thu Jan  1 1970)
xmt time D8EC964A.7882AFC8 (12:53:46.470 CET Thu Apr 30 2015)
filtdelay =    0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filtoffset =   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0
          
127.127.1.1 configured, our_master, stratum 1
ref ID .LOCL., time D8EC9879.804E9A1F (13:03:05.501 CET Thu Apr 30 2015)
our mode active, peer mode passive, our poll intvl 64, peer poll intvl 64
root delay 0.0000 msec, root disp 0.00, reach 377, sync dist 5.5571
delay 0.00 msec, offset 0.000 msec, dispersion 1.916
precision 2**10, version 4
org time D8EC9879.804E9A1F (13:03:05.501 CET Thu Apr 30 2015)
rcv time D8EC9879.804E9A1F (13:03:05.501 CET Thu Apr 30 2015)
xmt time D8EC9879.804E9A1F (13:03:05.501 CET Thu Apr 30 2015)
filtdelay =    0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filtoffset =   0.000   0.000   0.000   0.000   0.000   0.000   0.000   0.000
filterror =    0.977   1.952   2.942   3.902   4.892   5.852   6.812   7.772

[Thijs Van Severen]

tried several things:

ntp clear all

ntp reset drift

removed the server acl (so i'm not blocking anything)

> no effect

then i removed a server from the list (no ntp server x.x.x.x  + commit)

added it back to the list + commit

i can see the ref clock go to INIT a bit later GPS, after a while it syncs, and after a couple of minutes it breaks again ...

currently i removed all but 1 NTP server, configured that one as preferred, defined the interface for it (mgmt eth0) and set the mode to iburst

for now that seems to work ... hope it sticks

[Thijs Van Severen]

in the last few days i have had 2/3 occasions where the ASR lost sync with the NTP server

i guess that it is normal that this happens now and then, but i assumed that the ASR would just keep running and since it is configured as 'master 2' i assumed that the clients that sync to this ASR would never notice this

but they do : there seems to be an offset jump