NTP server problem

[Dennis Beul]

Hello,

I have a problem with the time synchronization via NTP between a Catalyst 2960 and Catalyst 6509. When I configure the 6509 switch as a NTP reference on the 2960, it does not synchronize with the 6509's NTP server. There is no reachability or ACL-related issue between both switches.

As soon as I configure a second Catalyst 6509 (which is completely identical to the other 6509 and in the same subnet) as a NTP server for the 2960, the time sync with the second 6509 happens immediality.

The first 6509 switch works as a NTP reference for at least 50 other switches and routers in the network - so why not for this one more switch? I checked some "debug ntp packet" and "debug ntp events" outputs and can clearly watch the NTP requests going out of the 2960, but on the 6509 just nothing happens - no debug outputs for this specific 2960, while requests from other devices come in all the time.

Maybe you have already experienced this strange behaviour in the past or got some deeper knowledge in the Cisco NTP server implementation. I could think of some sort of "maximum client limit" in the IOS NTP server, but could not find any mechanism like this in the standard NTP specification. Eventually, you can approve that this is a IOS-specific issue.

Any help or hint is highly appreciated. Thank you!

Regards

Dennis

[Marvin Rhoads]

I'm not aware of any limitation on the number of NTP clients for the IOS-implemented NTP server. It doesn't keep track of those things, AFAIK.

Since the debug on the non-working 6509 doesn't show any packets from the switch, can you perhaps span the port and look at the output in Wireshark to verify that they are indeed arriving from the switch? Perhaps the NTP requests from that particular 2960 are not making it to the 6509 for some strange reason.

[Dennis Beul]

Thank you for your feedback!

I will try to span the port between both switches and check for NTP packets.

But to be honest, I have low expectations to find a hint by this, because NTP works with the secondary 6509 switch and the packets have to travel through the primary 6509 to reach the secondary 6509 (the 2960 is directly connected to the primary 6509).

So I'm almost sure that it's not a reachability issue

[Leo Laohoo · Accepted solution]

All of my distro and access switches go back to a pair of 6500.  Both do NOT have "ntp master" on them but their time source are synchronized.

[Dennis Beul]

In this case, we have a pair of Catalyst 6509 too, running in HSRP mode with a common HSRP IP address. We normally just use the HSRP IP address for NTP sync, but in this case I used single IP addresses of each switch for troubleshooting.

[Marvin Rhoads]

You aren't pointing your requesting switch to the active HSRP address are you? You need to use the actual destination switch IP address as that is where the NTP replies will come from. The replies need to come from the same IP address as the requests are destined for since that is what the supplicant expects.

[Dennis Beul]

No, I leave HSRP completely out of this problem, just using single addresses of each switch. So I think this has nothing to do with HSRP .... as already said, I did the same on at least 50 other devices without any problems.

[Martin Ermel]

how exactly does your ntp configuration looks like? Are you using a key?

Can you post that config snippet here along with the output of a "show ntp status" on both the 6509 and the 2960?

[Dennis Beul]

The NTP configuration is very simple, no authentication or something, just "ntp server 10.x.x.x", that's all.

"show ntp status" outputs on 2960:

Cat2960#sh ntp status

Clock is unsynchronized, stratum 16, no reference clock

nominal freq is 119.2092 Hz, actual freq is 119.2082 Hz, precision is 2**17

reference time is D39D706B.8BCC985B (15:21:47.546 CEST Tue Jul 3 2012)

clock offset is -0.9954 msec, root delay is 31.27 msec

root dispersion is 7.87 msec, peer dispersion is 1.02 msec

Cat2960#sh ntp associations

      address         ref clock     st  when  poll reach  delay  offset    disp

~10.x.x.x     0.0.0.0          16     -    64    0     0.0    0.00  16000.

* master (synced), # master (unsynced), + selected, - candidate, ~ configured

Cat2960#sh ntp associations detail

10.x.x.x configured, insane, invalid, unsynced, stratum 16

ref ID 0.0.0.0, time 00000000.00000000 (01:00:00.000 CET Mon Jan 1 1900)

our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64

root delay 0.00 msec, root disp 0.00, reach 0, sync dist 0.000

delay 0.00 msec, offset 0.0000 msec, dispersion 16000.00

precision 2**5, version 3

org time 00000000.00000000 (01:00:00.000 CET Mon Jan 1 1900)

rcv time 00000000.00000000 (01:00:00.000 CET Mon Jan 1 1900)

xmt time D39D7112.89D810D5 (15:24:34.538 CEST Tue Jul 3 2012)

filtdelay =     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

filtoffset =    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00

filterror =  16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0 16000.0

"show ntp status" output on 6509:

Cat6509#sh ntp status

Clock is synchronized, stratum 3, reference is X.X.X.X

nominal freq is 250.0000 Hz, actual freq is 249.9970 Hz, precision is 2**18

reference time is D3A0134A.09E11B41 (15:21:14.038 CEST Thu Jul 5 2012)

clock offset is 0.8266 msec, root delay is 22.77 msec

root dispersion is 9.37 msec, peer dispersion is 0.67 msec

[Martin Ermel · Accepted solution]

Can you also post the output of the following commands:

cat6509:

show ntp assoc detail

show clock detail

show run | i clock

cat2960

show clock detail

show run | i clock

currently I assume the following:

the 2960 declares the ntp server as insane and invalid, perhaps because he does not see the reference clock for the ntp server which should be there:

Cat2960#sh ntp associations detail

10.x.x.x configured, insane, invalid, unsynced, stratum 16

ref ID 0.0.0.0, time 00000000.00000000 (01:00:00.000 CET Mon Jan 1 1900)

[...]

as a consequence the ntp server is declared as unsynced and gets a stratum 16 and your 2960 will not accept any update from this source.

When you use the second cat6509 (which works) what is the reference clock for this one?

[Alessio Andreoli]

Hi ,

Remember to set the master NTP in your network or it can sync with sombody else... Stratum 16 was also teh line that let you think about an issue. You generally shouldn't go over the stratuum level 4

HTH  

Alessio

[Dennis Beul]

Thanks for the feedback! I will check that reference clock point, that sounds pretty possible ... unfortunately, I have no access to the affected network today, so I will deliver the requested outputs on Monday.

Have a nice weekend guys.

[Dennis Beul]

Hi, here are the requested outputs:

Cat2960#sh clock detail
.09:24:57.931 CEST Tue Jul 10 2012
Time source is NTP
Summer time starts 02:00:00 CET Sun Mar 25 2012
Summer time ends 03:00:00 CEST Sun Oct 28 2012

Cat2960#show run | i clock
clock timezone CET 1
clock summer-time CEST recurring last Sun Mar 2:00 last Sun Oct 3:00
ntp clock-period 36029118

Cat6509#show ntp assoc detail
192.x.x.x configured, our_master, sane, valid, stratum 2
ref ID 192.x.x.x, time D3A65461.2D183953 (09:12:33.176 CEST Tue Jul 10 2012)
our mode client, peer mode server, our poll intvl 1024, peer poll intvl 1024
root delay 21.01 msec, root disp 5.63, reach 377, sync dist 22.430
delay 1.89 msec, offset -0.0748 msec, dispersion 0.26
precision 2**18, version 3
org time D3A655E8.D80B850A (09:19:04.843 CEST Tue Jul 10 2012)
rcv time D3A655E8.D84EE1FB (09:19:04.844 CEST Tue Jul 10 2012)
xmt time D3A655E8.D7D068C2 (09:19:04.843 CEST Tue Jul 10 2012)
filtdelay =     1.89    2.01    2.30    3.42    1.89    1.95    2.23   18.39
filtoffset =   -0.07   -0.17    0.13    0.65   -0.23   -0.16    0.02    7.10
filterror =     0.02    6.35   14.16   21.97   29.79   37.60   45.41   53.22

Cat6509#show clock detail
09:27:10.336 CEST Tue Jul 10 2012
Time source is NTP
Summer time starts 02:00:00 CET Sun Mar 25 2012
Summer time ends 03:00:00 CEST Sun Oct 28 2012

Cat6509#sh run | i clock
clock timezone CET 1
clock summer-time CEST recurring last Sun Mar 2:00 last Sun Oct 3:00
ntp clock-period 17180071

Thank you!

[Dennis Beul]

Hi guys,

after weeks of figuring out this issue with the Cisco TAC we finally solved the problem.

I just want to share the solution with you - it was an IOS bug, plain and simple:

NTP packets received but ignored by the NTP process

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtf03928

The described workaround fixed the problem. Maybe this will help someone to solve this issue faster

Thank you for your troubleshooting and helpful hints!

[Joe Man]

This worked for me, I had the same problem on a 3750 L3 switch.
I followed the instructions from Cisco:
no ntp
no ntp
(yes twice)
then re-entered the ntp server x.x.x.x and it started working