Air-gaped Nexus 9k as time source

[JaVa808]

need some help understanding the configurations i need to make in an air gaped environment where we cannot have an external time source and must use our core switches. Currently core switch 1 and core switch 2 have been designated as ntp master 1. The access switches then reference core switch 1 and 2, which are set to ntp master 2. 

I have a few additional core switches and access switches remote site that point to the core switch 1 and core switch 2. 

but two things have come up:

1. I see falsetickers when doing show ntp association

2. the time on the local switches where the access switch resides do not sync up with time there. It shows 15:28:00 instead. 

Not savvy with Nexus commands so this is currently what I have configured:

Currently my configuration on the nexus' are:

ntp server x.x.x.4 use-vrf default

ntp server x.x.x.3 use-vrf default 

ntp source-interface VLAN10

ntp master 1

access switches:

ntp logging

no ntp allow mode control 0

ntp master 2

ntp server x.x.x.4

ntp server x.x.x.3

Do I need to change up my commands on both?

[Christopher Hart]

Hello!

I'd like to ask a few clarifying questions on this issue.

1. You mentioned that this network is air-gapped, but the Nexus core switches are both configured with upstream NTP servers via the default VRF. What devices own those IP addresses?
2. What model of Nexus devices are your core and access switches?
3. What NX-OS software release is running on these Nexus devices?
4. Would you be able to share the output of show ntp peer-status from both Nexus core switches?

Thank you!

-Christopher

[JaVa808]

1. Sorry to have forgotten to include this info. the core switch 1 is x.x.x.4 and core switch 2 is x.x.x.5

2. C9372PX core switches and C2960X access switches.

3. nxos.7.0(3)I7(6)

4. Unable to at this moment, will circle back next week with those details. 

[Christopher Hart]

Hello!

Just to clarify, Core-1 is x.x.x.3 and Core-2 is x.x.x.4 - correct? The configuration you provided uses x.x.x.3 and x.x.x.4, not x.x.x.4 and x.x.x.5. Just want to make sure we're both on the same page!

I'm curious to see the output of show ntp peer-status from the Nexus core switches. It may be worth configuring them both to point to each other with ntp peer configuration as opposed to defining each other as servers. ntp peer configuration generally works better when you need to synchronize time between two clocks at the same stratum level.

Once we manage to get time correctly synchronized between both core switches, we can verify whether NTP associations on the access switches stabilize.

Thank you!

-Christopher

[JaVa808]

x.x.x.4 and x.x.x.3 is correct and no x.x.x.5. Didn't finish my first cup of coffee yet. 

I'll make sure to do the show ntp peer-status and we can go from there. 

Appreciate your reply. 

[JaVa808]

show ntp peer-status

Core1:

remote              local     st     poll    reach      delay             vrf
*127.127.1.0     x.x.x.4  1      16     377         0.00000

Core2:

remote              local     st     poll    reach      delay             vrf
*127.127.1.0     x.x.x.5  1      16     377         0.00000

[Christopher Hart]

Hello!

This output looks good to me. This indicates that each core switch is synchronizing with its local clock (which has an internal IP address of 127.127.1.0) and acting as an NTP master. 

As I said before, you may want to consider swapping out the ntp server configuration on both core switches with ntp peer configuration instead. For example, let's say Core 1 owns x.x.x.4, and Core 2 owns x.x.x.5. I recommend changing your configuration to be as follows:

Core-1# show running-config ntp
<snip>
ntp peer x.x.x.5
ntp master 1

Core-2# show running-config ntp
<snip>
ntp peer x.x.x.4
ntp master 1

Once that's done, allow a bit of time for both core switches to synchronize time between themselves (you can usually expedite this process by manually setting the clock of both switches to be the same time at the exact same time through a terminal emulator [SecureCRT, PuTTY, etc.] that allows synchronous input to multiple devices). Then, see if your downstream Catalyst devices (which should be NTP stratum 2 clients of both core switches) successfully synchronize with one of the two core switches.

Note that your downstream clients will only synchronize with one of the two core switches. That is expected behavior. However, I would not expect any of the clients to declare one of the core switches to be a falseticker - that indicates that some other significant source of time drift is being introduced between the two core switches.

If you see any odd NTP behavior after 1-2 days on your downstream Catalyst devices, please do share the output of show ntp associations from the relevant Catalyst devices.

I hope this helps - thank you!

-Christopher

[JaVa808]

Thank you for verifying. 

But on the nexus there is a feature NTP. should that be turned on also? so that pretty much wraps up the stratum 1 devices. 

And in trying to setup the downstream..i'm a bit thrown off by the access switches output when I do a show ntp association it shows both ntp servers as x~x.x.x.4 and x~x.x.x.5. X being falseticker. 

Dont know what that entails ? 

address    ref clock

x~x.x.x.4  127.127.1.0 

x~x.x.x.5  127.127.1.0 

is that right? or something else to dive into?

[Christopher Hart]

Hello! Apologies for the delay in response.

NX-OS does have a feature ntp command - however, it is enabled by default. There is no need to explicitly enable the NTP feature on Nexus switches - see output below from my lab demonstrating this.

N9K# show running-config ntp 

!Command: show running-config ntp
!Running configuration last done at: Fri Sep  4 18:02:25 2020
!Time: Mon Sep  7 13:29:43 2020

version 9.3(5) Bios:version 05.42 
ntp server 192.0.2.20 use-vrf management
ntp server 192.0.2.21 use-vrf management


N9K# show running-config all | include feature.ntp
feature ntp
N9K#

Since it has been a few weeks since your last post, can you confirm that the access switches still report both core switches as falsetickers? If so, can you confirm that the two core switches have a clock that is close to identical with each other by executing the show clock command on both switches at the same time and comparing timestamps?

Thank you!

-Christopher

[JaVa808]

Apologies Christopher, seems as I have not checked this since the previous post and left you hanging. 

let me doublecheck and see how it looks on the switches to put a close to this.