Solved: ACI - NTP Server not selected for sync

WarrenT-CO · ‎09-07-2022

Hello Cisco community, I have a situation where I've used the APIC quick start wizard to configure the NTP server for the ACI Fabric(Leaf/Spines) and I'm getting the error "NTP Server not selected for sync".

The default out-of-band EPG has been configured for the NTP server along with the default OOB Contract. The default date-time policy has been referenced in the default pod policy group and then referenced in the default pod profile.

I've also provided a snapshot of the error form the APIC GUI.

I'd appreciate any help or troubleshooting tips that anyone could provide. Thank you!

WarrenT-CO · ‎09-12-2022

This issue was resolved by configuring the management IP Address on the Leaf and Spine switches.

Andrea, thank you for your support!

View solution in original post

RedNectar · ‎09-07-2022

Hi,

Thanks for supplying the picture, and in you absolute faith that we'd be able to see immediately where you took that screendump. But I've only been working with ACI for 6 years, and unfortunately that is not enough time to have memorised every possible screen.

So I'm stuck finding the place where you got this screendump (I could find it in 5 mins probably, but I don't have 5 mins)

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

WarrenT-CO · ‎09-08-2022

Haha, I’ll remember to show the navigation path next time.

But in the mean time do you gave any recommendations for solving the problem?

Andrea Testino · ‎09-08-2022

Hi there!

Assuming all the configuration is correct:

Can you ping the NTP server from the Spine/Leafs?
Do you see any faults related to NTP?
What version are the APICs and Leaf/Spines on?

Run the following from CLI:

APIC:

admin@apic1# show clock
admin@apic1# ps -aux | grep ntp
admin@apic1# sho ntpq
admin@apic1# bash
admin@apic1:~> cat /etc/ntp.conf
admin@apic1:~> ntpstat
admin@apic1:~> ntpq -pn
admin@apic1:~> echo $?

Leaf:

admin@leaf1# show clock
admin@leaf1# show ntp peers
admin@leaf1# show ntp peer-status
admin@leaf1# vsh -c "show ntp status"
admin@leaf1# show ntp statistics peer ipaddr 192.168.70.34
admin@leaf1# show ntpq
admin@leaf1# ps -ef | egrep ntp
admin@leaf1# show ip int brief vrf management
admin@leaf1# iping 192.168.70.34 -V management
admin@leaf1# show ntp internal log-buffer
admin@leaf1# show ntp internal event-history config

You could run packet captures on the APIC/Leafs to make sure we are sending and receiving packets to/from the NTP server with:

leaf# tcpdump -i eth0 -f port 123
apic# tcpdump -i oobmgmt -f port 123

Let the captures run a few minutes.

It could be a myriad of things honestly (including the config) but we can start here and see. Attach outputs as a .txt and I'll take a look.

- Andrea, CCIE #56739 R&S

WarrenT-CO · ‎09-08-2022

Hello Thank you very much! for the reply.

1) I can ping the NTP server from the APIC. Also from the Leafs/Spines the ping is successful but I'm not sure what VRF the traffic is using. I tried VRF oob, but that doesn't seem to work. My understanding is that the Leaf/Spine switches are using MGMT0 for the NTP connection.

2)I attached a snap shot of the faults

3) The leafs/spines are on version 15.2(4e) and APIC is on version 5.2(3e)

APIC:

apic# show clock
Time : 08:08:44.610 UTC-06:00 Thu Sep 08 2022

apic# ps -aux | grep ntp
admin 18582 0.0 0.0 9108 888 pts/0 S+ 08:09 0:00 grep ntp
ntp 21179 0.0 0.0 42364 5184 ? Ss 02:59 0:01 /usr/sbin/ntpd -u ntp:ntp -u ntp:ntp -p /var/run/ntpd.pid

apic:~> cat /etc/ntp.conf
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
tinker panic 414624778
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
#restrict default ignore
restrict 127.0.0.1
#restrict -6 ::1

keysdir /etc/ntp/
keys /etc/ntp/keys

server 10.124.146.13 minpoll 4 maxpoll 6
server 10.124.146.12 minpoll 4 maxpoll 6
server 10.124.146.11 prefer minpoll 4 maxpoll 6

apic# sho ntpq
nodeid remote refid st t when poll reach auth delay offset jitter
------ - ------------------------------ ------------------------------ -------- -- -------- -------- -------- ---- -------- -------- --------
1 x 10.124.146.11 .GNSS. 1 u 12 16 377 none 0.079 15035.1 0.924
1 x 10.124.146.12 .GNSS. 1 u 12 16 377 none 0.050 11968.3 0.909
1 x 10.124.146.13 .GNSS. 1 u 5 16 377 none 0.070 240.631 0.922
2 x 10.124.146.11 .GNSS. 1 u 10 16 377 none 0.134 15096.5 1.157
2 x 10.124.146.12 .GNSS. 1 u 10 16 377 none 0.045 12029.7 1.174
2 x 10.124.146.13 .GNSS. 1 u 3 16 377 none 0.131 302.081 1.161
3 x 10.124.146.11 .GNSS. 1 u 14 16 377 none 0.103 15062.3 1.041
3 x 10.124.146.12 .GNSS. 1 u 13 16 377 none 0.086 11995.5 1.035
3 x 10.124.146.13 .GNSS. 1 u 7 16 377 none 0.076 267.852 1.038

apic:~> ntpstat
unsynchronised
polling server every 16 s

apic:~> ntpq -pn
remote refid st t when poll reach delay offset jitter
==============================================================================
x10.124.146.13 .GNSS. 1 u 3 16 377 0.093 243.088 0.924
x10.124.146.12 .GNSS. 1 u 10 16 377 0.049 11970.7 0.911
x10.124.146.11 .GNSS. 1 u 10 16 377 0.078 15037.6 0.925

apic:~> echo $?
0

LEAF:

Leaf# show clock
14:15:14.130019 MDT Thu Sep 08 2022

Leaf# show ntp peers
-----------------------------------------------------------------------------
Peer IP Address Serv/Peer Prefer KeyId Vrf
-----------------------------------------------------------------------------

Leaf# show ntp peer-status
Total peers : 0
* - selected for sync, + - peer mode(active),
- - peer mode(passive), = - polled in client mode
remote local st poll reach delay vrf
--------------------------------------------------------------------------------

Leaf# vsh -c "show ntp status"
Distribution : Disabled
Last operational state: No session

Leaf# show ntp statistics peer ipaddr 10.124.146.11 - - > no output from this command

Leaf# show ntpq
Incorrect command "show ntpq"

Leaf# iping 10.124.146.11 -V management
ping: bad context management

Leaf# show ip int brief vrf management
Bad context entered

Leaf# ps -ef | egrep ntp
root 16714 10428 0 Aug29 ? 00:00:14 /isan/bin/ntp
root 22018 16714 0 Sep07 ? 00:00:03 /isan/bin/ntpd -c /isan/etc/ntpd.conf -g -f /mnt/pss/ntp.drift -n -D 3 -l /var/sysmgr/mem_logs/ntpd_logs
admin 55222 62898 0 14:23 pts/0 00:00:00 grep -E ntp

WarrenT-CO · ‎09-08-2022

After checking the log-buffer, I see the log that references the null VRF. Could the reference to the NULL VRF be the issue? The log-buffer is attached.

Thu 08:58:48 MDT Sep 08 2022 Configured Provider with NULL vrf, returning

WarrenT-CO · ‎09-08-2022

Hello, the tcpdump command couldn't be executed.

apic# tcpdump -i oobmgmt -f port 123
tcpdump: oobmgmt: You don't have permission to capture on that device
(socket: Operation not permitted)

WarrenT-CO · ‎09-12-2022

This issue was resolved by configuring the management IP Address on the Leaf and Spine switches.

Andrea, thank you for your support!