APIC discovery issue

[pradeepser]

Having challenge is Discovering the nodes. After multiple reboot of ACI & Leaf I managed to register a directly connected leaf. However it took more than an hour to get IP address. Now registered Spine but discovery in progress for almost an hour.

During initial reboot I disabled POAP in the nodes. Is it causing issue? Is there a way to enable the POAP in the nodes.

ACI version 2.1.1h 

Thanks in advance for taking your time to reply!

[Tomas de Leon]

POAP is not used on N9K Switched running in ACI mode.  If you see the switches trying to use POAP then they are still in NXOS mode.

There can be various reasons why a node may not be discovered.  One of the common issues is that the Date & Time are considerably out of sync with the APIC controllers.  

We would be just guessing without looking at the fabric itself.  If you are in a lab (non-production) environment and are just setting up fabric:

• Try resetting all apics back to factory defaults (wiping configuration) and do the same on the switches.  
• Power off APIC2 & APIC3
• Configure APIC1 using the setup script
• Discover all Leafs & Spines
• Once all Leaf & Spines are discovered and joined the fabric, Power on APIC2 and configure.
• Once APIC2 joins the APIC Cluster and is "FULLY FIT", power APIC3 and repeat the process

If this goes smoothly, then you should be fine.  Otherwise, if you continue experience issues please open a Cisco TAC case so an ACI engineer can look at your fabric.

Thank you for participating in the Cisco Support Forum for ACI! If you have other questions related to this post, please let us know. If this response answers your questions, please mark this post "answered" and assign a rating to the response(s) provided. This will help notify other viewers that your question(s) is answered and this helps us provide better responses for this and future questions.

Thank you!

T.

[pradeepser]

Thanks Tomas for your response.

Time is in UTC and there is 7 hour difference between controller and leaf. I am able to see APIC in lldp.

(none)# show lldp neighbors
Capability codes:
  (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
  (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID            Local Intf      Hold-time  Capability  Port ID
APIC1          Eth1/1          120                    eth2-2

(none)# cat /mit/sys/summary

state            : out-of-service

APIC1# acidiag fnvread
      ID   Pod ID                 Name    Serial Number         IP Address    Role        State   LastUpdMsgId
--------------------------------------------------------------------------------------------------------------

Total 0 nodes

Is there any other commands in APIC to identify the exact issue?

[Tomas de Leon · Accepted solution]

The time difference between the APIC and the Leaf\Spine node may be the issue.

As Admin User, the steps are:

• Check the Date, using the "date" CLI command on the Leaf\Spine node. Verify that the date & time is same or within minutes of the APIC.
• If the date & time is out of sync (more then 5 minutes), then set the date & time to match the APIC(s) using the "date" CLI command on the Leaf\Spine node.

Note: you are only setting the "running system clock" not the hardware clock. you will need to sync the "running system clock" to the hardware clock to be persistent after a reload of the Leaf\Spine node.

• Compare the "running system clock" to the hardware clock using the "setup-hwclock.sh" CLI command on the Leaf\Spine node.
• Sync the "running system clock" to the hardware clock using the "setup-hwclock.sh --systohc" CLI command on the Leaf\Spine node.
• Once completed, you can now reload the Leaf\Spine node in order to re-initiate the fabric discovery process.

I hope this helps!

T.

[pradeepser]

We adjusted the APIC controller clock same as leaf/spine and it worked. Thanks Tomas!

Kumar