Can't get a ACI APIC controller to recommission - Help Please

[Del Murphy]

OK, I've beaten my head against the wall on this one for a while now, so I'm asking for help.

I have an ACI environment about a year or so ago I needed to expand it from a single POD to a Multipod deployment. I had the bright idea of decommissioning one of the three APIC controllers and adding it to the second POD.

I have it connected to a set of leafs in POD 2, but can't get the thing to recommission into the fabric. I have reset the controller like six times with the following commands:

acidiag touch clean

acidiag touch setup

acidiag reboot

I have the APIC configured with the same TEP address space at POD 1, the same Fiber Name as POD 1 the same Infr VLAN as POD 1, only the POD ID is different with it set to POD ID 2.

When I log in to APIC controller 1 and select SYSTEM|Controllers|Controllers|apic1|Cluster as seen by Node

it shows apic3 as out of service unregistered with a health state of never know. I have right-clicked and selected commission with no change.

On Leaf 201 and 202 in POD 2 that the APIC controller is attached to under Fibric | inventory the interface shows as orange with a switching state of disabled. I have enabled and reset the port with no change.

The ports oper State and Admin State are up and green. The link light is green. I'm at a loss for what to do to fix this.

[naveeku6]

Hi Del,

As per the statement, I could see Initial Fabric settings seems to be fine but not sure about the network connectivity between the POD's. Since APIC's needs network connectvity between the POD's, Kindly verify the IPN network connectivity. Also Ensure that the firmware and software versions on the APIC controller are compatible with those of the existing fabric. If they are not, you may need to upgrade the APIC controller's firmware and software to match those of the existing fabric.

-----------------------------------------
If you find my reply solved your question or issue, kindly click the 'Accept as Solution' button and vote it as helpful.

You can also learn more about Cisco ACI through our live Ask the Experts (ATXs) session. Check out the ATXs Resources [https://community.cisco.com/t5/data-center-and-cloud-knowledge/cisco-aci-ask-the-experts-resources/ta-p/4394491] to view the latest schedule for upcoming sessions, as well as the useful references, e.g. online guides, FAQs.
-----------------------------------------

[Del Murphy]

I can see all other devices in POD2, spine, and leaf switches. In the menu (System|Controllers|Controllers|apic1|Cluster As seen by node), I can see the APIC controller in POD2 as an unauthorized controller, but it will not accept the controller. So I would think I'm not having a communication problem with POD2.

[weylin.piegorsch]

Just to confirm - you don't have LLDP enabled in the IMC, correct?  I had a similar issue recently, same symptoms except single pod (switching state disabled; controller never known; etc).  That turned out to be the culprit in my case: if the IMC has LLDP enabled, it hijacks the LLDP subprocess, and prevents the APIC from properly negotiating with the leaf.

[Del Murphy]

Interesting thought, I do have LLDP active. I'll disable it and see if that helps.

[weylin.piegorsch]

just wanted to check in - how did it go?

[Del Murphy]

I had misremembered. We had Radius enabled, not LLDP, but I disabled it anyway to ensure it was an issue. Doing so has yet to resolve the issue. Good thought.

[weylin.piegorsch]

bummer. I wish you luck, I'd love to hear here when you get resolution.

[weylin.piegorsch]

@Del Murphy If I may follow-up for my own curiosity, were you able to identify the issue?

[Del Murphy]

Not yet, I have had time to spend on the issue. But in the next couple of weeks, I think it's time to call TAC and work out the issue. I'll post what the solution ends up being.

[rschulting]

Hi Del, did you solve the issue w/ TAC?

I have exactly the same issue. Initially brought up APIC3 with older sw version and obviously registration failed. Did a full upgrade to correct version and wiped its config twice now attempting to registrer but leaf port remains on switching disabled no matter what I do. And sure LLDP is off.

Hope you have the answer?

[Del Murphy]

I have not resolved this issue as of yet. Sorry.

[rschulting · Accepted solution]

PROBLEM SOLVED!

First you have to decomission APIC3! via System -> Controllers -> Controllers -> pick any registered APIC -> cluster as seen by node, right click on APIC3 and select decommission.
After whiping APIC3 and going through the setup again, you have to re-commission APIC3 via System -> Controllers -> Controllers -> pick any registered APIC -> cluster as seen by node, right click on APIC3 and select commission. After a while APIC3 did came up and slowly it became fully registered. During this I monitored eth1/1 on the spine201 and I did see switching state becoming green (enabled) as well.

[RedNectar]

hi @rschulting ,

If your question has been answered, it is a great idea to mark the question as being answered. EVEN IF YOU SUPPLIED THE ANSWER YOURSELF  This helps:

1. others with a similar problem find the correct answer
2. people who look for "unanswered" questions to answer finding this
3. prevent your question from becoming a "dead thread"

BTW - you may wish to edit/fix the typo - "whipping" APIC3 might be what you feel like doing, but "wiping" it is probably what you meant.

[rschulting]

Thx Chris and yes I sid have a typo...