Unable to login into APIC GUI after upgrade

[u.drechsel]

Hello together,

I upgraded my APIC cluster (3 devices) via GUI from version 3.2.1m to 3.2.2l. After reboot of the nodes I couldn't login into my APIC via GUI. I get the message:

REST Endpoint user authorization datastore is not initialized - Check Fabric Membership Status of this fabric node

CIMC is working, but I only could login into my APIC via CIMC with "rescue-user" and no password. There is a message:

Fabric discovery in progress ... (truncated)

If I try to use a command like "show ?" I get the message:

Error: Authentication cookie is not present.

System is not ready, please wait

I rebooted my APICs via CIMC but there's no success. In several support articles I found, that I have to reset my devices (APICs and switches) to factory default and start again with initial setup of APICs and fabric discovery???

Is there any way to get access to my APICs again without lost of configuration?

Hopefull,

Uwe Drechsel

[Manuel Velasco]

Hi Uwe,

It sounds like you APIC cluster is not fully fit or there is shard issue on these APICs.  Have you tried accessing these APICs via the CLI?  If you can you can get more information about the current stay of these APICs by running the “show controller”, “acidiag fnvread”, or “acidiag avread”. If you are unable to login to the CLI using the admin account you could try logging with username: “rescue-user” and with the admin password or no password.

Also have you try to logging from all 3 APIC IPs? If not try the others. It could be that only one is in the current state. 

Is the CIMC running version 1.5?  If it is try to power cycle the APICs by removing the power cables for 3 minutes, there is a CIMC memory leak that can cause related issues.  The memory issue exist on the CIMC 1.5 version.

You may also consider opening a case with Cisco TAC to help you recover your fabric 

[u.drechsel]

High Manuel,

thanks for your answer. to your questions:

- CIMC is version 3.0(3a),

- all controller show the same behaviour, Login only via "rescue-user" without password,

- "acidiag fnvread" is reporting all nodes are active,

- the output creens of "acidiag avread" are in the attachment. Sorry about the zip-file, unfortunately I couldn't use copy/paste in html based KVM.

I also tried to put three leafs which are connected to the three controllers and one of the connected spine to factory default. After reboot of the controllers via CIMC thre was no success.

I think, I'll open a TAC case.