ACI 3.1(2m)

[costavr]

Dear Cisco Support,

I just update the apic to the latest 3.1(2m) from v3.0(2k) and starting to see some strange behaviours that i'm going to list:

1) "the messaging layer was unabled to deliver the stimulus deserialization"error

Did I found the (v1.2) CSCuw34092 bug?

Indeed one controller has a description "Controller 3 is unhealthy because: Data Layer Partially Degraded Leadership"...

Should I swipe the 3rd apic controller as sugested in the article The-messaging-layer-was-unable-to-deliver-the-stimulus-rejected ?

2) only letters and number are allowed. bellow one example that becomes ok without the "_". Using REST it acepted.

Any help?

Thanks,

Valdemar

[Manuel Velasco]

Hi Costavr,

From CLI of APIC 1 and 3, can you provide the output of the following command:

acidiag avread

ps -ef | grep svc_if

[costavr]

In attach the output.

From acidiag output, the APIC 3 apears with a diferent version: 3.0(2k) rather than 3.1(2m).

In the GUI the Health state not ok confirmed (no fully fit)

[Manuel Velasco]

Looks like the upgrade for APIC 3 never happened.

Can you provide the output of the following commands from APIC3.

uptime

show version

acidiag version

bash

firmware upgrade status

Also please provide the log file below from APIC 3

/firmware/insieme_installer.log

[costavr]

Output in atach.

However, "insieme_installer.log: Permission denied"

[costavr]

In Gui, for APIC 3 Storage nothing apears.

Bellow pictures of apic2 and apic3 storage folder

[Manuel Velasco · Accepted solution]

It looks like APIC3 never upgraded.  

If these are production APICs, I would suggest to open a case with Cisco TAC to review the logs and find out why the upgrade on APIC3 never started (and or) completed. 

If this is a LAB would try the following:

1) it is possible that the upgrade didn’t complete because the APIC didn’t reboot, so you could manually reboot by running the “acidiag reboot”

2) If reboot doesn’t  fix the issue, you could try bringing down (keep APIC3 power off) to see if APIC1 and APIC2 can become fully-fit. If they become fully-fit, your can try to decommission APIC3 and manually re-image using the iso image of 3.1(2m). And then re-commission it back into the fabric.

[costavr]

1) Reboot doesn't work ... still in the same state.

2) Dificulties on how to shutdown apic3:

   a) Through GUI doens't affect

   b) directly to APIC, it requires root premission

   c) try Decomission but "Cannot self-decommission an appliance"

Any clue?

PS: There is always the last option - switch off physically but not possible remotely...

[costavr]

After a 2nd reboot (or GUI shutdown(?)) finally a "Fully Fit"state ;)

Thanks for helping.

[Manuel Velasco]

I am glad to hear that all of your APICs are fully-fit now.  To answer your last questions:

1) The only way to shutdown your APIC without root access is via the CIMC GUI.

2) To decommission or commission an APIC, you  should be able to do it from another APIC from the controller tab, if tu cluster is not fully-fit it won’t work that is why I was trying to bring APIC 3 down to see if the cluster of APICs 1 and 2 would become fully-fit

[costavr]

Indeed "Fully fit" is there but, storage folder @apic 3 still presenting nothing.

Looking to other ACI fabrics deployments, this not seems to be the expected output.

Back to some configuration and thinks looks not ok.

For instance,

> delete through GUI an L3Oout has no effect - the L3out remains there.

> In an user tenant, a new L3out cannot resolve the vrf from common. (on others existent L3out, no issues on this)

... starting to loose any trust at this point!

Any sugestion?

resync database (?), Downgrade to 3.0(2n), shut down/switp apic3...

[Manuel Velasco]

You can try:

1)to decommission apic3 from APIC 1 or 2.

2) after decommissioning, you can wipe clean APIC3 using the following commands:

acidiag touch clean

acidiag touch setup

acidiag reboot

3) After reboot go through the initial setup script

4) recommission APIC3 

This should force re-sync

let me know if this helps.

[costavr]

APIC3 Decommisiong done

After wipe it, and fullfile the setup utility the commisioning back was not succesfull.

1) The new version 2m was not in the apic so for the startup only version 2k available.

2) not able to login - (invalid login). Based on previous experiences, it seems that is waiting for the leader ...

3) In attach the acidiag hwcheck. - anything strange?

Before trying wipe it again, I decide to do some configurations in order to explore how fabric behaves with only 2 apic. So far, no issues found... apart of some "normal" diferences between versions like:

> It doesn't accept routerID as loopback in 2x l3Out (Using 1 l3out with loopback check and onther without looback check - works fine)

> Some issue as the 3.1i where it doesn't allow the change (manually) of names and number of VDS uplinks interfaces. (no issues on 3.0).

>...still listing.

[Manuel Velasco]

What do you mean by:

The new version 2m was not in the apic so for the startup only version 2k available.

If you are saying that APIC 3 still running a different version than APIC1 and APIC2, then that is the reason APIC3 was not successfully added to the fabric.  You can:

1)wipe API3 again :(

2)bring APIC 3 in stand-alone mode

3)upload correr image 2k

4) do policy upgrade on APIC 3

5) after successful upgrade wipe APIC again

6) add APIC3 to your existing fabric running the same version