cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3353
Views
5
Helpful
13
Replies
Highlighted
Beginner

ACI 3.1(2m)

Dear Cisco Support,

 

I just update the apic to the latest 3.1(2m) from v3.0(2k) and starting to see some strange behaviours that i'm going to list:

 

1) "the messaging layer was unabled to deliver the stimulus deserialization"error
3.12m_error.PNG

Did I found the (v1.2) CSCuw34092 bug?

Indeed one controller has a description "Controller 3 is unhealthy because: Data Layer Partially Degraded Leadership"...

Should I swipe the 3rd apic controller as sugested in the article The-messaging-layer-was-unable-to-deliver-the-stimulus-rejected ?

 

 

2) only letters and number are allowed. bellow one example that becomes ok without the "_". Using REST it acepted.

 

bgp_peer.png

 

Any help?

 

Thanks,

Valdemar

 

Everyone's tags (1)
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Cisco Employee

Re: ACI 3.1(2m)

It looks like APIC3 never upgraded.  

 

If these are production APICs, I would suggest to open a case with Cisco TAC to review the logs and find out why the upgrade on APIC3 never started (and or) completed. 

 

If this is a LAB would try the following:

1) it is possible that the upgrade didn’t complete because the APIC didn’t reboot, so you could manually reboot by running the “acidiag reboot”

 

2) If reboot doesn’t  fix the issue, you could try bringing down (keep APIC3 power off) to see if APIC1 and APIC2 can become fully-fit. If they become fully-fit, your can try to decommission APIC3 and manually re-image using the iso image of 3.1(2m). And then re-commission it back into the fabric.

 

View solution in original post

13 REPLIES 13
Highlighted
Cisco Employee

Re: ACI 3.1(2m)

Hi Costavr,

 

From CLI of APIC 1 and 3, can you provide the output of the following command:

 

acidiag avread

ps -ef | grep svc_if

Highlighted
Beginner

Re: ACI 3.1(2m)

In attach the output.

 

From acidiag output, the APIC 3 apears with a diferent version: 3.0(2k) rather than 3.1(2m).

In the GUI the Health state not ok confirmed (no fully fit)

 

Capture.PNG

Highlighted
Cisco Employee

Re: ACI 3.1(2m)

Looks like the upgrade for APIC 3 never happened.

 

Can you provide the output of the following commands from APIC3.

 

uptime

show version

acidiag version

 

bash

firmware upgrade status

 

Also please provide the log file below from APIC 3

/firmware/insieme_installer.log

 

 

 

Highlighted
Beginner

Re: ACI 3.1(2m)

Output in atach.

 

However, "insieme_installer.log: Permission denied"

 

 

Highlighted
Beginner

Re: ACI 3.1(2m)

In Gui, for APIC 3 Storage nothing apears.

Bellow pictures of apic2 and apic3 storage folder

storage2.PNGstorage3.PNG

Highlighted
Cisco Employee

Re: ACI 3.1(2m)

It looks like APIC3 never upgraded.  

 

If these are production APICs, I would suggest to open a case with Cisco TAC to review the logs and find out why the upgrade on APIC3 never started (and or) completed. 

 

If this is a LAB would try the following:

1) it is possible that the upgrade didn’t complete because the APIC didn’t reboot, so you could manually reboot by running the “acidiag reboot”

 

2) If reboot doesn’t  fix the issue, you could try bringing down (keep APIC3 power off) to see if APIC1 and APIC2 can become fully-fit. If they become fully-fit, your can try to decommission APIC3 and manually re-image using the iso image of 3.1(2m). And then re-commission it back into the fabric.

 

View solution in original post

Highlighted
Beginner

Re: ACI 3.1(2m)

1) Reboot doesn't work ... still in the same state.

2) Dificulties on how to shutdown apic3:

   a) Through GUI doens't affect

   b) directly to APIC, it requires root premission

   c) try Decomission but "Cannot self-decommission an appliance"

Any clue?

PS: There is always the last option - switch off physically but not possible remotely...

 

Highlighted
Beginner

Re: ACI 3.1(2m)

After a 2nd reboot (or GUI shutdown(?)) finally a "Fully Fit"state ;)

 

Thanks for helping.

 

Highlighted
Cisco Employee

Re: ACI 3.1(2m)

I am glad to hear that all of your APICs are fully-fit now.  To answer your last questions:

 

1) The only way to shutdown your APIC without root access is via the CIMC GUI.

 

2) To decommission or commission an APIC, you  should be able to do it from another APIC from the controller tab, if tu cluster is not fully-fit it won’t work that is why I was trying to bring APIC 3 down to see if the cluster of APICs 1 and 2 would become fully-fit

Highlighted
Beginner

Re: ACI 3.1(2m)

Indeed "Fully fit" is there but, storage folder @apic 3 still presenting nothing.

Looking to other ACI fabrics deployments, this not seems to be the expected output.

 

Back to some configuration and thinks looks not ok.

For instance,

> delete through GUI an L3Oout has no effect - the L3out remains there.

> In an user tenant, a new L3out cannot resolve the vrf from common. (on others existent L3out, no issues on this)

 

... starting to loose any trust at this point!

 

Any sugestion?

resync database (?), Downgrade to 3.0(2n), shut down/switp apic3...

 

Highlighted
Cisco Employee

Re: ACI 3.1(2m)

You can try:

1)to decommission apic3 from APIC 1 or 2.

2) after decommissioning, you can wipe clean APIC3 using the following commands:

acidiag touch clean

acidiag touch setup

acidiag reboot

3) After reboot go through the initial setup script

4) recommission APIC3 

 

This should force re-sync

 

let me know if this helps.

Highlighted
Beginner

Re: ACI 3.1(2m)

APIC3 Decommisiong done

After wipe it, and fullfile the setup utility the commisioning back was not succesfull.

 

1) The new version 2m was not in the apic so for the startup only version 2k available.

2) not able to login - (invalid login). Based on previous experiences, it seems that is waiting for the leader ...

3) In attach the acidiag hwcheck. - anything strange?

 

Before trying wipe it again, I decide to do some configurations in order to explore how fabric behaves with only 2 apic. So far, no issues found... apart of some "normal" diferences between versions like:

> It doesn't accept routerID as loopback in 2x l3Out (Using 1 l3out with loopback check and onther without looback check - works fine)

> Some issue as the 3.1i where it doesn't allow the change (manually) of names and number of VDS uplinks interfaces. (no issues on 3.0).

>...still listing.

Highlighted
Cisco Employee

Re: ACI 3.1(2m)

What do you mean by:

 

The new version 2m was not in the apic so for the startup only version 2k available.

 

If you are saying that APIC 3 still running a different version than APIC1 and APIC2, then that is the reason APIC3 was not successfully added to the fabric.  You can:

 

1)wipe API3 again :(

2)bring APIC 3 in stand-alone mode

3)upload correr image 2k

4) do policy upgrade on APIC 3

5) after successful upgrade wipe APIC again

6) add APIC3 to your existing fabric running the same version