cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1450
Views
5
Helpful
12
Replies
Highlighted
Participant

APIC 2 not responding !

I tried to upgrade APICs to ve 3.1.2o but one of my APIC (#2) is not responding to reload. 

I have tried reloading 3.1.1i from GUI, CLI manullay but no response. 

APIC 1 & 3 responding well. It seems configs are not applying on the #2. However controller status is fully fit. 

 

//Umair

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Cisco Employee

Re: APIC 2 not responding !

Suggest trying to clean reload the APIC:

 

1. Decomission the APIC2 from the UI - give it 10mins to complete.

2. From APIC2 CLI:

   acidiag touch clean

   acidiag reload

3.  Once its boots back up, recommission APIC2 fro the UI.  Allow 10mins to complete.

 

This should clean up the APIC's stale NTP config.  If this doesn't work, you can try to re-setup the APIC using:

1. Decomission the APIC2 from the UI - give it 10mins to complete.

2. From APIC2 CLI:

    acidiag touch clean

    acidiag touch setup

    acidiag touch reload

3.  Once its boots back up, recommission APIC2 fro the UI.  Allow 10mins to complete.

 

After this you'll need to re-assign the appropriate details via KVM console (IP, Hostname, etc).  The rest of the policies should be pushed once it joins the cluster.  Give it some time to go fully fit and it "should" upgrade itself to the Cluster's FW level.

 

Robert

View solution in original post

12 REPLIES 12
Highlighted
Cisco Employee

Re: APIC 2 not responding !

A few things to look at:

 

1. Get the output of 'acidiag avread' from APIC1

2. What does the KVM console for APIC2 show?

 

Robert

Highlighted
Participant

Re: APIC 2 not responding !

apic1# acidiag avread
Local appliance ID=1 ADDRESS=10.0.0.1 TEP ADDRESS=10.0.0.0/16 CHASSIS_ID=31d1ff42-0630-11e7-aa84-058c938afcc4
Cluster of 3 lm(t):1(2018-05-16T12:35:29.607+02:00) appliances (out of targeted 3 lm(t):2(2018-05-17T13:52:11.471+02:00)) with FABRIC_DOMAIN name=ACI-LAB set to version=apic-3.1(2o) lm(t):1(2018-05-17T14:16:22.453+02:00); discoveryMode=PERMISSIVE lm(t):0(1970-01-01T02:00:00.003+02:00)
        appliance id=1  address=10.0.0.1 lm(t):1(2018-05-16T13:03:39.793+02:00) tep address=10.0.0.0/16 lm(t):1(2018-05-16T13:03:39.793+02:00) oob address=172.19.1.20/24 lm(t):1(2018-05-16T12:35:30.048+02:00) version=3.1(2o) lm(t):1(2018-05-17T13:52:13.088+02:00) chassisId=31d1ff42-0630-11e7-aa84-058c938afcc4 lm(t):1(2018-05-17T13:52:13.088+02:00) capabilities=0X7FFFFFFFF--0X2020--0X7 lm(t):1(2018-05-17T14:20:30.246+02:00) rK=(stable,present,0X206173722D687373) lm(t):1(2018-05-16T12:35:30.053+02:00) aK=(stable,present,0X206173722D687373) lm(t):1(2018-05-16T12:35:30.053+02:00) cntrlSbst=(APPROVED, xx) lm(t):1(2018-05-17T13:52:13.088+02:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):1(2018-05-16T13:03:39.793+02:00) commissioned=YES lm(t):1(zeroTime) registered=YES lm(t):1(2018-05-16T13:03:39.793+02:00) standby=NO lm(t):1(2018-05-16T13:03:39.793+02:00) active=YES(2018-05-16T13:03:39.793+02:00) health=(applnc:255 lm(t):1(2018-05-17T13:52:15.087+02:00) svc's)
        appliance id=2  address=10.0.0.2 lm(t):3(2018-05-16T14:48:31.356+02:00) tep address=10.0.0.0/16 lm(t):2(2017-03-16T00:55:31.097+02:00) oob address=172.19.1.21/24 lm(t):1(2018-05-17T13:52:13.528+02:00) version=3.1(1i) lm(t):2(2018-05-17T13:52:13.189+02:00) chassisId=71fb2242-09d2-11e7-a7b7-abc87eae134c lm(t):2(2018-05-17T13:52:13.189+02:00) capabilities=0X7FFFFFFFF--0X2020--0 lm(t):2(2018-03-19T07:08:24.116+02:00) rK=(stable,present,0X206173722D687373) lm(t):1(2018-05-17T13:52:13.528+02:00) aK=(stable,present,0X206173722D687373) lm(t):1(2018-05-17T13:52:13.528+02:00) cntrlSbst=(APPROVED, xx) lm(t):0(zeroTime) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):2(2018-03-19T07:36:36.353+02:00) commissioned=YES lm(t):1(2018-05-16T12:35:29.606+02:00) registered=YES lm(t):3(2017-03-20T15:10:05.325+02:00) standby=NO lm(t):3(2018-05-16T12:35:29.606+02:00) active=YES(2018-05-17T13:52:12.951+02:00) health=(applnc:255 lm(t):2(2018-05-17T13:52:59.507+02:00) svc's)
        appliance id=3  address=10.0.0.3 lm(t):3(2018-05-16T14:48:24.642+02:00) tep address=10.0.0.0/16 lm(t):3(2017-03-20T15:09:59.852+02:00) oob address=172.19.1.22/24 lm(t):1(2018-05-17T13:52:13.484+02:00) version=3.1(2o) lm(t):3(2018-05-17T13:52:13.164+02:00) chassisId=6aa68208-0d6e-11e7-8534-abe884595d92 lm(t):3(2018-05-17T13:52:13.164+02:00) capabilities=0X7FFFFFFFF--0X2020--0 lm(t):3(2018-05-17T14:20:44.059+02:00) rK=(stable,present,0X206173722D687373) lm(t):1(2018-05-17T13:52:13.484+02:00) aK=(stable,present,0X206173722D687373) lm(t):1(2018-05-17T13:52:13.484+02:00) cntrlSbst=(APPROVED, xx) lm(t):0(zeroTime) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):3(2018-05-16T14:48:24.642+02:00) commissioned=YES lm(t):1(2018-05-16T12:35:29.606+02:00) registered=YES lm(t):1(2017-03-20T15:10:11.335+02:00) standby=NO lm(t):3(2018-05-16T14:48:24.642+02:00) active=YES(2018-05-16T14:20:16.190+02:00) health=(applnc:255 lm(t):3(2018-05-17T13:52:14.912+02:00) svc's)
---------------------------------------------
clusterTime=<diff=-1696893 common=2018-05-22T16:25:15.383+02:00 local=2018-05-22T16:53:32.276+02:00 pF=<displForm=0 offsSt=0 offsVlu=7200 lm(t):2(2018-05-17T13:52:11.446+02:00)>>
---------------------------------------------

Fault on the APIC 2 . But I did remove the NTP setting from GUI From 172.19.0.10 but it seems it does not propagate to APIC 2.

Properties
F1574
critical
2018-03-19T07:39:04.495+02:00
Raised
topology/pod-1/node-2/sys/ntpq-172.19.0.10
Ntp configuration on Node 2 for 172.19.0.10 is not reachable

 

Highlighted
Participant

Re: APIC 2 not responding !

This is from APIC 2

apic2# acidiag avread Local appliance ID=2 ADDRESS=10.0.0.2 TEP ADDRESS=10.0.0.0/16 CHASSIS_ID=71fb2242-09d2-11e7-a7b7-abc87eae134c Cluster of 3 lm(t):2(2018-03-19T07:08:23.679+02:00) appliances (out of targeted 3 lm(t):2(2018-05-17T13:52:11.471+02:00)) with FABRIC_DOMAIN name=ACI-LAB set to version=apic-3.1(2o) lm(t):1(2018-05-17T14:16:22.453+02:00); discoveryMode=PERMISSIVE lm(t):0(1970-01-01T02:00:00.003+02:00) appliance id=1 address=10.0.0.1 lm(t):3(2018-05-16T14:48:31.356+02:00) tep address=10.0.0.0/16 lm(t):1(2017-03-16T00:55:37.049+02:00) oob address=172.19.1.20/24 lm(t):2(2018-05-17T13:52:13.521+02:00) version=3.1(2o) lm(t):1(2018-05-17T13:52:13.088+02:00) chassisId=31d1ff42-0630-11e7-aa84-058c938afcc4 lm(t):1(2018-05-17T13:52:13.088+02:00) capabilities=0X7FFFFFFFF--0X2020--0 lm(t):1(2018-05-17T14:20:30.246+02:00) rK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:13.521+02:00) aK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:13.521+02:00) cntrlSbst=(APPROVED, xx) lm(t):0(zeroTime) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):1(2018-05-16T13:03:39.793+02:00) commissioned=YES lm(t):2(2017-03-20T15:10:05.325+02:00) registered=YES lm(t):3(2017-03-20T15:10:05.325+02:00) standby=NO lm(t):1(2018-05-16T13:03:39.793+02:00) active=YES(2018-05-17T13:52:13.092+02:00) health=(applnc:255 lm(t):1(2018-05-17T13:52:15.087+02:00) svc's) appliance id=2 address=10.0.0.2 lm(t):2(2018-03-19T07:36:36.353+02:00) tep address=10.0.0.0/16 lm(t):2(2018-03-19T07:36:36.353+02:00) oob address=172.19.1.21/24 lm(t):2(2018-05-17T13:52:11.258+02:00) version=3.1(1i) lm(t):2(2018-05-17T13:52:13.189+02:00) chassisId=71fb2242-09d2-11e7-a7b7-abc87eae134c lm(t):2(2018-05-17T13:52:13.189+02:00) capabilities=0X7FFFFFFFF--0X2020--0X7 lm(t):2(2018-03-19T07:08:24.116+02:00) rK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:11.268+02:00) aK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:11.268+02:00) cntrlSbst=(APPROVED, xx) lm(t):2(2018-05-17T13:52:13.189+02:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):2(2018-03-19T07:36:36.353+02:00) commissioned=YES lm(t):2(zeroTime) registered=YES lm(t):2(2018-03-19T07:36:36.353+02:00) standby=NO lm(t):2(2018-03-19T07:36:36.353+02:00) active=YES(2018-03-19T07:36:36.353+02:00) health=(applnc:255 lm(t):2(2018-05-17T13:52:59.507+02:00) svc's) appliance id=3 address=10.0.0.3 lm(t):3(2018-05-16T14:48:24.642+02:00) tep address=10.0.0.0/16 lm(t):3(2017-03-20T15:09:59.852+02:00) oob address=172.19.1.22/24 lm(t):2(2018-05-17T13:52:13.521+02:00) version=3.1(2o) lm(t):3(2018-05-17T13:52:13.164+02:00) chassisId=6aa68208-0d6e-11e7-8534-abe884595d92 lm(t):3(2018-05-17T13:52:13.164+02:00) capabilities=0X7FFFFFFFF--0X2020--0 lm(t):3(2018-05-17T14:20:44.059+02:00) rK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:13.521+02:00) aK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:13.521+02:00) cntrlSbst=(APPROVED, xx) lm(t):0(zeroTime) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):3(2018-05-16T14:48:24.642+02:00) commissioned=YES lm(t):1(2018-05-16T12:35:29.606+02:00) registered=YES lm(t):1(2017-03-20T15:10:11.335+02:00) standby=NO lm(t):3(2018-05-16T14:48:24.642+02:00) active=YES(2018-05-17T13:52:13.163+02:00) health=(applnc:255 lm(t):3(2018-05-17T13:52:14.912+02:00) svc's) --------------------------------------------- clusterTime=<diff=-1696892 common=2018-05-22T16:34:20.092+02:00 local=2018-05-22T17:02:36.984+02:00 pF=<displForm=0 offsSt=0 offsVlu=7200 lm(t):2(2018-05-17T13:52:11.446+02:00)>>
Highlighted
Cisco Employee

Re: APIC 2 not responding !

Suggest trying to clean reload the APIC:

 

1. Decomission the APIC2 from the UI - give it 10mins to complete.

2. From APIC2 CLI:

   acidiag touch clean

   acidiag reload

3.  Once its boots back up, recommission APIC2 fro the UI.  Allow 10mins to complete.

 

This should clean up the APIC's stale NTP config.  If this doesn't work, you can try to re-setup the APIC using:

1. Decomission the APIC2 from the UI - give it 10mins to complete.

2. From APIC2 CLI:

    acidiag touch clean

    acidiag touch setup

    acidiag touch reload

3.  Once its boots back up, recommission APIC2 fro the UI.  Allow 10mins to complete.

 

After this you'll need to re-assign the appropriate details via KVM console (IP, Hostname, etc).  The rest of the policies should be pushed once it joins the cluster.  Give it some time to go fully fit and it "should" upgrade itself to the Cluster's FW level.

 

Robert

View solution in original post

Highlighted
Participant

Re: APIC 2 not responding !

just a side question... is there any way to get the CIMC address from GUI or configuration of APIC ?
Highlighted
Cisco Employee

Re: APIC 2 not responding !

No. IMC is completely out of band for the platform. Has no interaction with the Host APIC OS. Either you know the IMC IP, or you can reboot it with a monitor/keyboard locally connected and watch the console. The IMC details will be shown on screen during the POST process.

Robert
Highlighted
Participant

Re: APIC 2 not responding !

 

ok it was  :)

 acidiag reboot

 

Highlighted
Cisco Employee

Re: APIC 2 not responding !

Yes.  Sorry typo.  'acidiag reboot'.

Highlighted
Participant

Re: APIC 2 not responding !

I did as you stated... Now come another hassle.
as APCI 1 & 3 are ver 3.1.2o and APIC 2 is 3.1.1i.
APIC 2 heath is 'Data Layer Partially diverge'

F1824
critical
2018-05-22T17:54:35.607+02:00
Raised
topology/pod-1/node-1/lon/incmptblClsPeer-version_incompatible_with_cluster-2-3.1(1i)-apic-3.1(2o)
Peer controller is incompatible: Version incompatible with cluster. The controller ID is 2, its version is 3.1(1i). Cluster Version is apic-3.1(2o)
Highlighted
Cisco Employee

Re: APIC 2 not responding !

Let's run the upgrade manually against APIC2.  If you have the APIC ISO for 3.1(2o) you can mount it to the IMC vMedia, and boot the APIC.  This will auto-upgrade APIC2.  After it completes, you should be able to reboot it and into the cluster successfully.

 

Robert

Highlighted
Participant

Re: APIC 2 not responding !

Hi Robert,
I just finish downgrading the other two APICs 1 & 3 to 3.1.1i.
Now it seems they are all fully fit.

 

However your second option is worth a try. As downgrading other two APICs is a bit risky ?


Thanks for your kind help !
// Umair

Highlighted
Cisco Employee

Re: APIC 2 not responding !

Are you going to re-attempt the upgrade?  Ensure the NTP policy across all APICs is consistent before doing so.

 

Robert