Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
3232
Views
5
Helpful
12
Replies

APIC 2 not responding !

Go to solution
kthned
Level 3
Level 3

‎05-22-2018 07:44 AM - edited ‎03-01-2019 05:33 AM

I tried to upgrade APICs to ve 3.1.2o but one of my APIC (#2) is not responding to reload. 

I have tried reloading 3.1.1i from GUI, CLI manullay but no response. 

APIC 1 & 3 responding well. It seems configs are not applying on the #2. However controller status is fully fit. 

 

//Umair

 

 

Labels:
0 Helpful
1 Accepted Solution

Accepted Solutions

Go to solution
Robert Burns
Cisco Employee
Cisco Employee
In response to kthned

‎05-22-2018 08:14 AM

Suggest trying to clean reload the APIC:

 

1. Decomission the APIC2 from the UI - give it 10mins to complete.

2. From APIC2 CLI:

   acidiag touch clean

   acidiag reload

3.  Once its boots back up, recommission APIC2 fro the UI.  Allow 10mins to complete.

 

This should clean up the APIC's stale NTP config.  If this doesn't work, you can try to re-setup the APIC using:

1. Decomission the APIC2 from the UI - give it 10mins to complete.

2. From APIC2 CLI:

    acidiag touch clean

    acidiag touch setup

    acidiag touch reload

3.  Once its boots back up, recommission APIC2 fro the UI.  Allow 10mins to complete.

 

After this you'll need to re-assign the appropriate details via KVM console (IP, Hostname, etc).  The rest of the policies should be pushed once it joins the cluster.  Give it some time to go fully fit and it "should" upgrade itself to the Cluster's FW level.

 

Robert

View solution in original post

12 Replies 12

Go to solution
Robert Burns
Cisco Employee
Cisco Employee

‎05-22-2018 07:51 AM

A few things to look at:

 

1. Get the output of 'acidiag avread' from APIC1

2. What does the KVM console for APIC2 show?

 

Robert

0 Helpful

Go to solution
kthned
Level 3
Level 3
In response to Robert Burns

‎05-22-2018 07:58 AM 

apic1# acidiag avread
Local appliance ID=1 ADDRESS=10.0.0.1 TEP ADDRESS=10.0.0.0/16 CHASSIS_ID=31d1ff42-0630-11e7-aa84-058c938afcc4
Cluster of 3 lm(t):1(2018-05-16T12:35:29.607+02:00) appliances (out of targeted 3 lm(t):2(2018-05-17T13:52:11.471+02:00)) with FABRIC_DOMAIN name=ACI-LAB set to version=apic-3.1(2o) lm(t):1(2018-05-17T14:16:22.453+02:00); discoveryMode=PERMISSIVE lm(t):0(1970-01-01T02:00:00.003+02:00)
        appliance id=1  address=10.0.0.1 lm(t):1(2018-05-16T13:03:39.793+02:00) tep address=10.0.0.0/16 lm(t):1(2018-05-16T13:03:39.793+02:00) oob address=172.19.1.20/24 lm(t):1(2018-05-16T12:35:30.048+02:00) version=3.1(2o) lm(t):1(2018-05-17T13:52:13.088+02:00) chassisId=31d1ff42-0630-11e7-aa84-058c938afcc4 lm(t):1(2018-05-17T13:52:13.088+02:00) capabilities=0X7FFFFFFFF--0X2020--0X7 lm(t):1(2018-05-17T14:20:30.246+02:00) rK=(stable,present,0X206173722D687373) lm(t):1(2018-05-16T12:35:30.053+02:00) aK=(stable,present,0X206173722D687373) lm(t):1(2018-05-16T12:35:30.053+02:00) cntrlSbst=(APPROVED, xx) lm(t):1(2018-05-17T13:52:13.088+02:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):1(2018-05-16T13:03:39.793+02:00) commissioned=YES lm(t):1(zeroTime) registered=YES lm(t):1(2018-05-16T13:03:39.793+02:00) standby=NO lm(t):1(2018-05-16T13:03:39.793+02:00) active=YES(2018-05-16T13:03:39.793+02:00) health=(applnc:255 lm(t):1(2018-05-17T13:52:15.087+02:00) svc's)
        appliance id=2  address=10.0.0.2 lm(t):3(2018-05-16T14:48:31.356+02:00) tep address=10.0.0.0/16 lm(t):2(2017-03-16T00:55:31.097+02:00) oob address=172.19.1.21/24 lm(t):1(2018-05-17T13:52:13.528+02:00) version=3.1(1i) lm(t):2(2018-05-17T13:52:13.189+02:00) chassisId=71fb2242-09d2-11e7-a7b7-abc87eae134c lm(t):2(2018-05-17T13:52:13.189+02:00) capabilities=0X7FFFFFFFF--0X2020--0 lm(t):2(2018-03-19T07:08:24.116+02:00) rK=(stable,present,0X206173722D687373) lm(t):1(2018-05-17T13:52:13.528+02:00) aK=(stable,present,0X206173722D687373) lm(t):1(2018-05-17T13:52:13.528+02:00) cntrlSbst=(APPROVED, xx) lm(t):0(zeroTime) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):2(2018-03-19T07:36:36.353+02:00) commissioned=YES lm(t):1(2018-05-16T12:35:29.606+02:00) registered=YES lm(t):3(2017-03-20T15:10:05.325+02:00) standby=NO lm(t):3(2018-05-16T12:35:29.606+02:00) active=YES(2018-05-17T13:52:12.951+02:00) health=(applnc:255 lm(t):2(2018-05-17T13:52:59.507+02:00) svc's)
        appliance id=3  address=10.0.0.3 lm(t):3(2018-05-16T14:48:24.642+02:00) tep address=10.0.0.0/16 lm(t):3(2017-03-20T15:09:59.852+02:00) oob address=172.19.1.22/24 lm(t):1(2018-05-17T13:52:13.484+02:00) version=3.1(2o) lm(t):3(2018-05-17T13:52:13.164+02:00) chassisId=6aa68208-0d6e-11e7-8534-abe884595d92 lm(t):3(2018-05-17T13:52:13.164+02:00) capabilities=0X7FFFFFFFF--0X2020--0 lm(t):3(2018-05-17T14:20:44.059+02:00) rK=(stable,present,0X206173722D687373) lm(t):1(2018-05-17T13:52:13.484+02:00) aK=(stable,present,0X206173722D687373) lm(t):1(2018-05-17T13:52:13.484+02:00) cntrlSbst=(APPROVED, xx) lm(t):0(zeroTime) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):3(2018-05-16T14:48:24.642+02:00) commissioned=YES lm(t):1(2018-05-16T12:35:29.606+02:00) registered=YES lm(t):1(2017-03-20T15:10:11.335+02:00) standby=NO lm(t):3(2018-05-16T14:48:24.642+02:00) active=YES(2018-05-16T14:20:16.190+02:00) health=(applnc:255 lm(t):3(2018-05-17T13:52:14.912+02:00) svc's)
---------------------------------------------
clusterTime=<diff=-1696893 common=2018-05-22T16:25:15.383+02:00 local=2018-05-22T16:53:32.276+02:00 pF=<displForm=0 offsSt=0 offsVlu=7200 lm(t):2(2018-05-17T13:52:11.446+02:00)>>
---------------------------------------------

Fault on the APIC 2 . But I did remove the NTP setting from GUI From 172.19.0.10 but it seems it does not propagate to APIC 2.

Properties
F1574
critical
2018-03-19T07:39:04.495+02:00
Raised
topology/pod-1/node-2/sys/ntpq-172.19.0.10
Ntp configuration on Node 2 for 172.19.0.10 is not reachable

 

0 Helpful

Go to solution
kthned
Level 3
Level 3
In response to kthned

‎05-22-2018 08:04 AM 

This is from APIC 2

apic2# acidiag avread 
Local appliance ID=2 ADDRESS=10.0.0.2 TEP ADDRESS=10.0.0.0/16 CHASSIS_ID=71fb2242-09d2-11e7-a7b7-abc87eae134c
Cluster of 3 lm(t):2(2018-03-19T07:08:23.679+02:00) appliances (out of targeted 3 lm(t):2(2018-05-17T13:52:11.471+02:00)) with FABRIC_DOMAIN name=ACI-LAB set to version=apic-3.1(2o) lm(t):1(2018-05-17T14:16:22.453+02:00); discoveryMode=PERMISSIVE lm(t):0(1970-01-01T02:00:00.003+02:00)
        appliance id=1  address=10.0.0.1 lm(t):3(2018-05-16T14:48:31.356+02:00) tep address=10.0.0.0/16 lm(t):1(2017-03-16T00:55:37.049+02:00) oob address=172.19.1.20/24 lm(t):2(2018-05-17T13:52:13.521+02:00) version=3.1(2o) lm(t):1(2018-05-17T13:52:13.088+02:00) chassisId=31d1ff42-0630-11e7-aa84-058c938afcc4 lm(t):1(2018-05-17T13:52:13.088+02:00) capabilities=0X7FFFFFFFF--0X2020--0 lm(t):1(2018-05-17T14:20:30.246+02:00) rK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:13.521+02:00) aK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:13.521+02:00) cntrlSbst=(APPROVED, xx) lm(t):0(zeroTime) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):1(2018-05-16T13:03:39.793+02:00) commissioned=YES lm(t):2(2017-03-20T15:10:05.325+02:00) registered=YES lm(t):3(2017-03-20T15:10:05.325+02:00) standby=NO lm(t):1(2018-05-16T13:03:39.793+02:00) active=YES(2018-05-17T13:52:13.092+02:00) health=(applnc:255 lm(t):1(2018-05-17T13:52:15.087+02:00) svc's)
        appliance id=2  address=10.0.0.2 lm(t):2(2018-03-19T07:36:36.353+02:00) tep address=10.0.0.0/16 lm(t):2(2018-03-19T07:36:36.353+02:00) oob address=172.19.1.21/24 lm(t):2(2018-05-17T13:52:11.258+02:00) version=3.1(1i) lm(t):2(2018-05-17T13:52:13.189+02:00) chassisId=71fb2242-09d2-11e7-a7b7-abc87eae134c lm(t):2(2018-05-17T13:52:13.189+02:00) capabilities=0X7FFFFFFFF--0X2020--0X7 lm(t):2(2018-03-19T07:08:24.116+02:00) rK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:11.268+02:00) aK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:11.268+02:00) cntrlSbst=(APPROVED, xx) lm(t):2(2018-05-17T13:52:13.189+02:00) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):2(2018-03-19T07:36:36.353+02:00) commissioned=YES lm(t):2(zeroTime) registered=YES lm(t):2(2018-03-19T07:36:36.353+02:00) standby=NO lm(t):2(2018-03-19T07:36:36.353+02:00) active=YES(2018-03-19T07:36:36.353+02:00) health=(applnc:255 lm(t):2(2018-05-17T13:52:59.507+02:00) svc's)
        appliance id=3  address=10.0.0.3 lm(t):3(2018-05-16T14:48:24.642+02:00) tep address=10.0.0.0/16 lm(t):3(2017-03-20T15:09:59.852+02:00) oob address=172.19.1.22/24 lm(t):2(2018-05-17T13:52:13.521+02:00) version=3.1(2o) lm(t):3(2018-05-17T13:52:13.164+02:00) chassisId=6aa68208-0d6e-11e7-8534-abe884595d92 lm(t):3(2018-05-17T13:52:13.164+02:00) capabilities=0X7FFFFFFFF--0X2020--0 lm(t):3(2018-05-17T14:20:44.059+02:00) rK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:13.521+02:00) aK=(stable,present,0X206173722D687373) lm(t):2(2018-05-17T13:52:13.521+02:00) cntrlSbst=(APPROVED, xx) lm(t):0(zeroTime) (targetMbSn= lm(t):0(zeroTime), failoverStatus=0 lm(t):0(zeroTime)) podId=1 lm(t):3(2018-05-16T14:48:24.642+02:00) commissioned=YES lm(t):1(2018-05-16T12:35:29.606+02:00) registered=YES lm(t):1(2017-03-20T15:10:11.335+02:00) standby=NO lm(t):3(2018-05-16T14:48:24.642+02:00) active=YES(2018-05-17T13:52:13.163+02:00) health=(applnc:255 lm(t):3(2018-05-17T13:52:14.912+02:00) svc's)
---------------------------------------------
clusterTime=<diff=-1696892 common=2018-05-22T16:34:20.092+02:00 local=2018-05-22T17:02:36.984+02:00 pF=<displForm=0 offsSt=0 offsVlu=7200 lm(t):2(2018-05-17T13:52:11.446+02:00)>>
0 Helpful

Go to solution
Robert Burns
Cisco Employee
Cisco Employee
In response to kthned

‎05-22-2018 08:14 AM

Suggest trying to clean reload the APIC:

 

1. Decomission the APIC2 from the UI - give it 10mins to complete.

2. From APIC2 CLI:

   acidiag touch clean

   acidiag reload

3.  Once its boots back up, recommission APIC2 fro the UI.  Allow 10mins to complete.

 

This should clean up the APIC's stale NTP config.  If this doesn't work, you can try to re-setup the APIC using:

1. Decomission the APIC2 from the UI - give it 10mins to complete.

2. From APIC2 CLI:

    acidiag touch clean

    acidiag touch setup

    acidiag touch reload

3.  Once its boots back up, recommission APIC2 fro the UI.  Allow 10mins to complete.

 

After this you'll need to re-assign the appropriate details via KVM console (IP, Hostname, etc).  The rest of the policies should be pushed once it joins the cluster.  Give it some time to go fully fit and it "should" upgrade itself to the Cluster's FW level.

 

Robert

Go to solution
kthned
Level 3
Level 3
In response to Robert Burns

‎05-22-2018 08:24 AM

just a side question... is there any way to get the CIMC address from GUI or configuration of APIC ?
0 Helpful

Go to solution
Robert Burns
Cisco Employee
Cisco Employee
In response to kthned

‎05-22-2018 08:27 AM

No. IMC is completely out of band for the platform. Has no interaction with the Host APIC OS. Either you know the IMC IP, or you can reboot it with a monitor/keyboard locally connected and watch the console. The IMC details will be shown on screen during the POST process.

Robert
0 Helpful

Go to solution
kthned
Level 3
Level 3
In response to Robert Burns

‎05-22-2018 08:39 AM - edited ‎05-22-2018 08:44 AM

 

ok it was  :)

 acidiag reboot

 

0 Helpful

Go to solution
Robert Burns
Cisco Employee
Cisco Employee
In response to kthned

‎05-22-2018 08:41 AM

Yes.  Sorry typo.  'acidiag reboot'.

0 Helpful

Go to solution
kthned
Level 3
Level 3
In response to Robert Burns

‎05-22-2018 09:04 AM - edited ‎05-22-2018 09:05 AM

I did as you stated... Now come another hassle.
as APCI 1 & 3 are ver 3.1.2o and APIC 2 is 3.1.1i.
APIC 2 heath is 'Data Layer Partially diverge'

F1824
critical
2018-05-22T17:54:35.607+02:00
Raised
topology/pod-1/node-1/lon/incmptblClsPeer-version_incompatible_with_cluster-2-3.1(1i)-apic-3.1(2o)
Peer controller is incompatible: Version incompatible with cluster. The controller ID is 2, its version is 3.1(1i). Cluster Version is apic-3.1(2o)
0 Helpful

Go to solution
Robert Burns
Cisco Employee
Cisco Employee
In response to kthned

‎05-22-2018 09:48 AM

Let's run the upgrade manually against APIC2.  If you have the APIC ISO for 3.1(2o) you can mount it to the IMC vMedia, and boot the APIC.  This will auto-upgrade APIC2.  After it completes, you should be able to reboot it and into the cluster successfully.

 

Robert

0 Helpful

Go to solution
kthned
Level 3
Level 3
In response to Robert Burns

‎05-22-2018 09:51 AM - edited ‎05-22-2018 09:52 AM

Hi Robert,
I just finish downgrading the other two APICs 1 & 3 to 3.1.1i.
Now it seems they are all fully fit.

 

However your second option is worth a try. As downgrading other two APICs is a bit risky ?


Thanks for your kind help !
// Umair

0 Helpful

Go to solution
Robert Burns
Cisco Employee
Cisco Employee
In response to kthned

‎05-22-2018 09:54 AM

Are you going to re-attempt the upgrade?  Ensure the NTP policy across all APICs is consistent before doing so.

 

Robert

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents

Save 25% on Day-2 Operations Add-On License