ACI secondary DC frabric to shutdown

[Sheraz.Salim]

Hi floks.

In some advice please. we running ACI multipod between to Data Centers. we classified these DCs as Primary and Secondary due to legacy setup prior to going into ACI deployment. The ACI deployment is in Network Centric.

we have APIC 2xAPIC in Primary DC and 1xAPIC is in Secondary and another on Standby APIC is also in Secondary DC. They are in cluster.

we doing Data center racks consolidation in our secondary DC to free up some rack space/s. therefore my question to you is.

1. what is the process/how to gracefully shudown the APIC in secondary DC, in Secondary DC we have two APIC (including the stand-alone APIC). also please could you share the document how to power them up best

2. Shutting down the secondary DC (APIC,leaf and spine) will we still be able to make changes on the APIC from the primary DC apic?

3. prior to begin the change I shall put the secondary DC (SPINE and Leafts in maintains or offline mode?) plus powering off the IPN in secondary DC. Is this correct approach? 

Thank you.

[Robert Burns]

1. To power down the 3rd APIC in Pod2 - from APIC 1 or 2, navigate to System > Controllers and right click on APIC3 and select 'Shutdown'.  This will power down the OS of that controller.  You can do the sam (I believe) with the standyb, but if not, you can simply reboot it from the CLI and during the reboot process (when it restarts), just power it off from the CIMC console.  To bring things back up, simply power them on from the CIMC console once the Pod2 switches are back online which will allow the APIC and standby to resync with the controllers in Pod1.

2.    Correct.  As long as you maintain quorum in the cluster (in your case 2 of 3 controllers online and active) then you have full functionality of the fabric.

3.  I don't believe you need to put the Spine or switches in GIR.  GIR is typically used to keep the switch powered on, but remove it from the datapath.  This is used to troubleshoot issues while the switch isn't forwarding traffic.  Once the controllers in Pod2 are off, you can simply pull the power to the Leafs and Spines in the pod.    This is how most customers handle maintenance of an entire pod.

Robert

[Sheraz.Salim]

@Robert Burns Thank you very much for getting back to me. I have just login into my APIC cluster and I see three APIC controllers. you are right the standalone APCI does not show up in the APIC Cluster GUI. I guess as it is in standalone thats why. BTW thank you for giving the step by step breaking down to show how to  shutdown the APIC in Secondary DC (pod2).

In order to connect to apic standby do I have to console to it? if do does the normal cisco console cable will work? Or you need a specific/special APIC console cable? I think I have one in my bag at home. if possible could you please give a link/documentation how to connect to CIMC or console to APIC (in our case Standby APIC pod2).

plus could you also advise one the Secondary DC apic is power down and so the standalone. which one should we bring up first. The Secondary DC apic (pod2) first or the Standalone apic (pod2) first. or both at the same time? is any order of preference? how long does it take to APIC to syn in cluster? is it few hours or few minutes?

The last question. Prior to change shall i shut down the Secondary DC apic first (pod2) and simply power off the cables (power cables) from the spines and leafs and IPNs. you already give the answer just want to re-confirm with you. once the SPINES and Leafs power back on APIC will automatically get them on board? or is there any manual work required?

looking forward to hear from you. Thank you very much.

[Robert Burns · Accepted solution]

"In order to connect to apic standby do I have to console to it? if do does the normal cisco console cable will work? Or you need a specific/special APIC console cable? I think I have one in my bag at home. if possible could you please give a link/documentation how to connect to CIMC or console to APIC (in our case Standby APIC pod2)."
RB> For the standy APIC you can open the remove KVM console via it's CIMC interface.  This allows you to power on/off the appliance remotely.  The setup instructions for CIMC are here: https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/5x/getting-started/cisco-apic-getting-started-guide-52x/initial-setup-52x.html  Essentially this is just a lights-on type of interface with a Web GUI where you can manage the appliance/server platform independently from the OS (even if the OS is shutdown, the IMC will remain online and accessible).

"plus could you also advise one the Secondary DC apic is power down and so the standalone. which one should we bring up first. The Secondary DC apic (pod2) first or the Standalone apic (pod2) first. or both at the same time? is any order of preference? how long does it take to APIC to syn in cluster? is it few hours or few minutes?"

RB> Shouldn't really matter, but I would bring up the Active (3rd) APIC first, then the standby.  It's more important they have reachability to the other APICs than just each other. 

"The last question. Prior to change shall i shut down the Secondary DC apic first (pod2) and simply power off the cables (power cables) from the spines and leafs and IPNs. you already give the answer just want to re-confirm with you. once the SPINES and Leafs power back on APIC will automatically get them on board? or is there any manual work required?"

RB>Yes, no manual intervention should be required to restore the fabric.  Assuming reachabilty is restored from the IPN in DC2 through to DC1, shouldn't be any issues.  Start with the IPN devices, power on those on first, followed by the Spines, then Leafs, then endpoints (including APICs)- working from the IPN outward.  I would recommend taking an APIC config back before the maintenance period and offload that to a remote locations (ex. SSH/SFTP server).  Also, not that its required, but if possible put a change control into effect for DC1 during the DC2 maintenance.  Again, not mandatory, but will reduce the resync period between Pods when things are restored.

Robert

[Sheraz.Salim · Accepted solution]

@Robert Burns sorry for the late responce as we were busy in the migration. the document help alot and thank you for answering my question.

Just to give you a break down what we encounters. The order of step were in pod2 we powered off the IPNs first. then SPINES and Leafs. I used the APIC cluster GUI to power off the Pod2 APIC. but as the standalone APIC was not in cluster I have login to CIMIC server. prior to logging to CIMIC sever I noted the standalone apic GUI was showing me this error when I put the username and password

"REST Endpoint user authorization datastore is not initialized - Check Fabric Membership Status of this fabric node"
as we were in change window so could not dig into this as it was stanalone server. might I need to pick this up some point.

once the ACI pod2 was in different rack the order I followed was bringing up the IPNs. SPINES and leafs and in the last I powered up the cluster APIC and later Standalone APIC. after half an hours we checked on these devices they were showing up in APIC cluster (Pod 2 APIC and SPINES/Leaf etc)

Thank you Robert Burns you were very helpful to answer my question. save a lot of time going to cisco TAC or finding an experience DC engineer to ask these questions

[aeparedes]

Hello, how are you?

Was it necessary to decommission the APIC, LEAF AND SPINE when turning them off?

Regards,

Andrés

[Robert Burns]

No.  Not necessary if you're bringing down the entire pod/fabric.

Robert

[aeparedes]

Ok Robert, thanks for your answer.
Best regards,

[aeparedes]

Ok Robert, thanks for your answer.

Best regards,

[Tomas Pizarro]

Thanks for this post!