ACI Pod Move (Multi-Pod Configuration)

[zachartl]

Hello,

We currently utilize an ACI Multi-Pod architecture. We've 3 Pods. The 3rd Pod was set up in our Main Data Center. We need to take that Pod to it's location. Yesterday, I tried to place the Two Spines and Leaf switch, comprising the Pod, in to Maintenance Mode. This produced a very stressful few moments where I briefly lost contact with the Controller after having placed One Spine and One Leaf switch in Maintenance Mode. I didn't get as far as putting the Second Spine in Maintenance Mode and then I got an error pertaining to Shards. I know the database is sharded and I thought this a potentially bad thing. I stopped and then reacquired my session with the Controller. I then put the Pod Spine and Leaf switch back into production - Commission.

Is there a good method to inform the Controller/s that the Pod is to be placed in Maintenance/Off-line Mode so we can relocate that Pod? We've 3 Controllers. Two located in Pod 1 and One Controller located in Pod 2. The Pod we want to move will not host a Controller.

Thank you,

Terry

[Robert Burns]

Strange, I wouldn't expect any of the controllers to be affected unless there was some dependency between Pod2 and Pod3 from a connectivity perspective.  When putting switches into maintenance mode, start with the Leafs, then Spines.  Work your way upward towards the IPN.

When you decommissioned the first Spine, you may need to give it a little time to settle down ~5-10mins.  During this time, you should retain full connectivity to all controllers in your case - did your Cluster Health remain in Fully Fit state or go into a Degraded state?

Robert  

[zachartl]

Hi Robert,

As best I can tell, the Cluster Remained Fully Fit. I was a bit rattled at what seemed like an hour of Controller session loss when in fact it was a good 10-15 seconds. The Shard error message was a bit unsettling too. The Pod we need to move has been in operation for sometime and not doing anything yet. All has been well really. Version 4.2.7f

Thank you for having responded and providing the great information.

Have a great weekend!

Terry

[Claudia de Luna]

Hi @zachartl ,

It's not unusual for the GUI to "get distracted" is the best way I can say it when making big changes.  I know when I push big config payloads that happens but its nothing prolonged.  I think it is more accurate to say it's less responsive while processing.  Shards are perfectly normal.  What you describe is not normal in my experience and the fact you got an error about shards makes me ask what your cluster state was like before you started your Maintenance Mode work (all up and fully fit?) and if you had unresolved Faults in the fabric.  I suspect you routinely check that but have to ask.

[zachartl]

Hi Claudia,

The Cluster has been up and as far as I can tell, in a fully fit state. I hadn't seen the Shard Error message before and so I found it unsettling. I wonder if it was a matter of the 3rd APIC being at the other Data Center (Pod 2) not being updated quick enough? I do have some faults in the fabric but they're usually minor ones. Would those add to the load or overhead of the ACI database/s?

Thank you for your response today.

Have a great weekend,

Terry