6296 FI and IOM 2208 : Server and Uplink 4-8 link change

[flamegri11ed]

Hi UCS Community,

We have 4 Server links per chassis per FI with our 3 chassis (2 x 16 B200 M3 and 1 x 3 B200 M5)

We have 6 Uplinks per FI to the 5548UPs.

Is it advisable to insert all cables (to fully populate the IOM) into all 3 chassis and the 5K's first and then to start the process starting with Chassis 3  or do I start with Chassis 3 for example with the procedure below

Chassis no.3

Plugin 4 new cables into A channel to appropriate FI e.g. Primary FI

Plugin 4 new cables into B channel to secondary Fi Subordinate

Change all-new inserted cable ports in UCSM to Server ports

Change Global Chassis policy from 4 Link to 8 Link

Vmotion all VM's from ESXi hosts to the ESXi hosts on Chassis 1,2

Shutdown Esxi hosts on Chassis 3

Acknowledge chassis

Wait 3 minutes for results and visibility of the 8 link change

Power up blades (Esxi hosts) and vmotion

Follow the same procedure for the other two Chassis No.1 and No.2

The downtime will only affect the Chassis that will go through the acknowledgement process?

Any advice would be appreciated.

[Wes Austin]

Everything you outlined sounds correct. You dont need to change your chassis discovery policy to 8 links, it will automatically utilize the new links after you acknowledge the chassis. You could even just add links between IOM and FI and ack the IOM by itself on a per fabric basis. (A side then B side.)

Only downtime during ack will be on the chassis being ack. To further minimize the impact, just ack single IOM A vs B at a time. Then you would not need to shutdown the ESXi hosts.

[flamegri11ed]

Thank you again @Wes Austin 

I have questions about :

the discovery policy remains on a 4 link, but each chassis does have its individual acknowledgement?

the IOM's are now fully populated (with the extra new ports 5-8) on all 3 chassis prior to IOM acknowledgement and one IOM

acknowledgement is done at a time A first, then B side.

IOM acknowledgement is done per A side then B side. ESXi hosts are still up but in maintenance mode. VM's are  V Motioned to chassis

1 and 2.

Once all IOM's have been acknowledged, e.g. Chassis 3, Do I then acknowledge the Chassis (No.3)

This is the procedure that will cover the other 2 chassis?

"You could even just add links between IOM and FI and ack the IOM by itself on a per fabric basis. (A side then B side.)"  I am not sure what you meant by this Wes.

Do you mean just do the IOM acknowledge per A side and B side,leave the chassis acknowledge for a later date till downtime is available. I am not sure what you mean by this.

[Wes Austin · Accepted solution]

You dont need to acknowledge the entire chassis. You can just ack each IOM (A and B side) one at a time and it will have the same effect as if you acknowledged the whole chassis.

The chassis discovery policy is only pertinent to "discovery" of the chassis. So changing the policy from 4 to 8 links is essentially saying " Any new chassis that I discover, ONLY discover it if there are 8 interfaces online between the FI and the IOM" It does not impact any of the currently discovered chassis.

Let me know if you need further clarity.

[flamegri11ed]

Hi @Wes Austin 

Thanks again for the assistance.

Added the 2 Network uplinks first(5K) and then proceeded to add all 24 cables to the FI's and IOM ports 5 -8, but only inserting them on the FI side.

Ports in the FI A+B were changed to Server for the 1st chassis as it was the first added first. ESXi hosts  put in Maintenance mode.Cables were inserted into IOM 1+2.

First IOM on FI A acknowledge. Waited for errors to clear. Proceeded with FI B IOM acknowledge. After errors cleared, hosts were taken out of maintenance mode. Proceeded with 2nd and 3rd chassis in a similar fashion.

[flamegri11ed]

Port Licenses were uploaded and installed after inserting cables into IOM's and Uplink ports to clear errors