Nexus 5548UP ISSU upgrade

[Joris Deprouw]

Hi all,

I want to upgrade our 2 5548UP nexusses with Nexus 2000 attached with VPC.

Our current version is 5.2(1)N1(7) and we want to upgrade to version 7.0(5)N1(1). 

Is there a possibility to do this without interruption? Is this setup eligible for ISSU?

Does anyone have experience with this?

Thanks,

Best Regards,

Joris

[glen.grant]

I'll be interested to see the answer , we tried to do this exact thing over the weekend and our n5k's had almost the same exact codes on it and it seemed to take down the whole vpc domain when we went to 7.X code .  We had gone thru all the prechecks and recomended actions  and everything looked fine before hand . We used the install all command and as soon as the 7.x started loading up we could not reach attached devices or even the other n5k switch in its mgt vlan .  We had gone from 5.1 code to 5.2 code which seemed to go fine and then had issues going to 7.X code. As soon as we back it out to 5.2 code everything normalized .  All the prechecks for 7.X code looked normal and it said it should be a non disruptive rolling upgrade,  NOT!!! We currently have a TAC case to try and determine what is going on with this as this is the 2nd time we tried to put 7.x on and it failed .  Maybe we have something misconfigured I don't know but it passed all prechecks  .  Upgrading IOS boxes was much less of PIA than these things .

[Joris Deprouw]

Hi all,

I have some more info now and ISSU will not be possible in our case since the Nexus 5548 has a L3 module inserted.

So we will do the disruptive upgrade. But maybe we can do this without losing connectivity to the edge devices.

Let's say we start upgrading the switch which has VPC primary role (like recommended by Cisco). Since the fexes are connected with vpc, does forwarding of traffic continues over the vpc link to the 2nd Nexus while we reboot the first 5548?

How about the versions? When we upgrade the primary Nexus, I assume, the fexes will be upgraded too. Won't there be a version mismatch with the secondary Nexus (which hasn't been upgraded yet) when the first Nexus reloads?

Thanks,

Br,
Joris

[Logan Kampsnider]

Hi Joris.

You can "minimize" disruption during a non-ISSU upgrade of your two 5Ks, as long as the FEXes are dual-homed to the 5Ks via vPC, which sounds like they are. The process works like this...

1. Upgrade the primary 5K first with the install all command. Upgrading the primary will also upgrade the FEXes, but will not reboot them. The primary is rebooted however. During the reboot the traffic traversing the FEXes will utilize the vPC link still active to the secondary 5K.

2. Upon the primary 5K being rebooted, it is now on the new code, whereas the FEXes and secondary 5K are still on the old. The primary will show the FEXes offline at this point. This is normal, as the secondary will still see them as online.

3. On the secondary, change your boot variables to the new code bins. Copy the running config to the startup config. DO NOT RELOAD YET. You now want to reload your FEXes, one at a time, verifying after each individual FEX reboot that they come back. Note, once the FEXes are rebooted they are running the new code, so the secondary will now show them as offline, whereas the primary can now see them as online. 

4. Once all the FEXes are manually reloaded you will reload the secondary 5K. NOTE: you are NOT using "install all"  command. Just do a simple reload, granted that you changed the boot variables in step #3 already. Also, do not save the running configuration when prompted when reloading, as it could orphan the FEXes from the secondary (per Cisco).

5. Once the secondary comes back up, everything should be on the new code and everything should be happy. Verify 5K and FEX code versions, FEX statuses, and vPC peer health as part of your cleanup.

Hope that helps.

Logan

[Joris Deprouw]

Hello Logan,

Thanks for you feedback. Most helpfull. When you do the install all on the primary 5k, doesn't he reboot the fexes aswell when you reload the main chassis? During my lab test, the fexes were also reloaded, after the install all procedure.

I have another question. What happens during the upgrade to servers connected with vpc directly to both of the Nexus 5k chassis. Will they continue to work?

I assume the connection to the 2nd nexus will stay up during the reboot of the 1st nexus. And the other way when rebooting the 2nd nexus.

Br,

Joris

[Logan Kampsnider]

Hi Joris,

The FEXes *shouldn't* reboot when you upgrade the primary. Please see the following link for reference on exact instructions (ref: "Upgrading a Dual-Homed FEX Access Layer")

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/upgrade/503_N1_1/n5k_upgrade_downgrade_503.html

During this procedure your server's network connections should be maintained through the secondary 5K.

However, if you are not comfortable with this upgrade method, there is another that I have discussed with Cisco as being workable where you shut down half your FEX ports on the primary 5K prior to the primary 5K upgrade. This allows you to upgrade half the FEXes at a time. This is useful if you have two FEXes per rack, and your hosts/servers are dual homed to these FEXes. The procedure would look like this.

1.On the primary 5K shut down half of the FEX ports. In this situation you would shut down the ports to one FEX per rack, leaving another FEX online in that rack to maintain host reachability. 

2.Upgrade the primary 5K. This will result in the primary 5K and half of the FEXes it can reach to be upgraded and rebooted, but leaving half of the FEXes and the secondary 5K as available.

3.Verify the secondary 5K is now operational primary. If it’s not then it will not be able to upgrade. Reboot the primary 5K again if the secondary is not operational primary. (in order for FEXes to grab upgrade files from the parent 5K being upgraded, it must either be primary or in operational primary mode)

4. Upgrade secondary 5K. This will result in the secondary 5K and the other half of the FEXes to be upgraded and rebooted.

5.Turn up all FEX ports the on primary 5K.

6.Verify stabilized and upgraded network.

Hope that helps.

Logan

[Nicolas Blancpain]

Great way to smoothly upgrade ! I had imagined this for my next operation.

Just one precision, when you state that the Nexus must be "primary", do you mean VPC role "primary" ? and or operational role "primary" ?

> show vpc

...

vPC role                          : primary, operational secondary         <==========