How to sequence N2Ks reload during N5K non-ISSU upgrade ?

[guillerm]

Environment :

2 N5K (56128) + 8 N2K, each of them is double-attached to each N5K

NX-OS 7.0.7

Those 2 N5Ks perform L3 functions (they are the Core switches in this environment), so non-disruptive ISSU upgrade cannot be performed (as stated here : http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5600/sw/upgrade/702_N1_1/n5600_upgrade_downgrade_700.html#pgfId-732571 );

Because of that, the upgrade sequence of these 2 N5Ks is as follows (checked recently) :

1) upgrade of N5K #1 via "instal all ..." command that results in the following steps :

a) at the beginnig, it indicates after the 1st checking phase that the upgrade will be disruptive ("Switch will be reloaded for disruptive upgrade.")

b) upgrade of all N2K takes place but without rebooting them

c) reload of N5K #1

d) then, all FEX appear in "AA Version Mismatch" status via show fex and there is no way to reload them manually, one at a time, via "reload fex xxx" : it says "FEX xxx is not present" ; I tried this reload command only from N5K #1

2) upgrade of N5K #2 via "instal all ..." command that results in the following steps :

a) at the beginnig, it indicates after the 1st checking phase that the upgrade will be disruptive ("Switch will be reloaded for disruptive upgrade.")

b) check of upgrade of all N2K takes place but without rebooting them

c) reload of N5K #2

d) then, all FEX are reloaded all together

This is a bit annoying since some WAN/Internet access are done via redundant routers/firewalls connected on different N2Ks;

So, as all these N2Ks are reloaded all together, the routers/firewalls redundancy cannot be used, and WAN/Internet flows are disconnected during about 3mns, while N2Ks are reloaded ;

Questions :

Is there a way to avoid, in this disruptive upgrade, the reload of all N2Ks together ?

it would be nice to be able to reload all N2K located in the room where 1 N5K is upgraded ;

I was wondering if the "reload fex xxx" command would have worked properly from the N5K #2 just after upgrading/reloading N5K #1 ...

this could have been a solution ;

thanks for any  feedback

[Reza Sharifi]

Is there a way to avoid, in this disruptive upgrade, the reload of all N2Ks together ?

Since your FEXs are attached to both 5k, there is no way to avoid that.  The only way you can avoid that by connecting some FEXs to one of the 5k and some to the other.  In this secnario you can avoid disrubtion, but your servers MUST connect to 2 different FEXs.

HTH

[Ganesh Hariharan]

Hello ,

As this disruptive upgarde then you should take care of host are connected to both the Fex to avoid massive disruption.

But if you have taken downtime window , i woudl suggest proceed on sequential way and do post upgarde testing relgiously.

If you have choosen non-disrutive mode then it wont have reloaded because Cisco NX-OS supports in-service software upgrades (ISSUs) that allow a Cisco Nexus device and any connected FEXs to be upgraded without any traffic disruption (with a brief control plane disruption).

A few conditions have to be met for the system to be upgraded via ISSU process - the access layer topology should be ISSU compliant, the current and target versions should be ISSU capable and the network should be stable.

If the conditions required for ISSU are not met or if the user intends to downgrade the software version, the installation process is disruptive. For example the Cisco Nexus device and any connected FEX are rebooted, which causes a disruption. If Cisco's virtual port channel (vPC) is configured on Cisco Nexus devices, it is possible to achieve an upgrade/downgrade with very minimal traffic disruption to servers/hosts.

Hope it Helps..

-GI

[guillerm]

thanks for your answers

As I said at the very beginning, our 2 N5Ks perform L3 functions and, so, this prevents us to do a non disruptive ISSU ;

and we can't really unconfigure these L3 functions on the N5Ks, even temporarily ;

1 question I have already asked :

I was wondering if the "reload fex xxx" manual command would have worked properly from the N5K #2 (not yet upgraded) just after upgrading/reloading N5K #1

If this command is not accepted on N5K#2, another option would be to physically power off/on each N2K at a time ; would it be a good thing to do ?

This would have allowed us to reload each N2K at a time, and so, the N2Ks located in N5K#1 room first, then N2Ks located in N5K#2 room

thanks for answering these last questions

[Andrew Bailey]

I am planning on doing the same thing here. I have not done it yet, but per this guide, under "Upgrading a Dual-Homed FEX Access Layer" step 3, you can reload the FEXs with the reload command from the secondary switch.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5000/sw/upgrade/521_N1_9a/n5000_upgrade_downgrade_521_n1_9a.html#pgfId-699400

I assume others can confirm this. I will trying it out soon.

[guillerm]

hello Reza,

your suggestion could help ;

my servers are directly connected on the N5Ks (or via BladeCenter internal switches, which is equivalent in the sense that they don't cross N2K) ;

most of machines connected on N2K are physically redundant, at least important ones such as Firewall, routers, WIFI controllers ...

so, if I physically disconnect or shut the FEX interfaces between N5K#1 and the N2Ks located in the room where the N5K#2 is located, and the reverse, so that, in summary, each N5K only see its local N2Ks, you think this will solve the probem of rebooting all N2K together ?

[ajavaid1983]

You are rebooting the FEXes from the wrong switch .should have been done from N5K#2  ..N2Ks have the new code but still running on  old code. N5k1 does not see them..Here are the steps..

      

Considering all your FEXes are dual-homed   and    the switches being upgraded  are: N5K-1, the VPC primary and N5K-2, the VPC secondary.

1. Configure FEX pre-provisioning for all FEXes in both N5K-1 and N5K-2.

2.  Upgrade of N5K-1 via "install all kickstart..." command . This will upgrade N5K-1 and download the new code to all the  FEXes connected. But, the FEXes won’t reload.FEXes remain operational using old code.

3.  Once upgraded N5K-1 will reboot and come back up with the new software and in VPC operational secondary role.N5K-2 becomes primary and and keeps FEXes running The FEXes won’t reload. FEXes remain operational using old code.

4.  Reload the FEXes one at a time using command “reload fex .." from N5k2...”Make sure you each FEX is up before moving to next.. The  FEXes come online with new NX-OS and associate themselves with N5K-1 which is running the new NX-OS (they will be offline with N5K-2).

5  . Finally  upgrade N5K-2 using command “install all kickstart ...” . N5K-2 will reload  and come up with new NX-OS. Once N5K-2 is fully operational as vPC secondary after the upgrade, FEXes will come online on N5K-2.



 -------------------------------------------------------------------------------------------