ISSU or Cold Restart + nexus 7010 with Dual SUP 2E

[kmothukuri]

Hi Experts ,

We are planning for OS upgrade on Nexus 7010 switches ( each has dual SUP 2E) from 6.2.2 to 6.2.20 .

As per Cisco documention , we have to go for intermediate release ie 6.2.8a before going for 6.2.20.

When we searched few blog related to OS upgrade using ISSU , most of the guys were preferring cold restart procedure instead of ISSU. in Few cases , ISSU installation got stopped at the middle.

Example is as below.

The ISSU installed correctly, although during the failover to the standby supervisor the OSPF process stalled and never recovered. I had to reload the entire chassis and roll back to original code. So for me....ISSU just needs an E, and we have the word ISSUE.

Need valuable suggestion from experts on the below points .

1. ISSU (or) Cold restart , which method is most successful one 

 In the below cisco documentation , it was mentioned that 

 "Ensure your hardware is supported, this is very important. ie investigate EPLD updates, which are updates to actual hardware line cards but they were not needed. Please see the following link for EPLD assistance -

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/best_practices/cli_mgmt_guide/cli_mgmt_bp/epld_upgrade.html

2.Does nexus 7010 switches requires EPLD upgrade too  ?

When we executed few commnds related to EPLD on nexus 7010 switches we are getting error messages and when we serached in google we could see below one.

Symptom : The show install epld status command returns “Could not pull epld logs from plog.”

Conditions : This problem sometimes occurs under these conditions:

Upgrade the active supervisor PMFPGA. After the supervisor resets itself and boots up, enter the show install epld status command.

Workaround : This issue is resolved.

• CSCuh57710

Suggetions would be appreciated as we are performing OS upgrade on nexus 7010 first time.

[Mark Malone]

Hi

From my experience , we usually do ISSU but yes it has broken a couple of times during the process and it takes a lot longer than a hard reboot , worked the last time well , time before that it completely hung due to a bug we had no control over also took down the 2ks that were directly connected , that was in the older H design before the X setups

If you can take the outage per switch at a time i would hard reboot but if like us your not giving that option you may have to use ISSU

You will know quick enough anyway where it will fail or not when you run the compatibility checks first , you must do this with NX-OS upgrade , it will tell you if its still going to be disruptive or not , but if it says your ok and you still hit a bug not much you can do there

I find it a lot more stable than the campus ISSU upgrades , its only failed maybe twice for us out of 6 times , while campus fails quite a lot even when following the docs , from one reason or another

7010s do EPLD upgrades too , there including under the 7000 series in the docs and in 6.2(8a) there is an EPLD upgrade so the bootup may take longer

Target Release	Current Release Supporting Direct ISSU Upgrade to Target Release	Current Release Supporting ISSD to Target Release
NX-OS Release 6.2(22)	6.2(20a), 6.2(20), 6.2(18), 6.2(16), 6.2(14), 6.2(12), 6.2(10), 6.2(8a), 6.2(8b)	No Support
NX-OS Release 6.2(20a)	6.2(20), 6.2(18), 6.2(16), 6.2(14), 6.2(12), 6.2(10), 6.2(8a), 6.2(8b)	No Support
NX-OS Release 6.2(20)	6.2(18), 6.2(16), 6.2(14), 6.2(12), 6.2(10), 6.2(8a), 6.2(8b)	No Support
NX-OS Release 6.2(18)	6.2(16), 6.2(14), 6.2(12), 6.2(10), 6.2(8a), 6.2(8b)	No Support
NX-OS Release 6.2(16)	6.2(14), 6.2(12), 6.2(10), 6.2(8a), 6.2(8b)	No Support
NX-OS Release 6.2(14)	6.2(12), 6.2(10), 6.2(8a), 6.2(8b)	No Support
NX-OS Release 6.2(12)	6.2(10), 6.2.(8a), 6.2(8b)	No Support
NX-OS Release 6.2(10)	6.2(8a), 6.2(8b), 6.2(8)	No Support
NX-OS Release 6.2(8b)	6.2(8a), 6.1(5a), 5.2(9a)	No Support
NX-OS Release 6.2(8a)	6.2(2), 6.2(2a), 6.2(6), 6.2(6a), 6.2(8), 6.1(3), 6.1(4), 6.1(4a), 6.1(5), 5.2(7), 5.2(9)	6.2(2),6.2(2a),6.2(6),6.2(6a),6.2(6b).6.2(8)
NX-OS Release 6.2(8)	6.2(2), 6.2(2a), 6.2(6), 6.2(6a)	6.2(2),6.2(2a),6.2(6), 6.2(6a)
NX-OS Release 6.2(6b)	6.2(2), 6.2(2a), 6.2(6), 6.2(6a)	6.2(2),6.2(2a),6.2(6), 6.2(6a)
NX-OS Release 6.2(6a)	6.2(2), 6.2(2a), 6.2(6), 6.1(3), 6.1(4), 6.1(4a), 6.0(4), 5.2(7), 5.2(9)	6.2(2),6.2(2a),6.2(6)
NX-OS Release 6.2(6)	6.2(2), 6.2(2a), 6.1(3), 6.1(4), 6.1(4a), 6.0(4), 5.2(7), 5.2(9)	6.2(2),6.2(2a)
NX-OS Release 6.2(2a)	6.2(2), 6.1(2), 6.1(3), 6.1(4), 6.1(4a), 6.0(4), 5.2(4), 5.2(5), 5.2(7), 5.2(9)	6.2(2)
NX-OS Release 6.2(2)	6.1(2), 6.1(3), 6.1(4), 6.1(4a), 6.0(4), 5.2(4), 5.2(7), 5.2(9)	No Support

Always read the release notes before each upgrade make sure there is nothing thats going to hit you , but this is a good doc for quick view of the upgrade process and what you will see

http://www.firewall.cx/cisco-technical-knowledgebase/cisco-data-center/1220-nexus-7000-nx-os-upgrade-via-issu.html#

example i see this in the release notes whihc may effect you

When you perform ISSU to Cisco NX-OS Release 6.2(16) from earlier Cisco NX-OS 6.x releases, you need to reload the F2 module or the switch to avoid congestion in the ingress traffic on ports in a F2 module.

.....

When you perform an ISSU to Cisco NX-OS Release 6.2(8)/6.2(8a)/6.2(8b) from an earlier release, the service “pixm_vl” in the switch might crash. Make sure that you review all the Upgrade/Downgrade issues in the “Resolved Caveats” section of the respective releases.

[kmothukuri]

Hi Mark ,

Thank you very much for the reply and information .

We are going to take downtime as we are planning for hard reboot of switches after each code upgrade.

Could you please provide us step by step procedure to upgrade nexus 7010 ( hard reboot ie traditional way of upgrade) , which has dual sup 2E module in it.

you have mentioned that "When you perform an ISSU to Cisco NX-OS Release 6.2(8)/6.2(8a)/6.2(8b) from an earlier release, the service “pixm_vl” in the switch might crash. Make sure that you review all the Upgrade/Downgrade issues in the “Resolved Caveats” section of the respective releases. "

Since we are not going with ISSU procedure to upgrade nexus 7010 switches, does the above-mentioned bug  still hit when we hard reboot the switch with 6.2(8a) OS ?  

[Mark Malone]

Hi
No its a bug specific to ISSU upgrade

if your doing a hard reboot , you add the new image to the each supervisor flash , including the standby sups set the boot statements save it , check the boot and then reload

Performing a Traditional Upgrade or Downgrade (Chassis Reload)

This procedure is recommended for these specific scenarios:

In lab environments where continuous uptime is not a requirement

In production environments in the unlikely event that an upgrade needs to be downgraded in a timely manner

In situations where ISSU or ISSD is not supported for the respective images

Before you begin

Save and back up all configurations before reloading the system to load the new software.

Power down unsupported line cards.
Procedure
Step 1

Configure the boot variable for the Cisco NX-OS software kickstart image.


switch(config)# boot kickstart bootflash:n7000-s1-kickstart.7.2.0.D1.1.bin

Step 2

Configure the boot variable for the Cisco NX-OS software system image.


switch(config)# boot system bootflash:n7000-s1-dk9.7.2.0.D1.1.bin

Step 3

Save the running configuration to the startup configuration.


switch# copy running-config startup-config vdc-all

Step 4

Verify that the "Current Boot Variables" and the "Boot Variables on the next reload" match the expected image.

Current Boot Variables:

sup-1
kickstart variable = bootflash:/n7000-s1-kickstart.7.2.0.D1.1.bin
system variable = bootflash:/n7000-s1-dk9.7.2.0.D1.1.bin
sup-2
kickstart variable = bootflash:/n7000-s1-kickstart.7.2.0.D1.1.bin
system variable = bootflash:/n7000-s1-dk9.7.2.0.D1.1.bin
No module boot variable set

Boot Variables on next reload:

sup-1
kickstart variable = bootflash:/n7000-s1-kickstart.7.2.0.D1.1.bin
system variable = bootflash:/n7000-s1-dk9.7.2.0.D1.1.bin
sup-2
kickstart variable = bootflash:/n7000-s1-kickstart.7.2.0.D1.1.bin
system variable = bootflash:/n7000-s1-dk9.7.2.0.D1.1.bin
No module boot variable set

Step 5

Verify that the image location and the image name match the above boot statements. In redundant supervisor chassis, the images auto-synchronize from to standby once the boot statements are set.

switch# show boot auto-copy list
switch# dir bootflash://sup-active/
161980383 May 15 17:52:03 2015 n7000-s1-dk9.7.2.0.D1.1.bin
29471232 May 15 18:01:38 2015 n7000-s1-kickstart.7.2.0.D1.1.bin

switch# dir bootflash://sup-standby/
161980383 May 15 18:04:55 2015 n7000-s1-dk9.7.2.0.D1.1.bin
29471232 May 15 18:06:18 2015 n7000-s1-kickstart.7.2.0.D1.1.bin

Step 6

After you verify the image location and statements, reload the Cisco NX-OS device.

switch# reload

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/7_x/nx-os/upgrade/guide/b_Cisco_Nexus_7000_Series_NX-OS_Software_Upgrade_and_Downgrade_Guide_Release_7-x.html#task_39E26688E1204F8CAAE876450A575E73

[kmothukuri]

Hi Mark , In Hard reboot ie non-issu procedure , you have mentioned that power down unsupported modules before upgrading the OS ,  is there any way to check the compatibility in Cisco ? 

[kmothukuri]

Hi Mark ,

Information provided by you is very helpful.

Is there any way to check whether existing modules on my nexus 7010 supports new image which I am going to load ?

I have one more question related to EPLD upgrade .In the cisco documentation it was mentioned that 

 EPLD image updates are not mandatory unless otherwise specified. The EPLD image upgrades are independent from the Cisco NX-OS In Service Software Upgrade (ISSU) process, which upgrades the system and kickstart images with no impact on the network environment.

How to determine whether a new image needs a EPLD upgrade .EPLD upgrade is only for I/O modules, so in case EPLD is not upgraded then will it power down existing modules.

We have EPLD image on the nexus 7010 but it's not installed.

Any command which shows EPLD versions of the existing modules.

If we go with hard reboot ie NON-ISSU procedure to load the image into nexus 7010, can we load target image directly onto nexus 7010,  instead of loading intermediate images and reloading the box with an intermediate image before reloading the box with a target image

Many Thanks in advance.

[Mark Malone · Accepted solution]

Is there any way to check whether existing modules on my nexus 7010 supports new image which I am going to load ?

yes use the show install all command like below with your image name , tahts really for ISSU checls but will check image compatibility too , you in same train so there wont be any issues with support anyway

example
show install all impact kickstart bootflash://sup-local/n5000-uk9-kickstart.7.0.6.N1.1.bin system bootflash:n5000-uk9.7.0.6.N1.1.bin

for hard reboots you dont need interim version only on ISSU jumps thats usually required

You wont need an EPLD upgrade in same train , ive upgraded my 7ks several times and never upgraded EPLD


################################### show install all status
This is the log of last installation.


Verifying image bootflash:/n5000-uk9-kickstart.7.3.3.N1.1.bin for boot variable "kickstart".
SUCCESS

Verifying image bootflash:/n5000-uk9.7.3.3.N1.1.bin for boot variable "system".
SUCCESS

Verifying image type.
SUCCESS

Extracting "system" version from image bootflash:/n5000-uk9.7.3.3.N1.1.bin.
SUCCESS

Extracting "kickstart" version from image bootflash:/n5000-uk9-kickstart.7.3.3.N1.1.bin.
SUCCESS

Extracting "bios" version from image bootflash:/n5000-uk9.7.3.3.N1.1.bin.
SUCCESS

Extracting "fexth" version from image bootflash:/n5000-uk9.7.3.3.N1.1.bin.
SUCCESS

Performing module support checks.
SUCCESS

[kmothukuri]

Hi Mark,

We are planning to proceed with traditional way ie cold restart (NON-ISSU) procedure for OS upgrade.

The last point I would like to check with you is,

After changing the boot version to the new code and before rebooting nexus 7010 switches ( which has dual SUP2E modules), is it mandatory to remove standby SUP 2E module so that switch boots with newer code and then insert back the standby SUP2E as soon as switch comes online, which syncs with Active SUP2E? 

Many Thanks...

[kmothukuri]

Hi Mark , n7k1 ---VPC --- n7k2 ..both have 6.2(2a) image ..if I reload n7k1 with new image is 6.2(20) and once n7k1 comes online , will three be any breakage of VPN peer-link due mismatch of OS versions of n7k1 and n7k2 ? After 20 minutes ,reload of n7k2 with 6.2(20) will be done once n7k1 is working fine with new code 6.2(20) ..what could be the status of vpc and peer keepalive in the above scenario and how would be the traffic flow from down stream switches which are connected to both n7ks using fabric path.. scenario : n7k1 (6.2(20)) ---- vpc ----n7k2 (6.2(2a)) ..

[Mark Malone]

Hi Ive been away hence slow reply , there will be a hit to the side you upgrade that's why its not advised to have orphan devices (single homed devices on 1 switch only ) its difficult to state the times etc as it depends on each individual setup and traffic flows just have an appropriate window , i always give 1 hour per switch clearance but usually takes me half that for full reboot of 2 and testing everything from scratch , have a full test plan of what was up before , take capture off mac and arp tables , routing table , stp and trunks outputs , vpc outputs etc , so if anythings wrong you have the details to correct it , plan for the worst possible scenario and you will be covered

read this below you can mitigate the break

Upgrading a vPC Domain with the Traditional Reload Method

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/tech_note/vpc_upgrade.html#pgfId-43316

When you use a traditional reload method to upgrade or downgrade your vPC domain, convergence times can be expected.
Before You Begin

Before you start the procedure, review the following information so that you know what to expect:

Expect and plan for some downtime during the traditional reload upgrade process. Layer 3 protocols converge during the reload and bringup process. Exact convergence times depend on the traffic flow, scale, and configuration. We recommend that you perform the traditional reload upgrade process during a time when downtime can be tolerated. If a nondisruptive software upgrade is desired, use the ISSU process.
Devices that are connected to orphan ports lose connectivity for the duration of the reload process for the respective switch to which they are single-homed.
We recommend that you use various vPC enhancements whenever possible to optimize the vPC domain in your environment. Review all vPC enhancements individually to determine if they apply to your environment.

– Delay restore (default)

– Graceful consistency check (default)

– ARP synchronization

– Auto-recovery, and auto-recovery reload delay

– Peer gateway

– PIM prebuild (not supported on F2 Series modules)

You can find additional enhancements and information in the following guides:

– Design and Configuration Guide: Best Practices for Virtual Port Channels (vPC) on Cisco Nexus 7000 Series Switches

http://www.cisco.com/en/US/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

– Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide

http://www.cisco.com/en/US/partner/products/ps9402/products_installation_and_configuration_guides_list.html

vPC does not support role preemption.
If you use a different procedure for your specific environment, it is possible that the vPC peers end up in the roles of “primary, operational secondary” and “secondary, operational primary.” These states are fully functional and the vPC peers can be left in that state. There is no need to alter those roles.
You do not need to alter the role priority on either vPC peer during the upgrade process. Altering the role priority does not affect the system until the peer link is flapped. If altering the role priority is desired, be aware that altering the role priority and then flapping the vPC peer link causes vPCs to reinitialize, can possibly trigger STP root convergence, and result in traffic loss. The following warning message appears when you attempt this type of change.