cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
6696
Views
5
Helpful
3
Replies

Nexus 5010 Upgrade Issue - Has Anybody Seen This?

Ryan Curry
Level 1
Level 1

Hi all, last night I upgraded our Nexus 5010's from version 4.1.3(N2)1a to 5.0.2(N1)1.  In our main data center, we have two Nexus 5010's (peer linked) with two Nexus 2148's hanging off them (a vpc with a leg to each 5010 on each 2k).  I've only performed an upgrade on the Nexus 5010's with the FEX's attached in a non-production setting so that I could document the upgrade process and validate my upgrade plan.  I performed the following steps last night to upgrade the system:

  • On N5K-1 I ran the install all kickstart <kickstart image> system <system image> and allowed the upgrade to continue knowing it would be disruptive.
  • During the process, I validated that the FEX's stayed online from N5K-2.
  • Once N5K-1 rebooted and displayed "image upgraded" via the "show fex" command, I rebooted FEX101 via N5K-2
  • After FEX101 rebooted and attached to N5K-1 (now that the images match) and everything looked good, I rebooted FEX102 via N5K-2
  • FEX102 attached to N5K-1 after the reboot and all was looking good.
  • (Here's where I may have messed up) On N5K-2, I set the kickstart and system boot variable to boot to the new images and rebooted N5K-2
  • When N5K-2 came back up, the FEX's kept flapping between Registering and Offline, after more investigation I found that the peer-link hadn't been established between the N5K's.  What's MORE interesting is that there was no VPC domain defined in the running-config; I could see it was defined in the startup-config but it was like something whacked out on boot.  At that point we decided to reboot N5K-2 hoping that the config would load properly.
  • Once N5K-2 rebooted the peer link established and the FEX's came back online and were alive on both N5K's.
  • At this point, we found that the port configurations for all FEX ports had been wiped on N5K-2 (N5K-1 still had their configurations).  This caused about 7 of our servers to freak out (I'm thinking because data was trying to come back across N5K-2 and was getting blackholed); I just copied the port config from our backup config and applied it to N5K-2, at that point everything was back up.

Soooooooooo, this begs two questions: a) when upgrading N5K-2, should I have performed the install all command instead of just setting the boot parameter (I swear I found an article saying NOT to run the command on the second 5K) and b) would this have caused the headache with the peer link (which I assume caused the ports to lose their config).

Thanks all, since we don't perform these upgrades all that often I want to be prepared for the next time we run through this process.

1 Accepted Solution

Accepted Solutions

Yes the configuration needs to be saved first before attempting an upgrade to make sure the last changes are saved.

The good news is that from the newer code you have, you can now use fex provisionning which will allow you to save the fex configuration and replace a FEX without loosing the configuration.

Here is the documentation showing you how to configure FEX provisioning:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_config_sync_ops.html

View solution in original post

3 Replies 3

Ryan Curry
Level 1
Level 1

After looking at the upgrade guide (http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/upgrade/421_n2_1/Cisco_Nexus_5000_Series_NX-OS_Software_Upgrade_and_Downgrade_Guide__Release_421.pdf), we're using the bottom option (Dual Homed Fiber Extenders) and it looks like I followed the correct steps EXCEPT I saved before I rebooted.  According to step 8, saving before the reboot of the secondary 5K may wipe out the 2k's configuration because they're no longer attached.  I'm suspecting that is what caused my headache last night.

Yes the configuration needs to be saved first before attempting an upgrade to make sure the last changes are saved.

The good news is that from the newer code you have, you can now use fex provisionning which will allow you to save the fex configuration and replace a FEX without loosing the configuration.

Here is the documentation showing you how to configure FEX provisioning:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_config_sync_ops.html

Thanks for the Link Lucien, I'll be checking into that option!