cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
9299
Views
30
Helpful
10
Replies

Nexus upgrade incompatible

roysm
Level 1
Level 1

Hi folks

I have a  Nexus 7700 6-slot switch currently on 7.2(1)D1(1). We are are looking to upgrade eventually to nxos 8 but reading through the upgrade guides, we need to upgrade first to 7.3(3)D1(1). When I do the upgrade it stops and says that 2 modules are incompatible.  When I run "show install impact" I get the output in the attached screenshot. The other screenshot shows the modules installed. 

 

As far as I can see in the upgrade guide at https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/7_x/nx-os/upgrade/guide/b_Cisco_Nexus_7000_Series_NX-OS_Software_Upgrade_and_Downgrade_Guide_Release_7-x.html and the release notes at https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/7_x/nx-os/release/notes/7x_nx-os_release_note.html#pgfId-1362862 I do not see any incompatibilities. The upgrade guide says that ISSU is supported for 7.2(1)D1(1) to 7.3(3)D1(1) upgrade.

 

What does this mean? Can I actually do an upgrade using ISSU? Anyone seen this or have any suggestions?

 

Many Thanks

Roy

 

 

1 Accepted Solution

Accepted Solutions

Thank you for the outputs!

 

The reason the upgrade showed "image incompatible" the first time is because the kickstart and system images did not match in the "install all" command -- Sample from the outputs shared:

 

# Install all kickstart n7700-s2-kickstart.7.2.1.D1.1.bin system n7700-s2-dk9.7.3.3.D1.1.bin
Installer will perform compatibility check first. Please wait.

 

Now, with the new outputs and the proper install all command, we can see that the upgrade is compatible but still cannot be of the ISSU type due to LACP constraints (it seems you may have lacp rate fast enabled which is not supported when doing ISSUs):

 

Additional info for this installation:
--------------------------------------

Service "lacp" in vdc 1: LACP: Upgrade will be disruptive as 8 switch ports and 0 fex ports are not upgrade ready!!
<snip>
show lacp issu-impact
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.

The following ports are not ISSU ready
Eth5/15 , Eth6/15 , Eth5/17 , Eth6/17 , Eth5/7 ,
Eth6/7 , Eth5/8 , Eth6/8 ,

 I'd check the configuration of the ports listed above and make sure they are not configured for LACP rate fast. Once that is corrected, you should be good to go.

 

Hope that helps!

- Andrea, CCIE #56739 R&S

View solution in original post

10 Replies 10

nazimkha
Level 4
Level 4

Looks like from 7.2(1)D1(1) you cannot go to 7.3(3)D1(1) with ISSU you may have to go with traditional reload upgrade

Table 11. Supported ISSU Paths for the Cisco Nexus 7000 Series Platform (Cisco NX-OS Release 7.2(1)D1(1))Current ReleaseRelease TrainNon-disruptive Upgrade

Cisco NX-OS Release 7.2(1)D1(1)

7.2

7.2(0)D1(1)

6.2(20a)

6.2(20)

6.2(18)

6.2(16)

6.2(14)

6.2(12)

6.2(10)

Andrea Testino
Cisco Employee
Cisco Employee

Hi,

 

Can you share the complete output to the following commands:

 

show install all impact kickstart bootflash:n7700-s2-kickstart.7.3.3.D1.1.bin system bootflash:n7700-s2-dk9.7.3.3.D1.1.bin
dir | i i 7.3.3
show file bootflash:n7700-s2-kickstart.7.3.3.D1.1.bin md5sum
show file bootflash:n7700-s2-dk9.7.3.3.D1.1.bin md5sum

Additionally, per release notes, 7.2(1)D1(1) to 7.3(3)D1(1) is definitely a valid ISSU step as you mentioned:

 

Screen Shot 2019-04-09 at 1.04.11 PM.png

Thanks,

- Andrea, CCIE #56739 R&S

Andrea

Thanks for confirming that the upgrade should be valid. Below is the output from the commands you suggest. 

 

#dir | i 7.3.3
577735934 Apr 08 10:39:58 2019 n7700-s2-dk9.7.3.3.D1.1.bin
47450112 Apr 08 10:22:10 2019 n7700-s2-kickstart.7.3.3.D1.1.bin

 

#show file bootflash:n7700-s2-kickstart.7.3.3.D1.1.bin md5sum
8cf5911f0f07c91e087a75d13b2146c1


#show file bootflash:n7700-s2-dk9.7.3.3.D1.1.bin md5sum
f90c20c1e27e7be052d332f5909c5f6f
#show install all impact kickstart bootflash:n7700-s2-kickstart.7.3.3.D1.1.bin system bootflash:n7700-s2-dk9.7.3.3.D1.1.bin
Installer will perform impact only check. Please wait.

Verifying image bootflash:/n7700-s2-kickstart.7.3.3.D1.1.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin for boot variable "system".
[####################] 100% -- SUCCESS

Performing module support checks. [####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Extracting "lcflnn7k" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Extracting "bios" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:/n7700-s2-kickstart.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Notifying services about system upgrade. [####################] 100% -- SUCCESS




Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive rolling
2 yes non-disruptive rolling
3 yes non-disruptive reset
4 yes non-disruptive reset
5 yes non-disruptive rolling
6 yes non-disruptive rolling




Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
1 lcflnn7k 7.2(1)D1(1) 7.3(3)D1(1) yes
1 bios v3.0.24(11/10/2013):v3.0.24(11/10/2013) v3.0.29(12/15/2015) yes
2 lcflnn7k 7.2(1)D1(1) 7.3(3)D1(1) yes
2 bios v3.0.24(11/10/2013):v3.0.24(11/10/2013) v3.0.29(12/15/2015) yes
3 system 7.2(1)D1(1) 7.3(3)D1(1) yes
3 kickstart 7.2(1)D1(1) 7.3(3)D1(1) yes
3 bios v3.1.0(02/27/2013):v3.1.0(02/27/2013) v3.1.0(02/27/2013) no
4 system 7.2(1)D1(1) 7.3(3)D1(1) yes
4 kickstart 7.2(1)D1(1) 7.3(3)D1(1) yes
4 bios v3.1.0(02/27/2013):v3.1.0(02/27/2013) v3.1.0(02/27/2013) no
5 lc1n7k 7.2(1)D1(1) 7.3(3)D1(1) yes
5 bios v2.0.18(01/12/13):v2.0.18(01/12/13) v2.0.32(12/16/13) yes
6 lc1n7k 7.2(1)D1(1) 7.3(3)D1(1) yes
6 bios v2.0.18(01/12/13):v2.0.18(01/12/13) v2.0.32(12/16/13) yes


Additional info for this installation:
--------------------------------------

Service "lacp" in vdc 1: LACP: Upgrade will be disruptive as 8 switch ports and 0 fex ports are not upgrade ready!!
Issue the "show lacp issu-impact" cli for more details.

 I have included the output for "show lacp issu-impact" as well. i have assumed that with dual supervisors that ISSU would restart them 1 at a time, thereby limiting any disruption. But I have seen other posts saying that both supervisors reboot at the same time. 

show lacp issu-impact
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.

The following ports are not ISSU ready
Eth5/15 , Eth6/15 , Eth5/17 , Eth6/17 , Eth5/7 ,
Eth6/7 , Eth5/8 , Eth6/8 ,

 

Also, for the completeness, this is the full output when I attempted the install. 

# Install all kickstart n7700-s2-kickstart.7.2.1.D1.1.bin system n7700-s2-dk9.7.3.3.D1.1.bin
Installer will perform compatibility check first. Please wait.

Verifying image bootflash:/n7700-s2-kickstart.7.2.1.D1.1.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin for boot variable "system".
[####################] 100% -- SUCCESS

Performing module support checks. [####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Extracting "lcflnn7k" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Extracting "bios" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:/n7700-s2-kickstart.7.2.1.D1.1.bin.
[####################] 100% -- SUCCESS

Extracting "lc1n7k" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS

Notifying services about system upgrade. [####################] 100% -- SUCCESS




Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive rolling
2 yes non-disruptive rolling
3 no n/a n/a Incompatible image
4 no n/a n/a Incompatible image
5 yes non-disruptive rolling
6 yes non-disruptive rolling

Not all components in the image are compatible.
ISSU can't continue

 

 

 

 

 

 

Thank you for the outputs!

 

The reason the upgrade showed "image incompatible" the first time is because the kickstart and system images did not match in the "install all" command -- Sample from the outputs shared:

 

# Install all kickstart n7700-s2-kickstart.7.2.1.D1.1.bin system n7700-s2-dk9.7.3.3.D1.1.bin
Installer will perform compatibility check first. Please wait.

 

Now, with the new outputs and the proper install all command, we can see that the upgrade is compatible but still cannot be of the ISSU type due to LACP constraints (it seems you may have lacp rate fast enabled which is not supported when doing ISSUs):

 

Additional info for this installation:
--------------------------------------

Service "lacp" in vdc 1: LACP: Upgrade will be disruptive as 8 switch ports and 0 fex ports are not upgrade ready!!
<snip>
show lacp issu-impact
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.

The following ports are not ISSU ready
Eth5/15 , Eth6/15 , Eth5/17 , Eth6/17 , Eth5/7 ,
Eth6/7 , Eth5/8 , Eth6/8 ,

 I'd check the configuration of the ports listed above and make sure they are not configured for LACP rate fast. Once that is corrected, you should be good to go.

 

Hope that helps!

- Andrea, CCIE #56739 R&S

Ok, thanks for pointing that out. Something so obvious, no matter how many times I looked at the command. I really should be more careful to check I copy and paste the right things

 

Regarding the LACP ports, I am a bit confused with this message. Is "lacp rate fast" the default setting? I do not see it in the config but I am more confused. The ports in question belong to 4 portchannels, each with the command "channel-group mode active". There are no specific LACP commands specified. Also, 2 of the portchannels are trunk ports, 1 is a routed interface and the other is an access port. 

 

There are also another 5 portchannels, 3 of which are LACP, but the interfaces do not appear in the issu-impact. What is different about these ports. Sample configs below:

 

Example of portchannel reported by issu-impact:

interface port-channel13
switchport
switchport mode trunk
switchport trunk allowed vlan 851-859,888


interface Ethernet5/7
switchport
switchport mode trunk
switchport trunk allowed vlan 851-859,888
channel-group 13 mode active
no shutdown


interface Ethernet6/7
switchport
switchport mode trunk
switchport trunk allowed vlan 851-859,888
channel-group 13 mode active
no shutdown

 

Example of portchannel not reported by issu-impact

interface port-channel3
mtu 9000
ip address ***********
ip ospf message-digest-key 1 md5 3 ************
no ip ospf passive-interface


interface Ethernet5/48
mtu 9000
channel-group 3 mode active
no shutdown


interface Ethernet6/48
mtu 9000
channel-group 3 mode active
no shutdown

 

I know I can go through these and set "no lacp rate fast" but I'm concerned that only some are reported as incompatible and others are not. Why? And I assume if I change the setting on the 7700 side, I need to change it on the other side of the portchannel as well?

 

Thanks

Roy

 

 

Roy,

 

Anytime! Happy to help. I'll try my best to concisely explain the LACP rate fast dilemma here.

 

For instance, you may have a server that requires LACP rate fast, in which case you wouldn't need it configured on the actual NX-OS device since we honor the request from the server (or any given attached device.) Conversely, if you wanted the Nexus to force an attached device to run with LACP rate fast, you would then configure that on the NX-OS device itself.

 

When a server (or attached device) requires NX-OS to use rate fast (or when the Nexus requires it), ISSU is not possible because the timeout is far too short to allow the control-plane to behave as expected during an ISSU operation.

 

You can verify the timers that an interface is using by entering the following CLI:

 

Example from my box:

RTP-AGG1# show lacp interface ethernet4/8
Interface Ethernet4/8 is up 
  Channel group is 20 port channel is Po20 
  PDUs sent: 134665 
  PDUs rcvd: 4501 
<snip>
Local Port: Eth4/8   MAC Address= 0-23-4-ee-c1-e8
  System Identifier=0x7f9b,  Port Identifier=0x8000,0x408
  Operational key=32788
  LACP_Activity=active
  LACP_Timeout=Long Timeout (30s) << I am configured as "normal"
  Synchronization=IN_SYNC
  Collecting=true
  Distributing=true
  Partner information refresh timeout=Short Timeout (3s)
Actor Admin State=61
Actor Oper State=61
Neighbor: 0x308
  MAC Address= 84-78-ac-b-60-43
  System Identifier=0x8000,  Port Identifier=0x8000,0x308
  Operational key=19
  LACP_Activity=active
  LACP_Timeout=short Timeout (1s) << Attached device is requesting "lacp rate fast"
  Synchronization=IN_SYNC
  Collecting=true
  Distributing=true
Partner Admin State=63
Partner Oper State=63
Aggregate or Individual(True=1)= 1

In the above state, although I am configured as "normal" locally, I cannot ISSU:

 

RTP-AGG1# show lacp issu-impact 
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.

The following ports are not ISSU ready
 Eth4/8      , 

In short, if you are not seeing any LACP rate fast configuration locally, you must have devices attached that are forcing NX-OS to operate at the fast rate and thus makes the chassis ineligible for ISSU. You can either shut these ports while the ISSU completes or toggle the configuration to normal on the attached devices (assuming not strictly required.).

 

Let me know if you have any other questions!

- Andrea, CCIE #56739 R&S

Andrea

 

Thanks for the explanation. It has confirmed what I found as well. The port-channels not reported are connected to other Cisco switches while the 4 port-channels with the issue are connected to Check Point firewalls and a Windows server. I have found where to change the settings for these but will require downtime. 

 

At least I have a way forward now, so thanks for the help.

 

Thanks

Roy 

I have several 3Ks that didn’t show that the server was requesting fast rate lacp but when I made the change on the server side it immediately fixed the issue.

Hi Andrea,

 

Apologies for jumping in on this post but I am hoping you might be able to give a quick answer to a query I have relating to this thread and ISSU. I have a 5k which shows (when I run show install all impact) inconsistencies in the LACP timers. I have liaised with the server team and they are unable/unwilling to change the fast rate timers on the servers. IF, we go ahead without changing these timers and the install then becomes 'disruptive' will the disruption only affect these misconfigured servers or will it have an impact on all services connected to the 5k? 

Thanks in advance,

Clive

Hi Clive,

My apologies on just now seeing this – Unfortunately if there are any LACP ports listed under the “show lacp issu-impact” as not ISSU ready, when you try to upgrade, the upgrade process will force the chassis itself to upgrade disruptively. A “disruptive” upgrade means the chassis itself will fully reload which in essence would impact all ports and not just the “LACP fast” ones.

Hope that helps. Let me know otherwise. Happy to help.
- Andrea, CCIE #56739 R&S

Review Cisco Networking for a $25 gift card