04-09-2019 01:01 AM
Hi folks
I have a Nexus 7700 6-slot switch currently on 7.2(1)D1(1). We are are looking to upgrade eventually to nxos 8 but reading through the upgrade guides, we need to upgrade first to 7.3(3)D1(1). When I do the upgrade it stops and says that 2 modules are incompatible. When I run "show install impact" I get the output in the attached screenshot. The other screenshot shows the modules installed.
As far as I can see in the upgrade guide at https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/7_x/nx-os/upgrade/guide/b_Cisco_Nexus_7000_Series_NX-OS_Software_Upgrade_and_Downgrade_Guide_Release_7-x.html and the release notes at https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/7_x/nx-os/release/notes/7x_nx-os_release_note.html#pgfId-1362862 I do not see any incompatibilities. The upgrade guide says that ISSU is supported for 7.2(1)D1(1) to 7.3(3)D1(1) upgrade.
What does this mean? Can I actually do an upgrade using ISSU? Anyone seen this or have any suggestions?
Many Thanks
Roy
Solved! Go to Solution.
04-11-2019 05:39 AM
Thank you for the outputs!
The reason the upgrade showed "image incompatible" the first time is because the kickstart and system images did not match in the "install all" command -- Sample from the outputs shared:
# Install all kickstart n7700-s2-kickstart.7.2.1.D1.1.bin system n7700-s2-dk9.7.3.3.D1.1.bin
Installer will perform compatibility check first. Please wait.
Now, with the new outputs and the proper install all command, we can see that the upgrade is compatible but still cannot be of the ISSU type due to LACP constraints (it seems you may have lacp rate fast enabled which is not supported when doing ISSUs):
Additional info for this installation:
--------------------------------------
Service "lacp" in vdc 1: LACP: Upgrade will be disruptive as 8 switch ports and 0 fex ports are not upgrade ready!!
<snip>
show lacp issu-impact
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.
The following ports are not ISSU ready
Eth5/15 , Eth6/15 , Eth5/17 , Eth6/17 , Eth5/7 ,
Eth6/7 , Eth5/8 , Eth6/8 ,
I'd check the configuration of the ports listed above and make sure they are not configured for LACP rate fast. Once that is corrected, you should be good to go.
Hope that helps!
04-09-2019 09:22 AM
Looks like from 7.2(1)D1(1) you cannot go to 7.3(3)D1(1) with ISSU you may have to go with traditional reload upgrade
Table 11. Supported ISSU Paths for the Cisco Nexus 7000 Series Platform (Cisco NX-OS Release 7.2(1)D1(1))Current ReleaseRelease TrainNon-disruptive Upgrade
Cisco NX-OS Release 7.2(1)D1(1) | 7.2 | 7.2(0)D1(1) 6.2(20a) 6.2(20) 6.2(18) 6.2(16) 6.2(14) 6.2(12) 6.2(10) |
04-09-2019 10:06 AM
Hi,
Can you share the complete output to the following commands:
show install all impact kickstart bootflash:n7700-s2-kickstart.7.3.3.D1.1.bin system bootflash:n7700-s2-dk9.7.3.3.D1.1.bin dir | i i 7.3.3 show file bootflash:n7700-s2-kickstart.7.3.3.D1.1.bin md5sum show file bootflash:n7700-s2-dk9.7.3.3.D1.1.bin md5sum
Additionally, per release notes, 7.2(1)D1(1) to 7.3(3)D1(1) is definitely a valid ISSU step as you mentioned:
Thanks,
04-11-2019 12:17 AM
Andrea
Thanks for confirming that the upgrade should be valid. Below is the output from the commands you suggest.
#dir | i 7.3.3
577735934 Apr 08 10:39:58 2019 n7700-s2-dk9.7.3.3.D1.1.bin
47450112 Apr 08 10:22:10 2019 n7700-s2-kickstart.7.3.3.D1.1.bin
#show file bootflash:n7700-s2-kickstart.7.3.3.D1.1.bin md5sum
8cf5911f0f07c91e087a75d13b2146c1
#show file bootflash:n7700-s2-dk9.7.3.3.D1.1.bin md5sum
f90c20c1e27e7be052d332f5909c5f6f
#show install all impact kickstart bootflash:n7700-s2-kickstart.7.3.3.D1.1.bin system bootflash:n7700-s2-dk9.7.3.3.D1.1.bin
Installer will perform impact only check. Please wait.
Verifying image bootflash:/n7700-s2-kickstart.7.3.3.D1.1.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS
Verifying image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin for boot variable "system".
[####################] 100% -- SUCCESS
Performing module support checks. [####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Extracting "lcflnn7k" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Extracting "bios" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Extracting "system" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Extracting "kickstart" version from image bootflash:/n7700-s2-kickstart.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Extracting "lc1n7k" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Notifying services about system upgrade. [####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive rolling
2 yes non-disruptive rolling
3 yes non-disruptive reset
4 yes non-disruptive reset
5 yes non-disruptive rolling
6 yes non-disruptive rolling
Images will be upgraded according to following table:
Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- -------------------- ------------
1 lcflnn7k 7.2(1)D1(1) 7.3(3)D1(1) yes
1 bios v3.0.24(11/10/2013):v3.0.24(11/10/2013) v3.0.29(12/15/2015) yes
2 lcflnn7k 7.2(1)D1(1) 7.3(3)D1(1) yes
2 bios v3.0.24(11/10/2013):v3.0.24(11/10/2013) v3.0.29(12/15/2015) yes
3 system 7.2(1)D1(1) 7.3(3)D1(1) yes
3 kickstart 7.2(1)D1(1) 7.3(3)D1(1) yes
3 bios v3.1.0(02/27/2013):v3.1.0(02/27/2013) v3.1.0(02/27/2013) no
4 system 7.2(1)D1(1) 7.3(3)D1(1) yes
4 kickstart 7.2(1)D1(1) 7.3(3)D1(1) yes
4 bios v3.1.0(02/27/2013):v3.1.0(02/27/2013) v3.1.0(02/27/2013) no
5 lc1n7k 7.2(1)D1(1) 7.3(3)D1(1) yes
5 bios v2.0.18(01/12/13):v2.0.18(01/12/13) v2.0.32(12/16/13) yes
6 lc1n7k 7.2(1)D1(1) 7.3(3)D1(1) yes
6 bios v2.0.18(01/12/13):v2.0.18(01/12/13) v2.0.32(12/16/13) yes
Additional info for this installation:
--------------------------------------
Service "lacp" in vdc 1: LACP: Upgrade will be disruptive as 8 switch ports and 0 fex ports are not upgrade ready!!
Issue the "show lacp issu-impact" cli for more details.
I have included the output for "show lacp issu-impact" as well. i have assumed that with dual supervisors that ISSU would restart them 1 at a time, thereby limiting any disruption. But I have seen other posts saying that both supervisors reboot at the same time.
show lacp issu-impact
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.
The following ports are not ISSU ready
Eth5/15 , Eth6/15 , Eth5/17 , Eth6/17 , Eth5/7 ,
Eth6/7 , Eth5/8 , Eth6/8 ,
Also, for the completeness, this is the full output when I attempted the install.
# Install all kickstart n7700-s2-kickstart.7.2.1.D1.1.bin system n7700-s2-dk9.7.3.3.D1.1.bin
Installer will perform compatibility check first. Please wait.
Verifying image bootflash:/n7700-s2-kickstart.7.2.1.D1.1.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS
Verifying image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin for boot variable "system".
[####################] 100% -- SUCCESS
Performing module support checks. [####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS
Extracting "lcflnn7k" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Extracting "bios" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Extracting "system" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Extracting "kickstart" version from image bootflash:/n7700-s2-kickstart.7.2.1.D1.1.bin.
[####################] 100% -- SUCCESS
Extracting "lc1n7k" version from image bootflash:/n7700-s2-dk9.7.3.3.D1.1.bin.
[####################] 100% -- SUCCESS
Notifying services about system upgrade. [####################] 100% -- SUCCESS
Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive rolling
2 yes non-disruptive rolling
3 no n/a n/a Incompatible image
4 no n/a n/a Incompatible image
5 yes non-disruptive rolling
6 yes non-disruptive rolling
Not all components in the image are compatible.
ISSU can't continue
04-11-2019 05:39 AM
Thank you for the outputs!
The reason the upgrade showed "image incompatible" the first time is because the kickstart and system images did not match in the "install all" command -- Sample from the outputs shared:
# Install all kickstart n7700-s2-kickstart.7.2.1.D1.1.bin system n7700-s2-dk9.7.3.3.D1.1.bin
Installer will perform compatibility check first. Please wait.
Now, with the new outputs and the proper install all command, we can see that the upgrade is compatible but still cannot be of the ISSU type due to LACP constraints (it seems you may have lacp rate fast enabled which is not supported when doing ISSUs):
Additional info for this installation:
--------------------------------------
Service "lacp" in vdc 1: LACP: Upgrade will be disruptive as 8 switch ports and 0 fex ports are not upgrade ready!!
<snip>
show lacp issu-impact
For ISSU to Proceed, Check the following:
1. All port-channel member port should be in a steady state.
2. LACP rate fast should not be enabled on member ports.
The following ports are not ISSU ready
Eth5/15 , Eth6/15 , Eth5/17 , Eth6/17 , Eth5/7 ,
Eth6/7 , Eth5/8 , Eth6/8 ,
I'd check the configuration of the ports listed above and make sure they are not configured for LACP rate fast. Once that is corrected, you should be good to go.
Hope that helps!
04-11-2019 07:50 AM
Ok, thanks for pointing that out. Something so obvious, no matter how many times I looked at the command. I really should be more careful to check I copy and paste the right things
Regarding the LACP ports, I am a bit confused with this message. Is "lacp rate fast" the default setting? I do not see it in the config but I am more confused. The ports in question belong to 4 portchannels, each with the command "channel-group mode active". There are no specific LACP commands specified. Also, 2 of the portchannels are trunk ports, 1 is a routed interface and the other is an access port.
There are also another 5 portchannels, 3 of which are LACP, but the interfaces do not appear in the issu-impact. What is different about these ports. Sample configs below:
Example of portchannel reported by issu-impact:
interface port-channel13
switchport
switchport mode trunk
switchport trunk allowed vlan 851-859,888
interface Ethernet5/7
switchport
switchport mode trunk
switchport trunk allowed vlan 851-859,888
channel-group 13 mode active
no shutdown
interface Ethernet6/7
switchport
switchport mode trunk
switchport trunk allowed vlan 851-859,888
channel-group 13 mode active
no shutdown
Example of portchannel not reported by issu-impact
interface port-channel3
mtu 9000
ip address ***********
ip ospf message-digest-key 1 md5 3 ************
no ip ospf passive-interface
interface Ethernet5/48
mtu 9000
channel-group 3 mode active
no shutdown
interface Ethernet6/48
mtu 9000
channel-group 3 mode active
no shutdown
I know I can go through these and set "no lacp rate fast" but I'm concerned that only some are reported as incompatible and others are not. Why? And I assume if I change the setting on the 7700 side, I need to change it on the other side of the portchannel as well?
Thanks
Roy
04-11-2019 09:13 AM
Roy,
Anytime! Happy to help. I'll try my best to concisely explain the LACP rate fast dilemma here.
For instance, you may have a server that requires LACP rate fast, in which case you wouldn't need it configured on the actual NX-OS device since we honor the request from the server (or any given attached device.) Conversely, if you wanted the Nexus to force an attached device to run with LACP rate fast, you would then configure that on the NX-OS device itself.
When a server (or attached device) requires NX-OS to use rate fast (or when the Nexus requires it), ISSU is not possible because the timeout is far too short to allow the control-plane to behave as expected during an ISSU operation.
You can verify the timers that an interface is using by entering the following CLI:
Example from my box:
RTP-AGG1# show lacp interface ethernet4/8 Interface Ethernet4/8 is up Channel group is 20 port channel is Po20 PDUs sent: 134665 PDUs rcvd: 4501 <snip> Local Port: Eth4/8 MAC Address= 0-23-4-ee-c1-e8 System Identifier=0x7f9b, Port Identifier=0x8000,0x408 Operational key=32788 LACP_Activity=active LACP_Timeout=Long Timeout (30s) << I am configured as "normal" Synchronization=IN_SYNC Collecting=true Distributing=true Partner information refresh timeout=Short Timeout (3s) Actor Admin State=61 Actor Oper State=61 Neighbor: 0x308 MAC Address= 84-78-ac-b-60-43 System Identifier=0x8000, Port Identifier=0x8000,0x308 Operational key=19 LACP_Activity=active LACP_Timeout=short Timeout (1s) << Attached device is requesting "lacp rate fast" Synchronization=IN_SYNC Collecting=true Distributing=true Partner Admin State=63 Partner Oper State=63 Aggregate or Individual(True=1)= 1
In the above state, although I am configured as "normal" locally, I cannot ISSU:
RTP-AGG1# show lacp issu-impact For ISSU to Proceed, Check the following: 1. All port-channel member port should be in a steady state. 2. LACP rate fast should not be enabled on member ports. The following ports are not ISSU ready Eth4/8 ,
In short, if you are not seeing any LACP rate fast configuration locally, you must have devices attached that are forcing NX-OS to operate at the fast rate and thus makes the chassis ineligible for ISSU. You can either shut these ports while the ISSU completes or toggle the configuration to normal on the attached devices (assuming not strictly required.).
Let me know if you have any other questions!
04-11-2019 01:37 PM
Andrea
Thanks for the explanation. It has confirmed what I found as well. The port-channels not reported are connected to other Cisco switches while the 4 port-channels with the issue are connected to Check Point firewalls and a Windows server. I have found where to change the settings for these but will require downtime.
At least I have a way forward now, so thanks for the help.
Thanks
Roy
04-11-2022 04:02 PM
I have several 3Ks that didn’t show that the server was requesting fast rate lacp but when I made the change on the server side it immediately fixed the issue.
11-05-2020 06:01 AM
Hi Andrea,
Apologies for jumping in on this post but I am hoping you might be able to give a quick answer to a query I have relating to this thread and ISSU. I have a 5k which shows (when I run show install all impact) inconsistencies in the LACP timers. I have liaised with the server team and they are unable/unwilling to change the fast rate timers on the servers. IF, we go ahead without changing these timers and the install then becomes 'disruptive' will the disruption only affect these misconfigured servers or will it have an impact on all services connected to the 5k?
Thanks in advance,
Clive
11-30-2020 06:04 AM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide