Solved: Good to know. My coworker

Daniel.patterson · ‎06-17-2016

I am in the process of installing a new ACI instance in our Production DC. I am at the point of upgrading the OS on the Fabric. I wanted to see if there are any known issues with going from my current versions to the new versions, ie version step-ups, etc.

I also wanted to see about recommended versions for each. I was told 1.5(6a) or below for the CIMC FW but was also sent the following doc that states 2.0(3i) is the recommended version.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/recommended-release/b_Minimum_and_Recommended_Cisco_ACI_Releases.html

CURRENT VERSIONS

CIMC FW - 1.5(4e)
APIC FW - 1.1(0.766m)
FABRIC - n9000-11.1(0.180d)

PROPOSED VERSIONS

CIMC FW - 2.0(3i)
APIC FW - 1.2(3e)
FABRIC - n9000-11.2(3e)

Any experience with these proposed versions? Any recommendations or concerns?

Daniel

Tomas de Leon · ‎06-17-2016

Daniel,

the APIC FW & Switch firmware that you listed are not official released images. They look more like an EFT\QA image from the past. That said "policy" upgrades are not supported for these releases.

my suggestions:

upgrade the APICs CIMC FW and drivers to versions tied to the 2.0(3i) release. Use the Host Upgrade Utility(HUU) to upgrade "ALL" (just CIMC version) for this release. Once the CIMC drivers are upgraded and you can access the CIMC & the KVM successfully you can proceed.
Make notes on your exisiting configuration so that you can manually rebuild your configuration after the ACI firmware installs.
Download your TARGET versions for APIC and Nodes down from www.cisco.com. After downloading verify the images md5 checksums match with online.
SCP\SFTP or use USB the TARGET switch image (n9000-11.2(3e)) to EACH Leaf & Spine and copy to image to BOOTFLASH. Once this done you can upgrade the Node firmware when you are ready to proceed.
schedule a maintenance window so that you can wipe the fabric apics & nodes and proceed with upgrades and reconfiguration.
make sure you have console access locally and\or remotely via a terminal server to all console ports on each Leaf & Spine.
Review and follow online instructions for "Installing APIC Software Using Virtual Media"
Perform these instructions on EACH APIC and leave EACH APIC at the setup script
Power down APIC2 & APIC3 using the CIMC.
Restore the nodes (leaf & spines) to factory default and have them boot up on the new TARGET Versions.

From the switch CLI:

setup-clean-config.sh aci-n9000-dk9.11.2.3e.bin
setup-bootvars.sh aci-n9000-dk9.11.2.3e.bin
cat /mnt/cfg/0/boot/grub/menu.lst.local
cat /mnt/cfg/1/boot/grub/menu.lst.local
reload

Once the Leaf & Spines come up on the NEW TARGET image, manually upgrade the BIOS & FPGAs using the following commands:

/bin/check-fpga.sh FpGaDoWnGrAdE
/usr/sbin/chassis-power-cycle.sh

The switches will reload and once they are all up you can proceed with the setup script on APIC1.

APIC1 should steal be setting at the setup script. At this time, you can proceed with configuring APIC1. Make sure to record your values that you use so that you can configure APIC2 & APIC3 appropriately.
Once APIC1 is configured and online, proceed with the Fabric discovery of ALL the Leaf & Spines.
Once ALL the Leaf & Spines are configured, proceed to configure APIC2. Once configured and it has successfully joined the APIC cluster, repeat the process for APIC3
Once ALL APICs have joined the cluster and are fully fit, you can begin reconfiguring the ACI configuration.

I hope this helps!

T.

View solution in original post

Tomas de Leon · ‎06-24-2016

I am little confused.

I have the .iso on my workstation. Open KVM on CIMC and activate the .iso from my workstation.

No logins or information that you listed...

View solution in original post

Tomas de Leon · ‎06-17-2016

Daniel,

the APIC FW & Switch firmware that you listed are not official released images. They look more like an EFT\QA image from the past. That said "policy" upgrades are not supported for these releases.

my suggestions:

upgrade the APICs CIMC FW and drivers to versions tied to the 2.0(3i) release. Use the Host Upgrade Utility(HUU) to upgrade "ALL" (just CIMC version) for this release. Once the CIMC drivers are upgraded and you can access the CIMC & the KVM successfully you can proceed.
Make notes on your exisiting configuration so that you can manually rebuild your configuration after the ACI firmware installs.
Download your TARGET versions for APIC and Nodes down from www.cisco.com. After downloading verify the images md5 checksums match with online.
SCP\SFTP or use USB the TARGET switch image (n9000-11.2(3e)) to EACH Leaf & Spine and copy to image to BOOTFLASH. Once this done you can upgrade the Node firmware when you are ready to proceed.
schedule a maintenance window so that you can wipe the fabric apics & nodes and proceed with upgrades and reconfiguration.
make sure you have console access locally and\or remotely via a terminal server to all console ports on each Leaf & Spine.
Review and follow online instructions for "Installing APIC Software Using Virtual Media"
Perform these instructions on EACH APIC and leave EACH APIC at the setup script
Power down APIC2 & APIC3 using the CIMC.
Restore the nodes (leaf & spines) to factory default and have them boot up on the new TARGET Versions.

From the switch CLI:

setup-clean-config.sh aci-n9000-dk9.11.2.3e.bin
setup-bootvars.sh aci-n9000-dk9.11.2.3e.bin
cat /mnt/cfg/0/boot/grub/menu.lst.local
cat /mnt/cfg/1/boot/grub/menu.lst.local
reload

Once the Leaf & Spines come up on the NEW TARGET image, manually upgrade the BIOS & FPGAs using the following commands:

/bin/check-fpga.sh FpGaDoWnGrAdE
/usr/sbin/chassis-power-cycle.sh

The switches will reload and once they are all up you can proceed with the setup script on APIC1.

APIC1 should steal be setting at the setup script. At this time, you can proceed with configuring APIC1. Make sure to record your values that you use so that you can configure APIC2 & APIC3 appropriately.
Once APIC1 is configured and online, proceed with the Fabric discovery of ALL the Leaf & Spines.
Once ALL the Leaf & Spines are configured, proceed to configure APIC2. Once configured and it has successfully joined the APIC cluster, repeat the process for APIC3
Once ALL APICs have joined the cluster and are fully fit, you can begin reconfiguring the ACI configuration.

I hope this helps!

T.

Daniel.patterson · ‎06-21-2016

Tomas - Where can I find the CIMC Firmware? I have been all over the cisco site and cannot seem to locate it.

Daniel

Daniel.patterson · ‎06-21-2016

I think I found it.

Downloads Home

Products

Servers - Unified Computing

UCS C-Series Rack-Mount Standalone Server Software

UCS C220 M3 Rack Server Software

Tomas de Leon · ‎06-24-2016

Daniel,

Just wanted to close the loop. I know you were working with Robert from Cisco TAC on your issues. Are the issues resolved now?

Thanks

T.

Daniel.patterson · ‎06-24-2016

I am attempting to setup the vMedia but keep getting an error about the mount protocol.

Setup

Volume: 1

Mount Type: NFS

Remote Share: 10.49.80.181:/home/cisco

Remote File: aci-apic-dk9.1.2.3h.iso

Mount Options: nolock or ro or rw

User Name: (This is the User to access the file on the share correct?)

Error: Invalid mount option for protocol.

Tomas de Leon · ‎06-24-2016

I am little confused.

I have the .iso on my workstation. Open KVM on CIMC and activate the .iso from my workstation.

No logins or information that you listed...

Daniel.patterson · ‎06-28-2016

I was adding a step and attempting to mount the iso from the CIMC prior to launching the KVM.

I have since launched the KVM, mounted the ISO, and upgraded each of the APICs but booting the KVM Mapping.

All APICs are fully fit and running 1.2(3h) while the Spines and Leafs are running the 11.2(3h).

Now I get to start configuring them.

Thanks for the help!

Daniel

Ajay Mass · ‎10-15-2016

What do you do if the /usr/sbin/chassis-power-cycle.sh command fails on the leaf?

L121# /usr/sbin/chassis-power-cycle.sh
*** Running INXOS PE IFC image ***
/usr/sbin/chassis-power-cycle.sh: line 9: cctrl2_fpga_test.bin: command not found

Tomas de Leon · ‎10-15-2016

Remove the power cables from the switch for about 30 seconds and then plug in the power cables and it should server the same purpose.

I hope this helps

T.

Ajay Mass · ‎10-15-2016

I was afraid that was my only other option. I was hoping to make some progress remotely. I guess I will deal with this on Monday.

Tomas de Leon · ‎10-15-2016

For that particular switch, you could policy downgrade to the previous release and then after it reboots and rejoins the fabric you can policy upgrade back to the desired release and the FPGA & BIOS upgrades should occur and the power reset "should" be invoked after the reload.

What model of switch is this? and if this a modular switch, which "line card" or module has the FPGA\BIOS mismatch?

T.

Ajay Mass · ‎10-15-2016

Good to know. My coworker and I are building a multi pod environment and these switches are across the inter-pod network. I will bring this up to him as an option.

Switch model: N9K-C93180YC-EX

FPGA version mismatch detected. Running version: 0x3 Expected version: 0x4

BIOS version mismatch detected. Running version:v07.17(09/10/2014) Expected version:v07.41(10/12/2015)

This was after an upgrade using setup-clean-config.sh aci-n9000-dk9.12.1.1h.bin

APIC Upgrade Process