1000v VSM upgrade wont even start

[Matt Gouveia]

The necessary BIN files for the kickstart and system Nexus IOS are in the bootflash, but when I run the 'install all' command it comes back with an error message. 

The error message is:

'Unable to send request to installer, give up.  Return code -1."  

This is an upgrade from nexus-1000v-mz.4.2.1.SV1.4.bin to nexus-1000v-mz.4.2.1.SV1.4a.bin, both kickstart and system.

Could not find anything on Cisco.com nor on the internet.  Has anyone seen this before?

[Matt Gouveia]

Here is the message in the system log:

%SYSMGR-3-SERVICE_TERMINATED: Service "installer" (PID 28869) has finished with error  code SYSMGR_EXITCODE_SYSERR (1).

[mipetrin]

Hi Matt,

Just a few things that I would like clarified:

1) How many VSMs do you have?

2) If 2, what is the status of the H/A pair? #show system redundancy status

3) On which VSM did you execute the 'install all' command?

4) Please verify the amount of free space available on the VSM. '#dir' should be enough

5) Please provide the output from

   - #show logging

   - #show install all failure-reason

   - #show install all failed-standby

Thanks,

Michael

[Matt Gouveia]

Hi Michael,

Thanks for looking into this. 

Here you go:

1. We have two VSMs

2. Show system redundancy status

-----------------------------------------------------

C-1000-PD-4-A# show system redundancy status

Redundancy role

---------------

      administrative:   primary

         operational:   primary

Redundancy mode

---------------

      administrative:   HA

         operational:   HA

This supervisor (sup-1)

-----------------------

    Redundancy state:   Active

    Supervisor state:   Active

      Internal state:   Active with HA standby

Other supervisor (sup-2)

------------------------

    Redundancy state:   Standby

    Supervisor state:   HA standby

      Internal state:   HA standby

3. Issued the 'install all' command on the primary/active VSM

4. 1,194,336,256 bytes free after placing the kickstart and system bin files onto the bootflash: directory

5.

show logging:

SYSMGR-3-SERVICE_TERMINATED: Service "installer" (PID 28194) has finished with error code SYSMGR_EXITCODE_SYSERR (1).

show install all failure-reason

"No install all failure-reason"

show install all failed-standby

[NO TEXT RETURNED AT ALL]

Thanks again Michael

[mipetrin]

Hi Matt,

Thanks for getting back to me. Just check the following additional things for me:

* There is enough space on both the active and stanby VSMs. I realize you've already check the active, but want the output from the same time

# dir bootflash://sup-active/

# dir bootflash://sup-standby/

* Can you ensure the filesystems are writable on the active/standby VSMs and not in a read-only state. You may also see logs indicating read only state

# show version > bootflash:primary.txt

# show version > bootflash://sup-standby/secondary.txt

* Lastly, the output from:

# show install all status

Thanks,

Michael

[Matt Gouveia]

Worked with my vendor and found the solution.  Ends up some how the two switches in the pair got out of synch in some granular way: nothing indicated there was a problem!  When we rebooted the Active switch the secondary one kept telling the the primary to reboot: it would go down, reboot, get to the login screen then reboot; over and over and over.

We took the primary out of the HA pair, reducing it to the Secondary being the only switch.  Rebooted the Primary.  Once it was up and stable we put it back into the pair and did the same with the Secondary, pulling it out of the HA pair and rebooting it.  <>  Bottom line we swapped the roles around a few times, rebooted them and in the process we brought them back into 100% synch.

Once they were in synch we did not get the error when doing the upgrade.  Followed the process Cisco PDF'ed and it worked like charm.

This was a strange occurrence even Cisco had not seen before.  Getting each VSM of the HA pair to reboot and re-synch solved it.