[SOLVED] N7k Supervisor does not become online

[JPavonM]

We are having some problems after replacing one of the supervisors in our Nexus 7000 chassis.

We have replaced the supervisor in module-5 for a new one because of a problem with the old one.
After that, the module has auto-downloaded the new 5.1.1 images from the supervisor in module-6 and in we have rebooted the module 5. The "show install al status" confirmed that and invite us to reboot manually the module-5 supervisor.
After booting up, the module stay in "powered-up" status but did not become online, neither we can see any information about it through cli in the active supervisor, nor serial number, nor model or software version, but yes, we can show its bootflash with the actual version of the images and with the old one version 4.0.4.
With the console cable in the module-5, we saw it was booting with the appropriate version 5.1.1 and everythis was going correct.
We have upgraded the EPLD image in module 5 also.
We also have introduced the command "no poweroff module 5" but the module-5 was up and running, but not online.

Any idea?

Kind regards

[aaraaz747]

Hi Jesus,

       Can you post the following command outputs.

1. n7k# sh module 5

2. n7k# sh module 6

3. Hope you have e.g Sup1 in module 5 as active & Sup1 in module 6 as ha-standbye ?  - check with Show module or the above 2 command outputs should do the same.

4. Or you may have to re-seat it...again this depends on which one is active or standby

Let me know

Regards

  A Raaz

P.S - not sure why your name comes up as *****.

[JPavonM]

Hi A Raaz,

Anybody see my nickname? it is J Pavon.

SUP in slot 6 is the active one, and the one in slot 5 is not online.

These are the outputs of the commands you told me:

#############################

NX# sh module 5
Ejector status could not be retrieved
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
5    0      Supervisor module-1X                                powered-up

Mod  Sw              Hw
---  --------------  ------

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------

Mod  Online Diag Status
---  ------------------

Chassis Ejector Support: Disabled

NX# sh module 6
Ejector status could not be retrieved
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
6    0      Supervisor module-1X             N7K-SUP1           active

Mod  Sw              Hw
---  --------------  ------
6    5.1(1)          1.6    

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
6    00-26-98-ww-vv-uu to 00-26-98-xx-zz-yy  JAF141abcde

Mod  Online Diag Status
---  ------------------
6    Pass

Chassis Ejector Support: Disabled

#############################

As you can see, there is no information about module 5 at all.

And this is a 'sh version' of the system:

#############################

Software
  BIOS:      version 3.22.0
  kickstart: version 5.1(1)
  system:    version 5.1(1)
  BIOS compile time:       02/20/10
  kickstart image file is: bootflash:///n7000-s1-kickstart.5.1.1.bin
  kickstart compile time:  12/25/2020 12:00:00 [10/24/2010 11:25:25]
  system image file is:    bootflash:///n7000-s1-dk9.5.1.1.bin
  system compile time:     9/2/2010 19:00:00 [10/24/2010 12:34:48]

Hardware
  cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")
  Intel(R) Xeon(R) CPU         with 4115848 kB of memory.
  Processor Board ID JAF1414AQHK

  Device name: NXDADMINE
  bootflash:    2000880 kB
  slot0:              0 kB (expansion flash)

Kernel uptime is 14 day(s), 0 hour(s), 54 minute(s), 9 second(s)

Last reset
  Reason: Unknown
  System version: 4.2(4)
  Service:

plugin
  Core Plugin, Ethernet Plugin

CMP (Module 6) ok
CMP Software
  CMP BIOS version:        02.01.05
  CMP Image version:       5.1(1) [build 5.0(0.66)]
  CMP BIOS compile time:   7/13/2008 19:44:27
  CMP Image compile time:  9/2/2010 19:00:00

CMP (Module 5) ok
CMP Software
  CMP BIOS version:        02.01.05
  CMP Image version:       4.0(4) [build 4.0(4)]
  CMP BIOS compile time:   7/13/2008 19:44:27
  CMP Image compile time:  10/20/2008 12:00:00
####################

It hasn't been possible to update the CMP of the module 5, but monitoring its console, it is booting from the same image than module 6, but after that, the login prompt appear but there is no output. However, in the console of module 6 appears a system message that it is concerned to NX-OS release 4.2, and it is solved:

SYSMGR-2-GSYNC_SNAPSHOT_SRVFAILED

and

%SYSMGR-2-STANDBY_BOOT_FAILED: Standby supervisor failed to boot up.

Thank you again

J Pavon

[aaraaz747]

Hi J Pavon,

      Ok, looks like something hasn't initialized or the upgrade process  may have got corrupted.

      1. Try re-seating the SUP1 in module slot 5  (this should not impact any live services as SUP1 in slot 6 is active)

          this may re-initiate the firmware upgrade, but do make sure you do it when convenient.

      2. If the above doesn't work, try re-flashing SUP1 in slot5 manually through console.

      3. Output of command n7k# sh mod

Standby Supervisor Fails to Boot

The standby supervisor does not boot after an upgrade. You may see the following system message:

Error Message     SYSMGR-2-STANDBY_BOOT_FAILED: Standby supervisor failed to boot up.

Explanation    This message is printed if the standby  supervisor doesn't complete its boot procedure (i.e. it doesn't reach  the login prompt on the local console) 3 to 6 minutes after the loader  has been loaded by the BIOS. This message is usually caused by boot  variables not properly set for the standby supervisor. This message can  also be caused by a user intentionally interrupting the boot procedure  at the loader prompt (by means of pressing ESC).

Recommended Action    Connect to the local console of the standby supervisor. If the supervisor is at the loader prompt, try to use the bootreload command for the standby supervisor from a vsh session on the active supervisor, specifying the force-dnld option. Once the standby is online, fix the problem by setting the boot variables appropriately. command to continue the boot procedure. Otherwise, issue a

Symptom	Possible Cause	Solution
Standby supervisor does not boot.	Active supervisor kickstart image booted from TFTP.	Reload the active supervisor from bootflash:.

      keep posted.

Regards

  A. Raaz

[JPavonM]

Thank you very much for your fast update A Raaz,

The problem with the faulty supervisor has been solved. The only action I take was to reload the slot 6 active supervisor, suddenly the slot 5 supervisor became online and active. Then I updated the CMP version from the slot 5 supervisor (as you can see in my last post it was different in both SUPs). It seems there were some kind of error during an upgrade from NX-OS 4.1 to 5.1 one mate did it some weeks ago. For some reason the slot 5 supervisor did not uptgrade the EPLD, CMP, and the NX-OS image versions in the bootflash, so the supervisor became offline. We have had to upgrade everything individually to this supervisor, EPLD and NX images, but the key step was to reload the active supervisor.

Kind regards.

[aaraaz747]

Hi J Pavon,

        That is great news. Good work.

Regards

  Abdul