Module shows PwrDown on 6800-X-LE

[ganesh.g1982]

After upgrading the rommon and ios on a part of VSS (2x6800-X-LE), one of the module in a 6800-X-LE is showing as PwrDwn in the module status. 

sh module switch 1
=========
Switch Number: 1 Role: Virtual Switch Standby
---------------------- -----------------------------
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 20 DCEF-X-LE 16P SFP+ Multi-Rate C6880-X-LE-16P10G xxxxxxxxxxx
5 20 6880-X-LE 16P SFP+ Multi-Rate (Hot) C6880-X-LE-SUP xxxxxxxxxxx

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 xxxx.xxxx.xxxx to xxxx.xxxx.xxxx 2.2 Unknown Unknown PwrDown

show logging:
=========
xxx xx xx:xx:xx.xxx : %OIR-SW1_STBY-3-VSS_PWRCYCLE: Card in Switch 1 module 1, is being power-cycled 'off (Reset - Module Reloaded During Download)'
xxx xx xx:xx:xx.xxx : %SM-SW1_STBY-4-BADEVENT: Event 'dnld_completed' is invalid for the current state 'disabled': scp_dnld_module 1 : 0 : 0

There is no problem faced on the other 6800-X-LE in the VSS pair during its rommon and IOS upgrade. Both module 1 and 2 (sup) are online, which is currently active in VSS.

The logging message shows that " Card in Switch 1 module 1, is being power-cycled 'off (Reset - Module Reloaded During Download)'", does it indicate that:

 -- The module restarted during the rommon upgrade? (This did not happen for sure) or

 -- The module physically reloaded during rommon upgrade? (The module was reseated when it is found in pwrdwn status after the upgrades)

Both the above actions did not happen during the rommon and IOS upgrade. Rommon upgrade was completed and then reloaded the chassis. Then the IOS upgrade was performed.

As part of troubleshooting:
-- Module was reset, power enable/disable, physical reseat etc done. But the module is still in the PwrDwn status. 

Looking for feedback if anyone had faced this issue earlier or any suggestions for troubleshooting.

[Leo Laohoo]

Console into the switch and reboot.

Post the entire boot-up process.

Did you upgrade to 15.5(1)SY4 (and later)?

[ganesh.g1982]

Yes, it is upgraded to 15.5(1).SY10. 

The entire boot process is very clean without any errors. When the switch comes online, and checks the show module switch 1, one of the module as shown in the first post, is PwdDown. 

[Leo Laohoo]

---

@ganesh.g1982 wrote:
The entire boot process is very clean without any errors.

---

Yes, it does and it will show up in console and during the bootup.

Starting from 15.5(1)SY4, there is a deliberately hidden information about performing a manual FPGA Upgrade.  This information is NOT FOUND in the 15.5(1)SY Release Notes.

The line card in question is in power down because the FPGA needs to be manually upgraded. 

Performing FPGA Upgrade

One final thing, the last option of the command, gold, is a hidden command.  This mean the entire word/command needs to be typed out.

[ganesh.g1982]

Thanks. 

The following procedure was exactly followed for 6800. It was not ISSU upgrade. 

Release Notes for Catalyst 6500, Supervisor Engine 2T, Supervisor Engine 6T, Catalyst 6880, Catalyst 6840, and Catalyst 6800 Series Switches ROMMON - Cisco

The steps followed:

upgrade rom-monitor switch 1 slot 5 file bootdisk:c6880x_rm.bin.SPA.151-02r.SYS4

Note: SUP is on slot 5 and Linecard on Slot1

After the entire process of the above is completed( like Erasing flash , Programming flash,Verifying magic in flash), the message received :

ROMMON image upgrade complete, Supervisor engine must be reloaded.

Then boot commands are updated to load 15.5.1.SY10 as well. The switch was Active at that moment, and the standby switch was already upgraded to new rommon and IOS as above and was in RPR mode. Then the redundancy force-switchover was initiated to switch in the active status to switch over to other switch and get this switch reloaded with the new rommon and IOS. 

However, when the switch came back online with one module (PwrDwn).

sh module switch 1
Switch Number: 1 Role: Virtual Switch Standby
---------------------- -----------------------------
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 20 DCEF-X-LE 16P SFP+ Multi-Rate C6880-X-LE-16P10G xxxxxxxxxxx
5 20 6880-X-LE 16P SFP+ Multi-Rate (Active) C6880-X-LE-SUP xxxxxxxxxxx

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 xxxx.xxxx.xxxx to xxxx.xxxx.xxxx 2.2 Unknown Unknown PwrDown
5 xxxx.xxxx.xxxx to xxxx.xxxx.xxxx 1.1 15.1(02r)SYS 15.5(1)SY10 Ok

show logging:
=========
xxx xx xx:xx:xx.xxx : %OIR-SW1_STBY-3-VSS_PWRCYCLE: Card in Switch 1 module 1, is being power-cycled 'off (Reset - Module Reloaded During Download)'
xxx xx xx:xx:xx.xxx : %SM-SW1_STBY-4-BADEVENT: Event 'dnld_completed' is invalid for the current state 'disabled': scp_dnld_module 1 : 0 : 0

On the peer switch(which is currently Active), there is no issue. The below the status:

sh module switch 2
Switch Number: 2 Role: Virtual Switch Active
---------------------- -----------------------------
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 20 DCEF-X-LE 16P SFP+ Multi-Rate C6880-X-LE-16P10G xxxxxxxxxxx
5 20 6880-X-LE 16P SFP+ Multi-Rate (Active) C6880-X-LE-SUP xxxxxxxxxxx

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 xxxx.xxxx.xxxx to xxxx.xxxx.xxxx 2.2 15.2(02r)SYL 15.5(1)SY10 Ok
5 xxxx.xxxx.xxxx to xxxx.xxxx.xxxx 1.0 15.1(02r)SYS 15.5(1)SY10 Ok

Same upgrade was done in another pair of VSS a few months ago with the same ROMMON image and IOS 15.5.1.SY8. There was no issue faced at that time for the pair.

[ganesh.g1982]

Regarding the FPGA upgrade for this particular model of switch and modules, the following info is found in the cisco. The modules used in the switches are not in the list, so I believe it is not applicable in the case as none of the linecard used are in the list given.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/15-5SY/config_guide/sup6T/15_5_sy_swcg_6T/enhanced_fast_software_upgrade_efsu.html

***** Secure FPGA is available only on C6816-X-LE, C6832-X-LE, C6824-X-LE-40G, C6840-X-LE-40G, Sup6T, C6800-32P10G, C6800-16P10G, C6800-8P10G, and C6800-8P40G-XL switching modules.

[Leo Laohoo]

Console into the switch and reboot. 

If the line card does not need an FPGA Upgrade, the following message SHOULD NOT APPEAR in the console during bootup: 

%C6K_PLATFORM-SW2-2-CRITICAL_FPGA_UPGRADE:
**********************************************************************
********************************************************************** 
************************ !! UPDATE REQUIRED !! ************************
 
Critical firmware upgrade available for Switch: 1 Module: 1.
Please refer Software release documentation for more information.
 
**** ******************************************************************

[ganesh.g1982]

Rebooted the switch using redundancy reload peer and checked the logs on console. The above message was not there. 

redundancy reload peer

Reload peer [confirm]

Preparing to reload peer

System Bootstrap, Version 15.1(02r)SYS4, RELEASE SOFTWARE
.
.
.
Current image running: Governator Boot Rom :Upgradable Region

Last reset cause: Software Reset, reset code 0x4

Governator platform with 2097152 Kbytes of main memory

#Booting c6880x-ipservicesk9-mz.SPA.155-1.SY10.bin ...

########################################
########################################
########################################

ELF image verification PASSED

Self extracting the image... [OK]

Self decompressing the image : #########################
########################################
############## [OK]

Cisco IOS Software, c6880x Software (c6880x-IPSERVICESK9-M), Version 15.5(1)SY10, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
.
.
.
.
.
.

*xxx xx:xx:xx.xxx: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor for this switch

*xxx xx:xx:xx.xxx: %C6KPWR-4-PSREDUNDANTBOTHSUPPLY: in power-redundancy mode, system is operating on both power supplies.
*xxx xx:xx:xx.xxx: %VSL_BRINGUP-6-MODULE_UP: VSL module in slot 5 switch 1 brought up
*xxx xx:xx:xx.xxx: %VSLP-5-VSLP_LMP_PASS_REASON: interface 5/1 : Link up
*xxx xx:xx:xx.xxx: %VSLP-5-VSLP_LMP_PASS_REASON: interface 5/2 : Link up
*xxx xx:xx:xx.xxx: %VSLP-5-RRP_ROLE_RESOLVED: Role resolved as STANDBY by VSLP
*xxx xx:xx:xx.xxx: %VSL-5-VSL_CNTRL_LINK: New VSL Control Link 1/5/1
Initializing as Virtual Switch STANDBY processor

*xxx xx:xx:xx.xxx: %VSLP-5-VSL_UP: Ready for control traffic

*xxx xx:xx:xx.xxx: %PFREDUN-SW1_STBY-6-STANDBY: Initializing for SSO mode in Default Domain

[ganesh.g1982]

Some more logs after reseating the module and reset and power disable/enable:

*xxx xx:xx:xx.xxx: %OIR-SW2-6-INSREM: Switch 1 Physical Slot 1 - Module Type LINE_CARD removed
*xxx xx:xx:xx.xxx: %OIR-SW2-6-INSREM: Switch 1 Physical Slot 1 - Module Type LINE_CARD inserted
*xxx xx:xx:xx.xxx: %OIR-SW1_STBY-3-VSS_PWRCYCLE: Card in Switch 1 module 1, is being power-cycled 'off (Reset - Module Reloaded During Download)'
*xxx xx:xx:xx.xxx: %SM-SW1_STBY-4-BADEVENT: Event 'dnld_completed' is invalid for the current state 'disabled': scp_dnld_module 1 : 0 : 0
*xxx xx:xx:xx.xxx: %PARSER-SW2-5-CFGLOG_LOGGEDCMD: User:network logged command:power enable switch 1 module 1
*xxx xx:xx:xx.xxx: %SM-SW1_STBY-4-BADEVENT: Event 'operator_power_on' is invalid for the current state 'occupied': cygnus_oir_bay slot1
*xxx xx:xx:xx.xxx: %OIR-SW2-6-INSREM: Switch 1 Physical Slot 1 - Module Type LINE_CARD removed
*xxx xx:xx:xx.xxx: %OIR-SW2-6-INSREM: Switch 1 Physical Slot 1 - Module Type LINE_CARD inserted
*xxx xx:xx:xx.xxx: %OIR-SW1_STBY-3-VSS_PWRCYCLE: Card in Switch 1 module 1, is being power-cycled 'off (Reset - Module Reloaded During Download)'
*xxx xx:xx:xx.xxx: %OIR-SW1_STBY-4-RESTART: The card in slot 1 has reached the maximum number of 5 restarts, this card will be disabled
*xxx xx:xx:xx.xxx: %SM-SW1_STBY-4-BADEVENT: Event 'dnld_completed' is invalid for the current state 'disabled': scp_dnld_module 1 : 0 : 0

[Leo Laohoo · Accepted solution]

---

@ganesh.g1982 wrote:
*xxx xx:xx:xx.xxx: %OIR-SW1_STBY-4-RESTART: The card in slot 1 has reached the maximum number of 5 restarts, this card will be disabled

*xxx xx:xx:xx.xxx: %SM-SW1_STBY-4-BADEVENT: Event 'dnld_completed' is invalid for the current state 'disabled': scp_dnld_module 1 : 0 : 0

---

Line card is dead.

[ganesh.g1982]

OK, so the next option is RMA.

When the new module is inserted, will it download automatically upgrade the rommon, if the version on the new is lower than the one in system currently? or should the upgrade rommon required again?

[Leo Laohoo]

When a new line card is inserted, check the logs.  Usually, it will do all updates automatically. 

Except the "new feature" of manually upgrading the FPGA.

[ganesh.g1982]

The line card is replaced with a spare, everything back to normal. 

Thanks very much for all your feedback. 

[Leo Laohoo]

Thanks for providing the update.