Cisco WLC 9800-40 Stuck in Boot

zamis.ali · ‎07-29-2020

Hi everyone,

Need urgent help on this please. Had issues booting up the 9800-40 WLC, so entered the rommon mode using the Break sequence and changed the config register using :

rommon > confreg 0x2142

and then reset.

The WLC is still having issues with booting up so I thought of trying to boot it via the USB in rommon mode. Seeing I changed the config register to 2142, it ignores the break sequence now and I cannot find a way to get back to the rommon mode. Can someone tell me how to sort this out please?

Regards

Leo Laohoo · ‎07-29-2020

Console into the switch and reboot.
Post the entire boot-up process.

zamis.ali · ‎07-29-2020

Hi Leo,

Thanks for the reply.

The booting process goes on in a loop and pretty much gives me the highlighted error and then back to where it started. The output is as follows:

Jul 28 02:01:14.116: %PMAN-3-PROCHOLDDOWN: R0/0: The process rif_mgr has been helddown (rc 139)
Jul 28 02:01:14.840: %PMAN-3-PROC_EMPTY_EXEC_FILE: R0/0: pvp: Empty executable used for process bt_logger
Jul 28 02:01:16.255: %PMAN-3-PROC_EMPTY_EXEC_FILE: R0/0: pvp: Empty executable used for process bt_logger
Jul 28 02:01:16.450: %PMAN-0-PROCFAILCRIT: R0/0: pvp: A critical process rif_mgr has failed (rc 139)
Jul 28 02:01:16.508: %PMAN-3-RELOAD_RP: R0/0: pvp: Reloading: Chassis will be reloaded

Chassis 1 reloading, reason - Non participant detected
Jul 28 02:02:18.759: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: process exit with reload fru code
Jul 28 02:02:20.851: %PMAN-3-PROCESS_NOTIFICATION: R0/0: pvp: System report core/RP_0-system-report_20200728-020220- Universal.tar.gz (size: 720 KB) generated and System report info at core/RP_0-system-report_20200728-020220-Universa l-info.txt

Initializing Hardware ...

System integrity status: 90170200 12030107

Current image running: Boot ROM0
Last reset cause: LocalSoft

C9800-40-K9 platform with 33554432 Kbytes of main memory

File size is 0x000015dd
Located packages.conf
Image size 5597 inode num 22, bks cnt 2 blk size 8*512
#
File size is 0x01f4acd5
Located C9800-rpboot.17.02.01a.SPA.pkg
Image size 32812245 inode num 712708, bks cnt 8011 blk size 8*512
#################################################################################################################### #################################################################################################################### ########################################################################################
Boot image size = 32812245 (0x1f4acd5) bytes

ROM:RSA Self Test Passed
ROM:Sha512 Self Test Passed

Package header rev 3 structure detected
Calculating SHA-1 hash...done
validate_package_cs: SHA-1 hash:
calculated f0b71f2c:ab639e9a:6f6229b8:d34a52b4:efc6a86d
expected f0b71f2c:ab639e9a:6f6229b8:d34a52b4:efc6a86d
Validating main package signatures

RSA Signed RELEASE Image Signature Verification Successful.
Image validated
Jul 28 02:04:38.180: %PMAN-3-PROCHOLDDOWN: R0/0: The process rif_mgr has been helddown (rc 139)
Jul 28 02:04:38.921: %PMAN-3-PROC_EMPTY_EXEC_FILE: R0/0: pvp: Empty executable used for process bt_logger
Jul 28 02:04:40.333: %PMAN-3-PROC_EMPTY_EXEC_FILE: R0/0: pvp: Empty executable used for process bt_logger
Jul 28 02:04:40.539: %PMAN-0-PROCFAILCRIT: R0/0: pvp: A critical process rif_mgr has failed (rc 139)
Jul 28 02:04:40.602: %PMAN-3-RELOAD_RP: R0/0: pvp: Reloading: Chassis will be reloaded

Chassis 1 reloading, reason - Non participant detected

Jul 28 02:05:42.846: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: process exit with reload fru code
Jul 28 02:05:45.094: %PMAN-3-PROCESS_NOTIFICATION: R0/0: pvp: System report core/RP_0-system-report_20200728-020544- Universal.tar.gz (size: 719 KB) generated and System report info at core/RP_0-system-report_20200728-020544-Universa l-info.txt

Leo Laohoo · ‎07-29-2020

@zamis.ali wrote:

Jul 28 02:11:25.836: %PMAN-3-PROCHOLDDOWN: R0/0: The process rif_mgr has been helddown (rc 139)
Jul 28 02:11:28.297: %PMAN-0-PROCFAILCRIT: R0/0: pvp: A critical process rif_mgr has failed (rc 139)

These two do not look good.
I can't find any bugs/information about the "rif_mgr".

Either raise a TAC Case or downgrade to the latest 16.12.X release.

zamis.ali · ‎07-29-2020

I could have downgraded provided I was able to get onto rommon and maybe try to install/boot via USB.

murmucka · ‎03-20-2023

Hi, do you had here the solution? i have the same problem. thnx

Leo Laohoo · ‎03-20-2023

@murmucka wrote:

Hi, do you had here the solution? i have the same problem. thnx

Create a new thread.

Post the entire boot-up process.

pieterh · ‎07-29-2020

booting a wlc9800 can take a loooooooooooong time!
you may just have interupted a normal boot?

apart from this you supply little information.
what logs do appear on the console? (where does it get "stuck")
is is a single or a HA configuration?
if HA does the other member function normal?
what was the original issue? and why did you change the confreg 0x2142 ?

zamis.ali · ‎07-29-2020

Hi Pieterh,

Thanks for the reply.

1. I have provided the logs above

2. It was HA. Upgraded the version to 17.2.1a, rebooted the first and second time, came up fine and when I rebooted it the third time it never came back up. Broke the HA and isolated the devices from the network as well to see the behavior with no change and it still keeps on looping around.

3. Same issue with the other device

4. I read it in a cisco forum to try and boot it while ignoring the config using the confreg 0x2142, reloaded the device and it still stops at the same loop spot, with an addition of not having access to rommon mode too.

pieterh · ‎07-29-2020

>>> Located C9800-rpboot.17.02.01a.SPA.pkg <<<

are you working on upgrading the controller to 17.02? and this failed?

or has the controller worked with this version earlier?

pieterh · ‎07-29-2020

hi there crossed you reply

the controller keeps a counter how many reboots have failed
after a number of reboots it should still enter ROMMON

zamis.ali · ‎07-29-2020

the controller rebooted fine after the upgrade and I could see that the HA pair was updated as well. When I rebooted it the 3rd time it never came back up. Also to point out, I did not make any changes within those boots.

pieterh · ‎07-29-2020

>>>

4. I read it in a cisco forum to try and boot it while ignoring the config using the confreg 0x2142, reloaded the device and it still stops at the same loop spot, with an addition of not having access to rommon mode too

<<<

this document says something different about confreg 0x2142,

although it mentions 9800-CL there is no different document for 9800-40

- By default, all 9800 appliances (9800-40, 9800-80, 9800-L) have the config-register set 0x2102. However, since they run Linux based IOS-XE, this setting is ignored and only way to prevent return to ROMMON is to configure # no service password-recovery.

Caveat: On 9800-L running ROMMON older than 16.12(3r), this bit in config-register is honored preventing 'break' into ROMMON to do password-recovery.

as your logs say 9800-40 and not a 9800-L this caveat seems not applicable?
and you should still be able to perform:

Step 1. Send 'break' key when you see ##### print on console for system loading image.