Another PW recovery thread (Catalyst 3650)

[erase startup reload]

I've got a 3650 that I need to perform a password recovery. I've tried the power on the device and press the mode button and the switch just booted as normal. I tried holding the button and I tried spamming the button, either worked. I also tried an option I found where you do break on the terminal. I use TeraTerm and default is alt+b. I spammed that and it just locked up my terminal.

First question: Can you confirm I'm supposed to press and HOLD the mode button?

Second question: How long does it usually take for all the system LEDs to stay solid green? I held it for 4-5 minutes and that seems absurd. 

[Peter Paluch]

Hi,
This is certainly unexpected. I suppose that the switch is a standalone switch (not in a stack), and that when the switch fully boots up (albeit with an unknown password), pressing the MODE button briefly truly changes the modes displayed on the LEDs; in other words, I would like to make sure that the MODE button alone is not damaged and works.
To your first question: Yes, you are supposed to press and hold the MODE button, ideally keeping it pressed as you power on the switch already. Hitting the button (repeatedly pressing and releasing in quick succession) is not going to do the trick. As for the second question, it may take about 15-60 seconds depending on when it started being pressed.
Best regards,
Peter

[erase startup reload]

Thank you for the clarification, Peter. You are correct in your assumption, this is a standalone single switch.I will test the mode button as you stated. I have another of the same exact model here I can test this with to see if I am able to get the desired results. Will update on the results of the test at my desk.

[erase startup reload]

Follow Up: I tested this on an identical model switch and it seems to work. Although, the result I recieved is slightly different than what the Cisco documentation provides.

From Cisco: "Reconnect the power cord to the or the active switch. Within 15 seconds, press the Mode button while the System LED is still flashing green. Continue pressing the Mode button until all the system LEDs turn on and remain solid; then release the Mode button."

The point I want to show is that only the SYST and ACTV LEDs turned on, orange in color, and remained solid until I let go of the button. It took roughly 50 seconds. This was good practice as I've only ever done password recovery on Brocade and theirs is easy, just press 'b' on boot. 

I will bring this second switch to my other location and test the first switch one last time. Worst case scenario, I will swap the devices.

Thank you for your time, Peter.

[erase startup reload]

One last clarification please: When you do the "password recovery" on a Cisco device, your only option is to rebuild the config, it doesn't just let you bypass the passwords, correct?

What I am use to with my previous company using Brocade is passwords are disabled so you are able to get all the way into global config without entering a password. At that point  you can remove aaa statements or usernames, etc. and then wr mem and reboot. When I complete the Cisco "password recovery" procedure, it appears it loads a default config and then I'm supposed to wr mem saving the default running config as my startup config. 

Please confirm I'm doing this properly.

Thank you!

[Peter Paluch · Accepted solution]

Hello,

Actually, by default, password recovery on IOS-XE based Cisco switches allows you to keep the original configuration intact - you just tell the device to bypass loading the startup-config that also contains the passwords. Once the switch boots, ignoring the stored configuration, you can access its configuration line and restore the configuration, then change the passwords as you are already logged in.

The steps of this procedure are described in the following document you have no doubt already visited:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/3se/system_management/configuration_guide/b_sm_3se_3650_cg/b_sm_3se_3650_cg_chapter_01111.html#task_1090048

I suggest just one change: Instead of using the copy startup-config running-config command to restore the configuration, use configure replace nvram:startup-config - this is a better way of loading the startup-config. The first command causes the startup-config to be merged with the current configuration; this might have unexpected side effects. The second command (configure replace) makes sure that at the end, the running-config is perfectly equal to the startup-config, performing all necessary additions and deletions.

The overall sequence of steps would be:

1. Break the boot process using the MODE button.
2. Enter the following sequence of commands:
   
   flash_init
   SWITCH_IGNORE_STARTUP_CFG=1
   boot flash:packages.conf
3. After the switch finishes loading, enter "No" when prompted whether to enter the initial configuration dialog.
4. After getting into the normal command line, enter the following commands:
   
   enable
   configure replace nvram:startup-config
5. After this stage, the original configuration of the switch including the unknown passwords is reinstated but since you are still in the privileged mode, you can still modify the configuration. Now change the passwords in the configure terminal as usual - as I do not know what exact type of authentication is used in your configuration (per-line passwords, username/passwords combinations, AAA new-model), I am not suggesting any specific way of doing it but if you'd like, I can suggest a couple ways of finding that out.
6. After modifying the passwords and exiting the configuration mode, save the configuration and reload the switch again:
   
   copy running-config startup-config
   reload
   
   
7. Continue with Step 8 in the document linked above - reload the switch, set the SWITCH_IGNORE_STARTUP_CFG variable back to 0, and when the switch boots up again, set the manual boot to disabled.

Best regards,
Peter

[erase startup reload]

FINAL UPDATE: This worked beautifully on my test switch. Brought it to the site and tried and it appears your guess was correct, the mode button is defective. Luckily, I had the config saved from when I originally wrote it up so I just copy and pasted the config on the new switch. 

The network gods have smiled upon us today!

Thanks so much, Peter!

[Peter Paluch]

Hello,

Thank you very much for letting me know! Admitted, a faulty MODE button is not something seen often. If that switch is under a contract or warranty, I encourage you to open a TAC case for this switch to have it RMAed - feel welcome to direct the TAC engineer to this thread.

Best regards,
Peter

[ziqex]

I have followed the steps but unable to enter configuration mode. 

I've tried rebooting the switch the issue is still present. 

Initial config reinstated but I cannot create/modify local user as unable to go into configuration mode.

Any advice?

Thank you.

Regards,

Daniel