4500-x/VSS Password Recovery while in production?

[RB6502]

I have been asked to try and recover the password on a 4500-X based VSS cluster that's currently in production.  The cluster is part of the network core, and the switch owner is trying to avoid an outage during this procedure by relying on VSS to keep the core up with one switch while the other switch undergoes the password recovery procedure.

The "usual" procedure for password recovery is to break into the switch during boot, set the config register to ignore the startup-config, reboot the switch into a "no-config" state (where you can easily enter enable mode), copy the startup-config to the running-config, update the enable password, save the running config, reset the config register to its previous value & reboot.

The procedure I found for recovering a 4500-X in VSS mode suggests that both switches should be power-cycled & break their boot sequences at the same time, vs one at a time.  The procedure also makes note that you should have the running-config on hand prior to starting password recovery, since in this procedure the running-conifig, rather than the startup-config, is the basis for preserving the switch's configuration.

The way I was going to approach this task was to first powercycle one switch in the cluster & break into it's boot sequence.  I understand from the VSS password recovery procedure that at this point, you must erase the switch's VSS index number (effectively put it in standalone mode) before the config register can be changed to ignore the startup-config.  Once that was done, I was planning on rebooting the switch so it would load with no configuration, then I would extract the startup-config to a text file.  If (hopefully) that configuration is not using password encryption, then I will know the enable password and can simply restore the config register and VSS index number variables respectively, and let it rejoin the cluster.  Then I can go in and update the password normally.

If the passwords are encrypted, this creates a problem as once I load the startup-config & change the enable password on this switch, the config files will no longer match when the switch rejoins the cluster.  Given that this switch is the "newcomer", I'm guessing that the configuration of the active switch will take precedence and overwrite the config with the updated password.

So is there any way to recover one switch in a cluster while the other one stays up (and then, vice-versa for the other switch) without incurring a significantly disruptive outage?

[Leo Laohoo]

You need to have physical access to the appliance to do password recovery.
You also need to physically disconnect all links to the appliance.
Once all the links are physically disconnected, reboot the appliance and force it into ROMMON.
And then do the standard password recovery proceedure.
NOTE: Pray no one has "disabled" the password recovery service.

[RB6502]

Thanks for the reply, but it didn't really answer my questions about what will happen when I try to get the switch with the updated password back into the VSS cluster.  From what Ive read, it seems like re-joining the cluster in that way will cause the switch to be rebooted & have it's configuration overwritten with that of the active switch's configuration, which will wipe out the updated password.

[Leo Laohoo]

---

@RB6502 wrote:

Thanks for the reply, but it didn't really answer my questions about what will happen when I try to get the switch with the updated password back into the VSS cluster.

---

If you want to change the password, it has to be done on the active supervisor card.  

Let me re-run the scenario: 

Let's say that the old sys admin has left.  He took the passwords with him and didn't tell anyone. 

You have a chassis with dual supervisor cards in them and you want to do a password recovery.  So here's what I propose you do:  

1.  Pull the standby supervisor card (active supervisor card still active so the system is still running); 

2.  Perform password recovery on the standby supervisor card; 

3.  Change the passwords on the standby supervisor card; 

4.  REMOVE the Active supervisor card (all system goes down) and ERASE the config; and

5.  Insert the standby supervisor card (with new passwords) -- Standby supervisor card now becomes active

[RB6502]

Hi Leo,

Yes, I like your procedure, but my concern is that there are two switches running as a VSS pair, meaning that they share one configuration.  So I could remove one switch from the VSS pair & leave one to keep the network up with (theoretically) minimal disruption.  I could then follow the recovery procedure you suggest.  But if I understand how VSS works in this scenario, when I go to re-join that switch to the VSS pair, as part of paring the switches the active switch (the one that was still running the network) will cause the joining switch to reboot, and overwrite the joining switch's configuration with the configuration from the active switch.  That will put us back where we started from.  What I'm looking for is a way to accomplish this without taking the network down during the procedure (if possible).

[Leo Laohoo]

Do you know what the SNMP ReadWrite community string is?

[punithrameshofficial]

Hey there @RB6502 
Was this ever fixed .?



@Leo Laohoo  Like always your answers fascinate me and with the finesse that you describe .


Unfortunately i have a VSS pair switch in production that seems to prompt only for the password and not the username on console 
- We have lost ssh  to the device and the site is a critical production site running 24X7.


They might never approve for a password recovery at all . In your experience could you suggest anything? 
these are 4500 switches in VSS mode, I have tried from the other flex pod switches does not seem to allow me in