Lost private keys after failed upgrade from 17.3.5a to 17.9.3

[Bernd Nies]

Hi,

An upgrade of Catalyst 9800-40 from 17.3.5a to 17.9.3 ended in an endless boot loop. Most likely because it was forgotten to upgrade ROMMON from 16.10(2r) 17.7(3r). After breaking boot process, boot old system the config was gone. Restored the startup-config from backup but now all the private keys are gone.

*Aug 21 2023 06:29:20.923: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI0 created succesfully
*Aug 21 06:29:23.807: private key not found
*Aug 21 06:29:23.812: private key not found
*Aug 21 06:29:23.818: private key not found
*Aug 21 06:29:23.819: private key not found
*Aug 21 06:29:23.824: private key not found
*Aug 21 06:29:23.870: private key not found
*Aug 21 06:29:23.886: private key not found
*Aug 21 06:29:23.888: private key not found
*Aug 21 06:29:23.892: private key not found
*Aug 21 06:29:23.897: private key not found

Why were these deleted? Can the private keys be restored? How can the private keys being backed up?

Thanks in advance,

Bernd

[marce1000]

 - Note that if  aes based password encryption was enabled on the original configuration, all keys and passwords would have to be reconfigured , 

 M.

[Bernd Nies]

Only "service password-encryption" is enabled in the original config. Last time when I updated within IOS XE 17.3.x that was not lost.

[Rich R]

Why were these deleted? Your guess (ROMMON) is as good as mine ...
Can the private keys be restored? No
How can the private keys being backed up? They can't.

We opened a TAC case to ask similar questions when we had just started using the 9800 - final answer below:

"To share an update, this case was in a Cisco pending state for past months since we were waiting on DE group for :
CSCvy56280 Failure to access Web GUI after backup/restore of the config

This bug has just been junked by the developers and I am tracking this internally to see what other alternatives or enhancements can be made with the business unit/product team.

As per the Developer and PKI team to avoid security violations, keys are not exposed in config, but stored in private NVRAM. So with backup/restore actions ,the current issues are pointing to a key binding state between the cert and the private key wherein there is need to manually re-generate the self signed cert .

As well with third party certs, there is a need to import again following restore as mentioned in the best practices section “Configuration file management”: https://www.cisco.com/c/en/us/products/collateral/wireless/catalyst-9800-series-wireless-controllers/guide-c07-743627.html#Generalcontrollersettings"

The bug ID mentioned is not viewable.

[Bernd Nies]

So it is expected behaviour to loose all the private keys and certificates when upgrading ROMMON software?

[Rich R]

> So it is expected behaviour to loose all the private keys and certificates when upgrading ROMMON software?
Absolutely not, that should not happen.  But it has happened, and may, or may not, have been caused by you upgrading without the correct ROMMON.  But either way, that is a moot point now because you are where you are.

Your questions were around backup and restore of the private keys to which I provided the answers which TAC gave us some time ago which is that they must be re-installed.

Best to always read through all the relevant docs below + release notes before starting on upgrade, test in lab before production and document all steps carefully so that you have a well tested and defined process when it comes time to deploy in production.  And you should always understand the recovery options for when things go wrong - because sometimes they do go wrong (avoid trying to use ISSU because that increases the risk of things going wrong - see various other threads on that).  We asked TAC the question precisely because we were documenting our restore process for replacing a failed 9800.

[Bernd Nies]

If password encryption is enabled with

key config-key password-encrypt <private-key> 
password encryption aes

Do all the SSID PSK and AP mgmtuser passwords get lost every time a software update is done?

What security benefit is it when the private key for encryption is in cleartext in the config?

[Rich R]

> Do all the SSID PSK and AP mgmtuser passwords get lost every time a software update is done? 
No

> What security benefit is it when the private key for encryption is in cleartext in the config?
That line does not appear in the config at all.  The device stores the key in private NVRAM (same place as cert keys) which are not user accessible so once configured you can never view the private key.