SSH disabled when downgrade ASR IOS from 16.09.04 to 16.03.02

mohanconnects · ‎06-12-2020

Hello,

we have downgraded some of Cisco routers/switches in our lab, for ASR when downgraded from 16.09.04 to 16.03.02, the SSH gets disabled, have to manually create ssh keys again to make it work using below command:

crypto key generate rsa usage-keys label router-key

But this issue not occurs when try to upgrade from 16.03.02 to 16.09.04.

Usually, before downgrading/upgrading, devices configuration always has taken back up with command wr & copy run start.

Image used for ASR devices:

asr1001x-universalk9.16.03.02.SPA.bin
asr1001x-universalk9.16.09.04.SPA.bin

Below is device log upon booting after device downgrade from 16.09.04 to 16.03.02

%INIT: waited 0 seconds for NVRAM to be available

platform qfp utilization monitor load 80
          ^
% Invalid input detected at '^' marker.

no license smart enable
           ^
% Invalid input detected at '^' marker.



Press RETURN to get started!


*Jun 12 10:01:47.874: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized
*Jun 12 10:01:48.767: %LICENSE-6-EULA_ACCEPT_ALL: The Right to Use End User License Agreement is accepted
*Jun 12 10:01:48.790: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = asr1001x Next reboot level = adventerprise and License = adventerprise
*Jun 12 10:01:52.179: %IOSD_INFRA-6-IFS_DEVICE_OIR: Device bootflash added
*Jun 12 10:01:55.281: dev_pluggable_optics_selftest attribute table internally inconsistent @ 0x125

*Jun 12 10:02:00.905: %NBAR-6-CACHE_SYNC_INFO: Cache synchronizati

User Access Verification

Username: on. Initialized.
*Jun 12 10:02:02.534: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Jun 12 10:02:06.630: %VOICE_HA-7-STATUS: CUBE HA-supported platform detected.
*Jun 12 10:02:07.125: %LINK-3-UPDOWN: Interface Lsmpi0, changed state to up
*Jun 12 10:02:07.144: %LINK-3-UPDOWN: Interface EOBC0, changed state to up
*Jun 12 10:02:07.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface VoIP-Null0, changed state to up
*Jun 12 10:02:07.145: %LINEPROTO-5-UPDOWN: Line protocol on Interface LI-Null0, changed state to up
*Jun 12 10:02:07.145: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Jun 12 10:02:07.151: %LINK-3-UPDOWN: Interface LIIN0, changed state to up
*Jun 12 10:02:09.692: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-intf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001
*Jun 12 10:01:52.861: %CMLIB-6-THROUGHPUT_VALUE: SIP0: cmand:  Throughput license found, throughput set to 2500000 kbps
*Jun 12 10:02:09.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi0, changed state to up
*Jun 12 10:02:09.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC0, changed state to up
*Jun 12 10:02:09.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Jun 12 10:02:09.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN0, changed state to up
*Jun 12 10:02:11.360: %PARSER-4-BADCFG: Unexpected end of configuration file.

*Jun 12 10:02:11.382: %SYS-4-CONFIG_NEWER: Configuration from version 16.9 may not be correctly understood
*Jun 12 10:02:11.439: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1, changed state to up
*Jun 12 10:02:11.444: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback2, changed state to up
*Jun 12 10:02:11.447: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback21, changed state to up
*Jun 12 10:02:11.448: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback57, changed state to up
*Jun 12 10:02:11.450: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback66, changed state to up
*Jun 12 10:02:12.678: %SYS-5-CONFIG_I: Configured from memory by console
*Jun 12 10:02:12.685: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 0/0, interfaces disabled
*Jun 12 10:02:12.686: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 0/1, interfaces disabled
*Jun 12 10:02:12.686: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 0/2, interfaces disabled
*Jun 12 10:02:12.689: %SPA_OIR-6-OFFLINECARD: SPA (BUILT-IN-2T+6X1GE) offline in subslot 0/0
*Jun 12 10:02:12.690: %SPA_OIR-6-OFFLINECARD: SPA (SPA-8X1GE-V2) offline in subslot 0/1
*Jun 12 10:02:12.691: %SPA_OIR-6-OFFLINECARD: SPA (NIM-2MFT-T1/E1) offline in subslot 0/2
*Jun 12 10:02:12.693: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0
*Jun 12 10:02:12.693: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F0
*Jun 12 10:02:12.725: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 0
*Jun 12 10:02:12.942: %SYS-5-RESTART: System restarted --
Cisco IOS Software [Denali], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 08-Nov-16 18:21 by mcpre
*Jun 12 10:02:16.013: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Jun 12 10:02:16.013: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Jun 12 10:02:18.153: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/2, changed state to down
*Jun 12 10:02:18.155: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/3, changed state to down
*Jun 12 10:02:18.155: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/4, changed state to down
*Jun 12 10:02:18.156: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/5, changed state to down
*Jun 12 10:02:18.677: %SYS-6-BOOTTIME: Time taken to reboot after reload =  176 seconds
*Jun 12 10:02:19.152: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Jun 12 10:02:19.153: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/2, changed state to down
*Jun 12 10:02:19.155: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/3, changed state to down
*Jun 12 10:02:19.156: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/4, changed state to down
*Jun 12 10:02:19.156: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/5, changed state to down
*Jun 12 10:02:19.737: %SPA_OIR-6-ONLINECARD: SPA (SPA-8X1GE-V2) online in subslot 0/1
*Jun 12 10:02:20.152: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Jun 12 10:02:20.385: %SPA_OIR-6-ONLINECARD: SPA (BUILT-IN-2T+6X1GE) online in subslot 0/0
*Jun 12 10:02:21.511: %LINK-3-UPDOWN: Interface GigabitEthernet0/1/4, changed state to down
*Jun 12 10:02:22.384: %LINK-3-UPDOWN: Interface Crypto-Engine0/0/8, changed state to up
*Jun 12 10:02:23.384: %LINEPROTO-5-UPDOWN: Line protocol on Interface Crypto-Engine0/0/8, changed state to up
*Jun 12 10:02:22.582: %LINK-3-UPDOWN: SIP0/1: Interface GigabitEthernet0/1/4, changed state to down
*Jun 12 10:02:23.024: %LINK-3-UPDOWN: SIP0/0: Interface GigabitEthernet0/0/2, changed state to down
*Jun 12 10:02:29.015: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/2, changed state to up
*Jun 12 10:02:29.024: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/3, changed state to up
*Jun 12 10:02:29.034: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/4, changed state to up
*Jun 12 10:02:29.044: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/5, changed state to up
*Jun 12 10:02:30.015: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/2, changed state to up
*Jun 12 10:02:29.017: %LINK-3-UPDOWN: SIP0/0: Interface GigabitEthernet0/0/2, changed state to up
*Jun 12 10:02:30.024: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/3, changed state to up
*Jun 12 10:02:30.035: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/4, changed state to up
*Jun 12 10:02:30.044: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/5, changed state to up

Why the ssh keys got disabled and how to fix this?

Thanks,

Mohan

Deepak Kumar · ‎06-12-2020

Hi,

I am not sure about specific your case but many times it may happen as Higher version supports to latest algorithms and your certificate may be migrated in the latest algorithm module. But now you downgraded again then it downgraded version does not support to new algorithm and certificate may corrupt or deleted.

Check the release note for all details as changes etc. Personally, I am enabling telnet as well while upgrading or downgrading the IOS. I faced the same and lost ssh access due to a bug.

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

Leo Laohoo · ‎06-12-2020

16.3.2? Have you ever tried upgrading to a newer 16.3.X version, like 16.3.10?

mohanconnects · ‎06-13-2020

downgrading to 16.3.10 also disabled ssh.

Leo Laohoo · ‎06-13-2020

Post the complete output to the following commands:

sh version
sh run | i ssh
sh run | b line vty 0

mohanconnects · ‎12-10-2020

Sorry of responding late, we upgraded to 16.09.04, I will post when we try to downgrade again. thanks

Richard Burts · ‎12-12-2020

I am not sure of the specifics of this issue but I believe that a general explanation might clarify what is going on. A key part of the explanation relates to this message generated during the boot process:

*Jun 12 10:02:11.382: %SYS-4-CONFIG_NEWER: Configuration from version 16.9 may not be correctly understood

I believe that these are the major points in the explanation:

- the older version of code used a certain syntax for the SSH keys

- the newer version of code uses a different syntax for the SSH keys

- when you upgrade from older to newer the upgrade process manages the change in syntax details, translates from old syntax to new syntax, and the SSH keys are preserved

- when you downgrade from newer to older there is no conversion process for the syntax for SSH keys and the existing SSH keys are deleted because the old syntax does not know how to handle them.

This is, in fact, a fairly common issue when downgrading versions of code that some new syntax is not compatible with the older version of code and some features may get dropped.

HTH

Rick