Re: SSH Configuration in my ISR4000's running IOS-XE after I was hacke

Elito Haylett · ‎11-29-2023

I wanted to bring this very important matter to your attention if you're running Cisco IOS XE Software, Version 17.09.02a on any platform that supports it.

Last week my ISR4431 and 4331 were hacked due to the following bug (Webui) that was discovered in IOS XE after running the Cisco CLI Analyzer:

ECH-ISR4431

1 Result

IOS-XE System Diagnostics

Diagnostic Checks: 4055

Nov 25th 2023 4:19:43 pm (2 hours ago)

1 Danger 2 Warning 3 Info

json

This device is showing evidence of encountering CSCwh87343: Cisco IOS XE Software Web UI Privilege Escalation Vulnerability

SYMPTOMS:

Our investigation has determined that the actors exploited two previously unknown issues.

The attacker first exploited CVE-2023-20198 to gain initial access and issued a privilege 15 command to create a local user and password combination. This allowed the user to log in with normal user access.

The attacker then exploited another component of the web UI feature, leveraging the new local user to elevate privilege to root and write the implant to the file system. Cisco has assigned CVE-2023-20273 to this issue.

CVE-2023-20198 has been assigned a CVSS Score of 10.0.
CVE-2023-20273 has been assigned a CVSS Score of 7.2.

Both of these CVEs are being tracked by CSCwh87343.

For steps to close the attack vector for these vulnerabilities, see the Recommendations section of this advisory.
This advisory is available at the following link:
https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-iosxe-webui-privesc-j22SaA4z

CONDITIONS:

Please refer to the Security Advisory.

MITIGATION:

Workaround:

Please refer to the Security Advisory.

ADDITIONAL INFORMATION:

Please refer to the Security Advisory.

CSCwh87343

EVIDENCE:

N/A

Cisco IOS XE Software, Version 17.09.02a

...

ip http secure-server

...

Nov 25 15:59:26: %SYS-5-CONFIG_P: Configured programmatically by process SEP_webui_wsma_http from console as ehaylett on vty5

This device is susceptible to CSCwh87343: Cisco IOS XE Software Web UI Privilege Escalation Vulnerability

This device is showing evidence of encountering CSCwh60107: In the show tech file, "enable secret" does not get hidden.

My devices between sites were operating normal with with IKEv2 site to site VPN. On 11/20/2023 we lost connectivity between the sites thinking it was an issue with one of the ISP's. We discovered that it wasn't ISP related and started troubleshooting. In the process we didn't notice any real subtle changes in the configuration because all focus was on trying to get the commuincation between the sites back up. We started debugging IKEv2 SA to see if there were issues with the key authentication that would prevent the tunnels from coming up and according to the debugs and the following was the output:

ECH-ISR4331-220#deb crypto ipsec error
Crypto IPSEC Error debugging is on
ECH-ISR4331-220#

000411: Nov 21 23:33:51.380: IKEv2:(SESSION ID = 1,SA ID = 1):Received Packet [From 100.38.124.198:500/To 108.58.36.170:500/VRF i0:f0]
Initiator SPI : 5CB4EE671638C304 - Responder SPI : DB9B54F44B839912 Message id: 0
IKEv2 INFORMATIONAL Exchange REQUEST
Payload contents:

000412: Nov 21 23:33:51.380: IKEv2:(SESSION ID = 1,SA ID = 1):parsing ENCR payload
000413: Nov 21 23:33:51.380: IKEv2:(SESSION ID = 1,SA ID = 1):parsing DELETE payload DELETE
000414: Nov 21 23:33:51.380: IKEv2:(SESSION ID = 1,SA ID = 1):parsing NOTIFY payload NOTIFY(DELETE_REASON)

000415: Nov 21 23:33:51.380: IKEv2:(SESSION ID = 1,SA ID = 1):Building packet for encryption.

000416: Nov 21 23:33:51.381: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 100.38.124.198:500/From 108.58.36.170:500/VRF i0:f0]
Initiator SPI : 5CB4EE671638C304 - Responder SPI : DB9B54F44B839912 Message id: 0
IKEv2 INFORMATIONAL Exchange RESPONSE
Payload contents:
ENCR

000417: Nov 21 23:33:51.382: IKEv2:(SESSION ID = 1,SA ID = 1):Process delete request from peer
000418: Nov 21 23:33:51.382: IKEv2:(SESSION ID = 1,SA ID = 1):Processing DELETE INFO message for IKEv2 SA [ISPI: 0x5CB4EE671638C304 RSPI: 0xDB9B54F44B839912]
000419: Nov 21 23:33:51.382: IKEv2:(SESSION ID = 1,SA ID = 1):Check for existing active SA
000420: Nov 21 23:33:51.382: IKEv2:(SESSION ID = 1,SA ID = 1):Delete all IKE SAs
000421: Nov 21 23:33:51.382: IKEv2:(SESSION ID = 1,SA ID = 1):Deleting SA
000422: Nov 21 23:33:51.383: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
000423: Nov 21 23:33:51.384: IPSEC: sa null
000424: Nov 21 23:34:05.416: IPSEC(ERROR): ipsec_isakmp_sa_initiate_internal

000425: Nov 21 23:35:01: %SEC-6-IPACCESSLOGP: list SDM_1 permitted udp 172.168.102.6(39954) -> 172.168.100.4(162), 1 packet
000426: Nov 21 23:35:43.417: IPSEC(ERROR): ipsec_isakmp_sa_initiate_internal

000427: Nov 21 23:38:08.436: IPSEC(ERROR): ipsec_isakmp_sa_initiate_internal

000428: Nov 21 23:38:33.252: IKEv2:% Getting preshared key from profile keyring ECH138220
000429: Nov 21 23:38:33.252: IKEv2:% Matched peer block 'ECH-ISR4431-138'
000430: Nov 21 23:38:33.252: IKEv2:(SESSION ID = 0,SA ID = 0):Searching Policy with fvrf 0, local address 108.58.36.170
000431: Nov 21 23:38:33.252: IKEv2:(SESSION ID = 0,SA ID = 0):Using the Default Policy for Proposal
000432: Nov 21 23:38:33.252: IKEv2:(SESSION ID = 0,SA ID = 0):Found Policy 'default'
000433: Nov 21 23:38:33.253: IKEv2:(SESSION ID = 1,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH public key, DH Group 19
000434: Nov 21 23:38:33.254: IKEv2:(SESSION ID = 1,SA ID = 1):(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
000435: Nov 21 23:38:33.254: IKEv2:(SESSION ID = 1,SA ID = 1):Request queued for computation of DH key
000436: Nov 21 23:38:33.254: IKEv2:(SESSION ID = 1,SA ID = 1):IKEv2 initiator - no config data to send in IKE_SA_INIT exch
000437: Nov 21 23:38:33.254: IKEv2:(SESSION ID = 1,SA ID = 1):Generating IKE_SA_INIT message
000438: Nov 21 23:38:33.254: IKEv2:(SESSION ID = 1,SA ID = 1):IKE Proposal: 1, SPI size: 0 (initial negotiation),
Num. transforms: 9
AES-CBC SHA512 SHA384 SHA512 SHA384 DH_GROUP_256_ECP/Group 19 DH_GROUP_2048_MODP/Group 14 DH_GROUP_521_ECP/Group 21 DH_GROUP_1536_MODP/Group 5

000439: Nov 21 23:38:33.255: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 100.38.124.198:500/From 108.58.36.170:500/VRF i0:f0]
Initiator SPI : 8205FF1EFDBA5568 - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N VID VID VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)

000440: Nov 21 23:38:33.256: IKEv2:(SESSION ID = 1,SA ID = 1):Insert SA
000441: Nov 21 23:38:33.256: IPSEC(ERROR): crypto_notify_rp for ACL: SDM_1, handle: 0x8000000040000005, local: 172.168.102.0/24, remote: 172.168.100.0/24, Rejected notify RP, elapse time 5 < 1000

000442: Nov 21 23:38:33.286: IKEv2:(SESSION ID = 1,SA ID = 1):Received Packet [From 100.38.124.198:500/To 108.58.36.170:500/VRF i0:f0]
Initiator SPI : 8205FF1EFDBA5568 - Responder SPI : 4E35E68F2317E51D Message id: 0
IKEv2 IKE_SA_INIT Exchange RESPONSE
Payload contents:

000443: Nov 21 23:38:33.286: IKEv2:(SESSION ID = 1,SA ID = 1):parsing SA payload SA
000444: Nov 21 23:38:33.286: IKEv2:(SESSION ID = 1,SA ID = 1):parsing KE payload KE
000445: Nov 21 23:38:33.286: IKEv2:(SESSION ID = 1,SA ID = 1):parsing N payload N
000446: Nov 21 23:38:33.286: IKEv2:(SESSION ID = 1,SA ID = 1):parsing VID payload VID
000447: Nov 21 23:38:33.286: IKEv2:(SESSION ID = 1,SA ID = 1):parsing VID payload VID
000448: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):parsing VID payload VID
000449: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):parsing VID payload VID
000450: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):parsing NOTIFY payload NOTIFY(NAT_DETECTION_SOURCE_IP)
000451: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):parsing NOTIFY payload NOTIFY(NAT_DETECTION_DESTINATION_IP)
000452: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):parsing CERTREQ payload CERTREQ
000453: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):parsing NOTIFY payload NOTIFY(HTTP_CERT_LOOKUP_SUPPORTED)

000454: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):Processing IKE_SA_INIT message
000455: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):Verify SA init message
000456: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):Processing IKE_SA_INIT message
000457: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):Checking NAT discovery
000458: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):NAT not found
000459: Nov 21 23:38:33.287: IKEv2:(SESSION ID = 1,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH secret key, DH Group 19
000460: Nov 21 23:38:33.292: IKEv2:(SESSION ID = 1,SA ID = 1):(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
000461: Nov 21 23:38:33.292: IKEv2:(SESSION ID = 1,SA ID = 1):Request queued for computation of DH secret
000462: Nov 21 23:38:33.292: IKEv2:(SESSION ID = 1,SA ID = 1):(SA ID = 1):[IKEv2 -> Crypto Engine] Calculate SKEYSEED and create rekeyed IKEv2 SA
000463: Nov 21 23:38:33.292: IKEv2:(SESSION ID = 1,SA ID = 1):(SA ID = 1):[Crypto Engine -> IKEv2] SKEYSEED calculation and creation of rekeyed IKEv2 SA PASSED
000464: Nov 21 23:38:33.292: IKEv2:(SESSION ID = 1,SA ID = 1):Completed SA init exchange
000465: Nov 21 23:38:33.292: IKEv2:(SESSION ID = 1,SA ID = 1):Check for EAP exchange
000466: Nov 21 23:38:33.292: IKEv2:(SESSION ID = 1,SA ID = 1):Generate my authentication data
000467: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):Use preshared key for id 108.58.36.170, key len 10
000468: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
000469: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
000470: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):Get my authentication method
000471: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):My authentication method is 'PSK'
000472: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):Check for EAP exchange
000473: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):Generating IKE_AUTH message
000474: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):Constructing IDi payload: '108.58.36.170' of type 'IPv4 address'
000475: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):ESP Proposal: 1, SPI size: 4 (IPSec negotiation),
Num. transforms: 3
AES-CBC SHA512 Don't use ESN
000476: Nov 21 23:38:33.293: IKEv2:(SESSION ID = 1,SA ID = 1):Building packet for encryption.
Payload contents:
VID IDi AUTH SA TSi TSr NOTIFY(INITIAL_CONTACT) NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)

000477: Nov 21 23:38:33.294: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 100.38.124.198:500/From 108.58.36.170:500/VRF i0:f0]
Initiator SPI : 8205FF1EFDBA5568 - Responder SPI : 4E35E68F2317E51D Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:
ENCR

000478: Nov 21 23:38:33.319: IKEv2:(SESSION ID = 1,SA ID = 1):Received Packet [From 100.38.124.198:500/To 108.58.36.170:500/VRF i0:f0]
Initiator SPI : 8205FF1EFDBA5568 - Responder SPI : 4E35E68F2317E51D Message id: 1
IKEv2 IKE_AUTH Exchange RESPONSE
Payload contents:

000479: Nov 21 23:38:33.319: IKEv2:(SESSION ID = 1,SA ID = 1):parsing ENCR payload
000480: Nov 21 23:38:33.319: IKEv2:(SESSION ID = 1,SA ID = 1):parsing VID payload VID
000481: Nov 21 23:38:33.319: IKEv2:(SESSION ID = 1,SA ID = 1):parsing IDr payload IDr
000482: Nov 21 23:38:33.319: IKEv2:(SESSION ID = 1,SA ID = 1):parsing AUTH payload AUTH
000483: Nov 21 23:38:33.319: IKEv2:(SESSION ID = 1,SA ID = 1):parsing SA payload SA
000484: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):parsing TSi payload TSi
000485: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):parsing TSr payload TSr
000486: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):parsing NOTIFY payload NOTIFY(SET_WINDOW_SIZE)
000487: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):parsing NOTIFY payload NOTIFY(ESP_TFC_NO_SUPPORT)
000488: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):parsing NOTIFY payload NOTIFY(NON_FIRST_FRAGS)

000489: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):Process auth response notify
000490: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):Searching policy based on peer's identity 'ECH-ISR4431-138.dyndns.org' of type 'FQDN'
000491: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):Searching Policy with fvrf 0, local address 108.58.36.170
000492: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):Using the Default Policy for Proposal
000493: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):Found Policy 'default'
000494: Nov 21 23:38:33.320: IKEv2:(SESSION ID = 1,SA ID = 1):Verify peer's policy
000495: Nov 21 23:38:33.321: IKEv2:(SESSION ID = 1,SA ID = 1):Peer's policy verified
000496: Nov 21 23:38:33.321: IKEv2:(SESSION ID = 1,SA ID = 1):Get peer's authentication method
000497: Nov 21 23:38:33.321: IKEv2:(SESSION ID = 1,SA ID = 1):Peer's authentication method is 'PSK'
000498: Nov 21 23:38:33.321: IKEv2:(SESSION ID = 1,SA ID = 1):Get peer's preshared key for ECH-ISR4431-138.dyndns.org
000499: Nov 21 23:38:33.321: IKEv2:(SESSION ID = 1,SA ID = 1):Verify peer's authentication data
000500: Nov 21 23:38:33.321: IKEv2:(SESSION ID = 1,SA ID = 1):Use preshared key for id ECH-ISR4431-138.dyndns.org, key len 10
000501: Nov 21 23:38:33.321: IKEv2:(SESSION ID = 1,SA ID = 1):[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
000502: Nov 21 23:38:33.321: IKEv2:(SESSION ID = 1,SA ID = 1):[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
000503: Nov 21 23:38:33.322: IKEv2:(SESSION ID = 1,SA ID = 1):Verification of peer's authentication data PASSED
000504: Nov 21 23:38:33.322: IKEv2:(SESSION ID = 1,SA ID = 1):Check for EAP exchange
000505: Nov 21 23:38:33.322: IKEv2:(SESSION ID = 1,SA ID = 1):Processing IKE_AUTH message
000506: Nov 21 23:38:33.323: IKEv2:Requesting IPsec policy verification by ikev2 osal engine

000507: Nov 21 23:38:33.323: IKEv2:(SESSION ID = 1,SA ID = 1):IPSec policy validate request sent for profile ECH-ISR4331-220_Profile with psh index 1.

000508: Nov 21 23:38:33.324: IKEv2:(SESSION ID = 1,SA ID = 1):(SA ID = 1):[IPsec -> IKEv2] Callback received for the validate proposal - PASSED.

000509: Nov 21 23:38:33.324: IKEv2:(SESSION ID = 1,SA ID = 1):IKEV2 SA created; inserting SA into database. SA lifetime timer (86400 sec) started
000510: Nov 21 23:38:33.324: IKEv2:(SESSION ID = 1,SA ID = 1):Session with IKE ID PAIR (ECH-ISR4431-138.dyndns.org, 108.58.36.170) is UP
000511: Nov 21 23:38:33.324: IKEv2:(SESSION ID = 0,SA ID = 0):IKEv2 MIB tunnel started, tunnel index 1
000512: Nov 21 23:38:33.324: IKEv2:(SESSION ID = 1,SA ID = 1):Load IPSEC key material
000513: Nov 21 23:38:33.325: IKEv2:(SESSION ID = 1,SA ID = 1):(SA ID = 1):[IKEv2 -> IPsec] Create IPsec SA into IPsec database
000514: Nov 21 23:38:33.346: IKEv2:(SESSION ID = 1,SA ID = 1):(SA ID = 1):[IPsec -> IKEv2] Creation of IPsec SA into IPsec database PASSED
000515: Nov 21 23:38:33.347: IKEv2:(SESSION ID = 1,SA ID = 1):Checking for duplicate IKEv2 SA
000516: Nov 21 23:38:33.347: IKEv2:(SESSION ID = 1,SA ID = 1):No duplicate IKEv2 SA found
000517: Nov 21 23:41:01: %SEC-6-IPACCESSLOGP: list SDM_1 permitted udp 172.168.102.6(39954) -> 172.168.100.4(162), 3 packets

ECH-ISR4331-220#show crypto ikev2 stats
--------------------------------------------------------------------------------
Crypto IKEv2 SA Statistics
--------------------------------------------------------------------------------
System Resource Limit: 0 Max IKEv2 SAs: 0 Max in nego(in/out): 40/400
Total incoming IKEv2 SA Count: 0 active: 0 negotiating: 0
Total outgoing IKEv2 SA Count: 1 active: 1 negotiating: 0
Incoming IKEv2 Requests: 0 accepted: 0 rejected: 0
Outgoing IKEv2 Requests: 3 accepted: 3 rejected: 0
Rejected IKEv2 Requests: 0 rsrc low: 0 SA limit: 0
IKEv2 packets dropped at dispatch: 0
Incoming Requests dropped as LOW Q limit reached : 0
Incoming IKEV2 Cookie Challenged Requests: 0
accepted: 0 rejected: 0 rejected no cookie: 0
Total Deleted sessions of Cert Revoked Peers: 0
Total init sa request rejected due to queue limit : 0
Sessions with Quantum Resistance: 0 Manual: 0 Dynamic: 0
PPK Identity Mismatch: 0
PPK Retrieve Failure - ALL: 0 With PPK Required: 0
PPK Authentication Failure - ALL: 0 With PPK Required: 0

ECH-ISR4331-220#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
1 108.58.36.170/500 100.38.124.198/500 none/none READY
Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:19, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/373 sec
CE id: 1003, Session-id: 3
Local spi: 8205FF1EFDBA5568 Remote spi: 4E35E68F2317E51D

IPv6 Crypto IKEv2 SA

ECH-ISR4331-220#show crypto ikev2 diagnose error
Exit Path Table - status: enable, current entry 4, deleted 0, max allow 50

Error(3): A supplied parameter is incorrect

-Traceback= 1#19e6c295e8d59533dd61baf308ca9dd4 :55EB7BF45000+13465776 :55EB7BF45000+1345A304 :55EB7BF45000+133938DD :55EB7BF45000+1337C71B :55EB7BF45000+1337BF7F :55EB7BF45000+1333B1AF :55EB7BF45000+13345412 :55EB7BF45000+13345DE3 :55EB7BF45000+133AD3B1 :55EB7BF45000+133ADA68 :55EB7BF45000+13361F2A :55EB7BF45000+13409A2D

Error(3): Unable to updat SK values with the PPK

-Traceback= 1#19e6c295e8d59533dd61baf308ca9dd4 :55EB7BF45000+1337E9BF :55EB7BF45000+1333CC76 :55EB7BF45000+13345412 :55EB7BF45000+13345DE3 :55EB7BF45000+133AD3B1 :55EB7BF45000+133ADA68 :55EB7BF45000+13361F2A :55EB7BF45000+13409A2D

Error(2): A supplied parameter is incorrect

-Traceback= 1#19e6c295e8d59533dd61baf308ca9dd4 :55EB7BF45000+133AB3F4 :55EB7BF45000+1336F21D :55EB7BF45000+13375F3B :55EB7BF45000+133386A2 :55EB7BF45000+13345412 :55EB7BF45000+13345DE3 :55EB7BF45000+133AD3B1 :55EB7BF45000+133ADA68 :55EB7BF45000+13361F2A :55EB7BF45000+13409A2D

Error(1): Detected an invalid IKE SPI

-Traceback= 1#19e6c295e8d59533dd61baf308ca9dd4 :55EB7BF45000+133AC52A :55EB7BF45000+133ADA68 :55EB7BF45000+13361F2A :55EB7BF45000+13409A2D

000405: Nov 21 23:25:20: %FMFP-3-OBJ_DWNLD_TO_DP_STUCK: F0/0: fman_fp_image: AOM download to Data Plane is stuck for more than 1800 seconds due to resolve object: obj[246] type[32] 'Void Intf (5)', resulting in it being a pending-issue object
000406: Nov 21 23:26:01: %SEC-6-IPACCESSLOGP: list SDM_1 permitted udp 172.168.102.6(39954) -> 172.168.100.4(162), 2 packets

We could't understand why the tunnels weren't being established after they were working and no configuration changes were made. We looked at the log and this is what we discovered:

1) Someone was able to access the device/s using the "webui account" on the main site router from an unknown address as displayed in the log file output below. They also configured an admin account on the remote site router because they made changes on that device.

Nov 21 21:36:20: %SYS-6-LOGOUT: User webui has exited tty session 874(38.54.93.133)
Nov 21 21:36:20: %SYS-6-TTY_EXPIRE_TIMER: (exec timer expired, tty 874 (38.54.93.133)), user webui
Nov 21 21:10:01: %BUFCAP-6-DISABLE: Capture Point CAP disabled.
Nov 21 21:09:10: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: webui] [Source: 38.54.93.133] [localport: 22] at 21:09:10 est Tue Nov 21 2023

2) Once they were in they configured an IKEv2 session between my site (destaddr=100.38.124.198) and their sites (srcaddr=167.94.138.101), (srcaddr=167.248.133.172) etc. as you can see by all the different source addresses to reroute traffic. The sessions were failing because I was making changes to the IKEv2 Keyring and profile on the main router during my troubleshooting process and thus disrupting their remote session until they were able to log back in to see the changes I made then applied it to their router. They were using these encryption algorythym which I never configured in the devices.

Nov 21 21:58:10: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x78629A0F(2019727887), srcaddr=167.94.138.101, input interface=GigabitEthernet0/0/2
Nov 22 00:57:06: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ssh-ed25519-cert-v01@openssh.com,ssh-ed25519 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 22 00:57:06: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 22 04:48:53: %CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 161.35.213.143 failed its sanity check or is malformed
Nov 22 01:47:20: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 103.194.243.188
Nov 22 06:58:55: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 205.210.31.94
Nov 22 07:23:34: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x78629A0F(2019727887), srcaddr=167.248.133.172, input interface=GigabitEthernet0/0/2
Nov 22 12:10:43: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 198.235.24.200
Nov 22 11:06:55: %CRYPTO-6-IKMP_POLICY_DEFAULT: Using ISAKMP Default policies
Nov 22 11:06:54: %CRYPTO-6-IKMP_POLICY_DEFAULT: Using ISAKMP Default policies
Nov 22 17:20:54: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Aggressive mode failed with peer at 192.241.211.5
Nov 22 17:20:54: %CRYPTO-6-IKMP_POLICY_DEFAULT: Using ISAKMP Default policies
Nov 22 19:14:06: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 205.210.31.204
Nov 22 19:40:34: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x30303030(808464432), srcaddr=71.6.146.185, input interface=GigabitEthernet0/0/2
Nov 22 23:49:57: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 124.65.227.154
Nov 23 00:58:32: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 218.92.0.112
Nov 23 00:55:52: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 125.71.200.138
Nov 23 00:45:02: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 205.210.31.184
Nov 23 00:44:51: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ssh-ed25519 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 23 00:44:50: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ecdsa-sha2-nistp521 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 23 00:44:50: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ecdsa-sha2-nistp384 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 23 00:44:49: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ecdsa-sha2-nistp256 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 23 00:44:48: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 104.156.155.30
Nov 23 00:44:47: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ssh-dss server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 23 00:44:46: %SSH-3-NO_MATCH: No matching kex algorithm found: client diffie-hellman-group1-sha1 server curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
Nov 23 00:43:43: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ssh-ed25519-cert-v01@openssh.com,ssh-ed25519 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 23 00:43:42: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 23 02:28:12: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 218.92.0.25
Nov 23 02:15:12: %CRYPTO-6-IKMP_POLICY_DEFAULT: Using ISAKMP Default policies
Nov 23 01:52:12: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x4D658221(1298498081), srcaddr=167.94.138.111, input interface=GigabitEthernet0/0/2
Nov 23 05:32:15: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x4D658221(1298498081), srcaddr=162.142.125.90, input interface=GigabitEthernet0/0/2
Nov 23 09:05:23: %CRYPTO-6-IKMP_POLICY_DEFAULT: Using ISAKMP Default policies
Nov 23 09:05:22: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Aggressive mode failed with peer at 169.228.66.212
Nov 23 09:05:22: %CRYPTO-6-IKMP_POLICY_DEFAULT: Using ISAKMP Default policies
Nov 23 08:50:22: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 141.98.11.90
Nov 23 07:07:12: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 205.210.31.44
Nov 23 12:48:49: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 198.235.24.69
Nov 23 12:14:15: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 141.98.11.90
Nov 23 11:59:31: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 182.16.184.204
Nov 23 13:51:19: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 61.177.172.160
Nov 23 15:57:27: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
Nov 23 15:51:29: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
Nov 24 10:42:05: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x488B1472(1217074290), srcaddr=43.157.20.143, input interface=GigabitEthernet0/0/2
Nov 24 08:44:26: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x78629A0F(2019727887), srcaddr=162.142.125.88, input interface=GigabitEthernet0/0/2
Nov 24 06:54:19: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x3127FCB0(824704176), srcaddr=71.6.134.235, input interface=GigabitEthernet0/0/2
Nov 24 01:40:48: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:46: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:44: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:39: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:37: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:35: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:33: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:32: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:30: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:29: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:27: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:25: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:22: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:19: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:16: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:15: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:13: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:11: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:10: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:07: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:04: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:40:01: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:58: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:56: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:53: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:51: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:49: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:47: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:46: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:43: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:42: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:41: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:39: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:38: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:36: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:35: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:30: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:29: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:27: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:26: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:25: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:23: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:22: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:21: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:19: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:16: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:14: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:12: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:08: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:07: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:06: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:05: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:03: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:39:01: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:58: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:56: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:54: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:53: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:51: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:49: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:44: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:42: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:41: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:35: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:31: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:30: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:28: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:27: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:26: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:24: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:23: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:22: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:20: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:19: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:18: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:16: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:15: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:14: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:13: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:11: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:09: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:07: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:05: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:38:04: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:59: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:58: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:56: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:55: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:54: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:52: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:51: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:50: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:49: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:47: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:46: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:37:45: %SSH-3-NO_MATCH: No matching cipher found: client aes128-cbc,blowfish-cbc,3des-cbc server chacha20-poly1305@openssh.com,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-gcm,aes256-gcm,aes128-ctr,aes192-ctr,aes256-ctr
Nov 24 01:15:14: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 87.236.176.194
Nov 24 00:45:55: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=100.38.124.198, prot=50, spi=0x4D658221(1298498081), srcaddr=167.248.133.163, input interface=GigabitEthernet0/0/2
Nov 24 00:25:37: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ssh-ed25519-cert-v01@openssh.com,ssh-ed25519 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 24 00:25:34: %SSH-3-NO_MATCH: No matching hostkey algorithm found: client ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 server rsa-sha2-512,rsa-sha2-256,ssh-rsa
Nov 24 00:08:25: %SSH-4-SSH2_UNEXPECTED_MSG: Unexpected message type has arrived. Terminating the connection from 205.210.31.229

We mitigated the compromise by:

1) Changing the webui password on both routers to a more secure password and deleted any addtional accounts I didn't create. 2) Changing both static IP and dynamic IP addresses. Deleted the dyndns host name and will create a new hostname one to map to my dynamic IP. 3) Installed the new IOS-XE (isr4400-universalk9.17.12.02.SPA.bin) images that fixed the bug on both devices

What are these ssh commands still in the devices?

no ip ssh rekey time
no ip ssh rekey volume
ip ssh server authenticate user publickey
ip ssh server authenticate user keyboard
ip ssh server authenticate user password
no ip ssh server peruser session limit
ip ssh server certificate profile
server
no ocsp-response include
user
no ocsp-response required
ip ssh server algorithm mac hmac-sha2-256-etm@openssh.com hmac-sha2-512-etm@openssh.com
ip ssh server algorithm encryption chacha20-poly1305@openssh.com aes128-gcm@openssh.com aes256-gcm@openssh.com aes128-gcm aes256-gcm aes128-ctr aes192-ctr aes256-ctr
ip ssh server algorithm kex curve25519-sha256 curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group14-sha256 diffie-hellman-group16-sha512
ip ssh server algorithm hostkey rsa-sha2-512 rsa-sha2-256 ssh-rsa
ip ssh server algorithm authentication publickey keyboard password
ip ssh server algorithm publickey ssh-rsa ecdsa-sha2-nistp256 ecdsa-sha2-nistp384 ecdsa-sha2-nistp521 ssh-ed25519 x509v3-ecdsa-sha2-nistp256 x509v3-ecdsa-sha2-nistp384 x509v3-ecdsa-sha2-nistp521 rsa-sha2-256 rsa-sha2-512 x509v3-rsa2048-sha256
ip ssh client algorithm mac hmac-sha2-256-etm@openssh.com hmac-sha2-512-etm@openssh.com
ip ssh client algorithm encryption chacha20-poly1305@openssh.com aes128-gcm@openssh.com aes256-gcm@openssh.com aes128-gcm aes256-gcm aes128-ctr aes192-ctr aes256-ctr
ip ssh client algorithm kex curve25519-sha256 curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group14-sha256 diffie-hellman-group16-sha512

We also noticed these scripts that were on the bootflash right around the time of the compromise which we believe were installed by the hackers and believe they have no significance for the operation of the IOS:

487 16624 Nov 16 2023 07:06:25.0000000000 +00:00 /bootflash/y
488 16624 Nov 16 2023 07:09:05.0000000000 +00:00 /bootflash/tclproxy.tcl
490 14874 Nov 16 2023 22:37:44.0000000000 +00:00 /bootflash/IOSmap.tcl
495 42121 Nov 16 2023 21:33:49.0000000000 +00:00 /bootflash/services.list
496 12690 Nov 17 2023 04:55:23.0000000000 +00:00 /bootflash/socks5.tcl
497 1389 Nov 17 2023 04:53:12.0000000000 +00:00 /bootflash/test_server.tcl
498 536 Nov 17 2023 04:53:51.0000000000 +00:00 /bootflash/pkgIndex.tcl

Regards,

Elito

Leo Laohoo · ‎11-29-2023

1. Most important question: Why is GUI enabled to an internet-facing router?

2. The security bulletin Multiple Vulnerabilities in Cisco IOS XE Software Web UI Feature was announced 16 October 2023. It was published in multiple IT-related publications and some major TV/cable networks.

3. Why was the router's IOS not updated to the fixed release first announced in 22 October 2023?

4. Why was the workaround (announced 20 October 2023) not implemented?

Elito Haylett · ‎11-30-2023

Hello Leo,

Thank you very much for your response. These are all great questions you posted and if I was more advance and a dedicated Cisco person where my only job was to manage the day to day operation of the devices then it probably wouldn't have happened. I'm like most people on this forum who doesn't have the expertise or indepth knowledge with Cisco command line and the thousands of features it provides. The reason most of us novices/beginners are on this forum is because we're seeking help or guidance with configurations that's beyond the scope of what we know. This has been a great help for me and I'm sure thousands of others that don't have access to the Cisco expertise that most medium to larger sized organizations do. So I want to thank all who are on here providing us with the answers or guidance we are seeking for the issues we're facing.

1) The reason the GUI was enabled is because it provided a level of configuration and management that novices or beginners like me wouldn't otherwise be able to do through the CLI. I didn't know it wasn't best practice to thave the GUI enabled on an Internet facing router. Most environments that were affected by this bug like me probabIy had the GUI enabled too on their Internet facing router as well or the bug wouldn't have been discovered. I also didn't anticipate there to be an issue with the webui code that would be a backdoor for hackers to get into the system.

Cisco used to have two free pc installable configuration and management application, Cisco Network Assistant and Cisco Configuration Professional NMS (similar to ASDM) for small environments that they did away with. This was ideal for very small network environments or novice network administrators to perform the daily tasks of managing their devices/network. If this was still available the Webui GUI wouldn't need to be installed/enabled on any of the IOS XE platforms and more importantly it would have supported other non IOS XE switches and routers making it hands down the best free tool for managing small environments.

2) I didn't have access to the security bulletins nor was signed up to receive any. I also didn't hear about it from any other source.

3) I didn't know about the release until I found out the VPN tunnel with our other site unexpectedly dropped with no changes in the config, our environment nor issues with the ISP's. It's only when I started troubleshooting and running debugs and started seeing strange tunnels trying to be established with unknown IP addresses is when I knew someone was in the router. I then ran the Cisco CLI Analyzer and discovered the bugs after it conlcuded and made suggestions to address what was found.

Before I knew the device was compromised I thought maybe the IOS could have hit a bug with the crypto engine or something so I posted the issue on the VPN forum seeking help trying to understand what was going on but a day or two after I posted I found out about the hack.

4) I didn't know about it until I ran the Cisco CLI Analyzer.

SSH Configuration in my ISR4000's running IOS-XE after I was hacked

ECH-ISR4431