topic Cisco ISE 3.1 After Patch Update Issue in Network Access Control

Cisco ISE 3.1 After Patch Update Issue

azman.mansor — Sun, 18 Jun 2023 12:18:28 GMT

Dear Cisco Support,

We have 2 Cisco ISE 3.1 appliance.. Recently we have update using patch 7 update.

Once device ISE #1 has been updated we unable to access/view login page but able to ping that IP.

trying to ssh but looks like password does not work anymore to ssh.

We able to access ISE #2 device via browser but the configuration were difference and all setting looks was not there such a policies, configuration and etc.. trying to ssh to ISE #2 devices but same scenario occurred as device ISE #1.

Please help how can we access back our ISE

Thank You

Azman

Re: Cisco ISE 3.1 After Patch Update Issue

Mark Elsen — Sun, 18 Jun 2023 16:55:48 GMT

>...trying to ssh but looks like password does not work anymore to ssh.
- Can you clarify this observation to become exact , meaning do you get an authorization failure or something else ?

Re: Cisco ISE 3.1 After Patch Update Issue

azman.mansor — Sun, 18 Jun 2023 23:53:46 GMT

Hi Marce,

Now i'm able to ssh to the #1 ISE using very very old password i've set long time ago. probably at the initial setup.

But i'm still not able to login using webpage browser. Below were patching i've been updated

Version information of installed applications
---------------------------------------------

Cisco Identity Services Engine
---------------------------------------------
Version : 3.1.0.518
Build Date : Tue Aug 10 04:28:55 2021
Install Date : Tue Jun 28 12:04:42 2022

Cisco Identity Services Engine Patch
---------------------------------------------
Version : 3
Install Date : Tue Jun 28 14:01:42 2022

Cisco Identity Services Engine Patch
---------------------------------------------
Version : 7
Install Date : Sun Jun 18 15:44:49 2023

Re: Cisco ISE 3.1 After Patch Update Issue

azman.mansor — Mon, 19 Jun 2023 01:24:38 GMT

below is the application status running

ISE PROCESS NAME STATE PROCESS ID
--------------------------------------------------------------------
Database Listener running 8669
Database Server running 125 PROCESSES
Application Server running 23513
Profiler Database running 15353
ISE Indexing Engine not running
AD Connector running 25082
M&T Session Database running 4404
M&T Log Processor running 23760
Certificate Authority Service running 24912
EST Service running 42501
SXP Engine Service disabled
TC-NAC Service disabled
PassiveID WMI Service disabled
PassiveID Syslog Service disabled
PassiveID API Service disabled
PassiveID Agent Service disabled
PassiveID Endpoint Service disabled
PassiveID SPAN Service disabled
DHCP Server (dhcpd) disabled
DNS Server (named) disabled
ISE Messaging Service running 11468
ISE API Gateway Database Service running 14084
ISE API Gateway Service running 20317
Segmentation Policy Service disabled
REST Auth Service disabled
SSE Connector disabled
Hermes (pxGrid Cloud Agent) disabled

Re: Cisco ISE 3.1 After Patch Update Issue

Mark Elsen — Mon, 19 Jun 2023 05:22:05 GMT

- Reboot the involved nodes ,

Re: Cisco ISE 3.1 After Patch Update Issue

azman.mansor — Tue, 20 Jun 2023 00:18:55 GMT

Reboot the node won't effect the ISE indexing started again. application start stop didn't work as well. After open TAC with cisco found the ntp server not synchronize with the ISE. ISE engine were not running at all effected the GUI unable to access but still able to ping. We swing to the secondary ISE to allow the operation running again until we fix issue on the primary.

Re: Cisco ISE 3.1 After Patch Update Issue

MaErre21325 — Tue, 20 Jun 2023 10:43:43 GMT

I have a similar problem, after patching to patch 7 i'm no more able neither to ping, ssh or https my standalone Pan.

Do you have any tpis?

Thank you

Re: Cisco ISE 3.1 After Patch Update Issue

Mark Elsen — Tue, 20 Jun 2023 10:46:35 GMT

- @MaErre21325 Rest assured the only thing you can do is reboot the standalone PAN , and check it's (health) state afterwards , the more important question (then) becomes : did you take a configuration backup before applying patch 7 (e.g),

Re: Cisco ISE 3.1 After Patch Update Issue

MaErre21325 — Tue, 20 Jun 2023 10:50:58 GMT

Hi @Mark Elsen,

no, i didn't take a backup...and i'm really worried about recreate every single policy

Re: Cisco ISE 3.1 After Patch Update Issue

Mark Elsen — Tue, 20 Jun 2023 11:29:30 GMT

- @MaErre21325 , I am sorry but you always must make a backup before upgrades ,

Re: Cisco ISE 3.1 After Patch Update Issue

Aref Alsouqi — Tue, 20 Jun 2023 13:24:08 GMT

Is this a VM or a hardware appliance? could you please check the console screen and share what it says?

Re: Cisco ISE 3.1 After Patch Update Issue

MaErre21325 — Tue, 20 Jun 2023 13:57:47 GMT

it's a vm and luckly from console all the services were up, just reloaded it and worked

thank you

Re: Cisco ISE 3.1 After Patch Update Issue

Mark Elsen — Tue, 20 Jun 2023 14:55:13 GMT

- Remember to take backups in the future and also take a (configuration)
backup >now< !

M.

Re: Cisco ISE 3.1 After Patch Update Issue

MaErre21325 — Tue, 20 Jun 2023 15:01:20 GMT

Yes, lesson learned!

Re: Cisco ISE 3.1 After Patch Update Issue

azman.mansor — Wed, 21 Jun 2023 00:02:46 GMT

Hi Cisco support,

just want to share what we have done on the issue after patch update.

I've raise to TAC engineer support and and we troubleshoot the problem together. We found one of the symptom is ntp server unable to reach from the ISE itself. For temporary we swing to the secondary ISE to allow production run until we solve again issue on ntp server configuration. They tried to restart the application service and reboot the appliance. Even they have done it from the root but engine indexing still unable to start because of the ntp server communication.

Today we have a call session with Cisco again to solve issue on the primary. Will share the outcome then

Re: Cisco ISE 3.1 After Patch Update Issue

azman.mansor — Mon, 26 Jun 2023 01:06:32 GMT

Hi,

Just want to share what we have done on primary ISE. We've deregistered the primary one and do again the application stop and start ISE and we rebooted the node. ISE indexing engine able to run. Then next we need to figure out either the routing in the configuration because we still unable to access the GUI but server was pingable.

Re: Cisco ISE 3.1 After Patch Update Issue

hslai — Mon, 26 Jun 2023 15:49:56 GMT

@azman.mansor If SSH working, then routing should not be an issue. I would suggest two things to get more info.

Take a packet capture between ISE and the admin workstation
Move the admin workstation to the same subnet as ISE

Re: Cisco ISE 3.1 After Patch Update Issue

azman.mansor — Tue, 27 Jun 2023 01:43:28 GMT

Hi hslai,

I do the step #2 from your advise seat in the same vlan and its work and able to access the GUI.

I can summarize it I only can access the GUI if I seat in the same vlan and unable to reach the GUI if outside the that vlan.

Since we already narrow down the issue, can give the advise what need to do next to allow us access it from another vlan again.

Thanks

Re: Cisco ISE 3.1 After Patch Update Issue

Aref Alsouqi — Tue, 27 Jun 2023 11:53:19 GMT

There are no restrictions to access ISE CLI or GUI from a different VLAN or a remote subnet, what you need is just a network connectivity that will route the traffic between the endpoint and ISE, and to allow the traffic to pass through if you should have any security device in the middle. Also, ISE allows restricting the accesses to itself based on IP address (Administration > Admin Access > Settings > Access > IP Access) but the default settings allow all IPs. If you could access ISE CLI then you could reset the GUI password, but if you can't I don't think there are any other options rather than trying to recover the administrator password as per the below doc, but tbh I think you are hitting a bug that is causing all these anomalies:

https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/200568-ISE-Password-Recovery-Mechanisms.html#anc5