ISE-PIC web interface refusing connection

[Scott123]

I have a standalone Cisco ISE-PIC virtual machine and as of a few days ago whenever I browse to the web interface (https://isepic_IP_FQDN) I get "ISEPIC_FQDN refused to connect".  I have tried the following:

• confirmed DNS resolves the FQDN correctly and I get the same issue using the IP address
• can ping and SSH to the cli from the pc that I am browsing from
• confirmed IPv6 is enabled globally and in Gigabit Ethernet 0 in the running config
• confirmed ise application services all have status as "running", have stopped/started ise application (application stop/start ise)
• reloaded isepic from cli
• checked tcp ports listening on isepic and tcp/443, tcp/80 are not listed (see output in attached file)
• ran wireshark on pc that I am browsing to isepic on and see the SYN packet going to isepic and isepic returning a RST packet.
• disabled ipv6 in running config, stop/start application ise, enabled ipv6 in running config, stop/start application ise. tcp/443 and 80 still not appearing in "show port" and RST packet still being returned by isepic.

ISEPIC version: 3.2.0.542, patch 5 and 6 installed, running on virtual machine, single ISE-PIC node.  Otherwise the ISEPIC seems to be working. 

I wonder if the web server is not starting but do not know how to check. I can see from our WSA cli > isedata that it is connected to the ISE-PIC and receiving username-ip mappings, so ISE-PIC seems to otherwise be working.

Anyone have any ideas on what may be causing this, further troubleshooting ideas and if someone can post the output of "show ports" so I can compare that would be helpful.

I have logged TAC case SR698359589 and am waiting for an update.

Thanks

[ahollifield]

Why a single node? What about HA?  Did a certificate expire?  what is the use-case for ISE-PIC in the first place?

[Scott123]

I am using ISE-PIC to monitor Active Directory user logins and pass username-to-IP mappings the Cisco WSA. No certificates expired.

[ahollifield]

are snapshots enabled on the VM? Is it within ISE node spec for CPU and RAM requirements? Is the disk thick or thin provisioned?

[Scott123]

The ISE-PIC VM has 8 CPUs, 16 GB RAM, 400 GB disk (thin provisioned). Snapshots are enabled.

[ahollifield]

What version? All of that is your issue. Snapshots MUST be disabled. You do not have enough resources assigned

https://cs.co/ise-scale

[Scott123]

Thanks ahollifield. I don't understand understand how it can be a resource or snapshot issue. The CPU, RAM and disk utilisation is very low. However I increased CPU to 16 and RAM to 64GB. Utilisation graphs attached. The spikes are during reload after making the changes.

There are no snapshots taken for this VM. Our sys admin told me that to disable snapshots there is a setting that limits the max number of VMs to zero but that will make no difference is we have taken no snapshots anyway. Can you please explain why limiting snapshots to zero will help?

[ahollifield]

The oracle database within ISE does not support snapshots. When the hypervisor pauses the disk it will corrupt the database and lead to the exact symptoms you are experiencing.

[Aref Alsouqi]

I totally agree with Adam, snapshots are not supported with ISE.

"If the Snapshot feature is enabled on the VM, it might corrupt the VM configuration. If this issue occurs, you might have to reimage the VM and disable VM snapshot."

Cisco Identity Services Engine Installation Guide, Release 3.2 - Cisco Secured Network Server Series Appliances and Virtual Machine Requirements [Cisco Identity Services Engine] - Cisco

[Scott123]

Thanks guys, that is good information.  The ISE-PIC vm (RHEL8, compatibility ESXi 70. U2 and later) is on VMware vCenter 7.0.3T build 24322018.

I have taken a snapshot of the ISE-PIC VM in the past so perhaps that has caused the web interface service to fail. Interesting that the ISE-PIC is still receiving user logon events from the AD agents and passing username-ip mappings to the WSA proxy.  The only use case for ISE-PIC in my environment is to detect user logons to Microsoft Active Directory and pass username-IP mappings to Cisco WSA web proxy.

I am planning to restore the vm from a backup when I know the web interface worked then disable snapshots for the vm by setting vm > Options > Advanced > General > Configuration Parameters > snapshot.MaxSnapshots = 0 . Do you think this setting will do the trick?  

I the web interface does not come up then I will be build a new ISE-PIC vm.

Cheers

[Aref Alsouqi]

Yeah I think that's the way to turn the snapshots off.