Unable to ssh or access ISE GUI intermittently

[Madura Malwatte]

I am facing this issue on my new ISE deployment where intermittently I am unable to access the ISE nodes. These are VM's deployed using OVA.

During the problem state if I try to ssh, I can enter the login password but after hitting enter, the cursor moves to a new line and does not show any output (just stuck there). During this time I cannot access the GUI either, but ping works fine.

I think if I leave it for a while I have get access again, but I end up just reseting the VM. I checked the I/O write bandwidth and its showing 105MB (this is after the reset, would the historical info be lost after reset?). This is happening on multiple nodes.

What can try to troubleshoot this problem?

[anthonylofreso]

What happens if you try to access the ISE VM via the hypervisor console?

[Madura Malwatte]

Hi Anthony,

When trying to access via the hypervisor console I get the same issue - login prompt, after entering password and hit enter, cursor blinks on a new line and nothing happens.

[anthonylofreso]

Are you seeing any alarms for failed login attempts or admin account locked on the dashboard page (screen cap attached)?

and are there any lockout policies enabled under:

• Administration > System > Admin Access > Lock/Suspend Settings
• Administration > System > Admin Access > Account Disable Policy

Reason I ask... We're running ISE 2.2 Patch 5, and my Admin account will continually get disabled. I receive syslog emails that say account was disabled due to failed login attempts. but when I look at the details of the message within the ISE gui, they read that the account was disabled due to inactivity.

There's a TAC case open. No resolution yet...

Also, this only happens with the local Admin account. I'm always able to login to the GUI with my AD credentials.

[Madura Malwatte]

No, I don't see any such alarms. But I now have disabled the lock/suspend setting. Let's see if this helps.

[anthonylofreso]

you might turn on Alarm Notifications here: Administration > System > settings > Alarm Settings > Alarm Notification (Make sure your SMTP Server settings are appropriately configured)

I've found more details come across in the emails sometimes.

[howon · Accepted solution]

Since this is a new deployment, I suspect that it may be due to the VM environment. Make sure to allocate enough CPU/RAM for the VM and confirm that the CPU/RAM resources are dedicated for it. For a small deployment you need to allocate 12vCPU and 16GB RAM. Also, make sure the snapshot is disabled for the VM.

[Madura Malwatte]

Hi howon,

These VM's were deployed via OVA and I didn't get the option to change the resource allocation. And it's thick provisioned.

[Madura Malwatte]

Hi howon,

You may be right and could be related to snapshots. I found a similar thread, and I think there is netapp or similar running on top.

https://community.cisco.com/t5/identity-services-engine-ise/ise-2-3-hangs-every-4-hrs/td-p/3543969

[Madura Malwatte]

So I have some good news. We did have netapp running which was taking backups (using snapshots) of the ISE VM's. We disabled the ISE VM's from the netapp backup and so far so good, haven't lost gui or console/ssh access. Thanks for the suggestion Howon.