topic Re: Unable to ssh or access ISE GUI intermittently in Network Access Control

Unable to ssh or access ISE GUI intermittently

Madura Malwatte — Tue, 21 Aug 2018 03:29:54 GMT

I am facing this issue on my new ISE deployment where intermittently I am unable to access the ISE nodes. These are VM's deployed using OVA.

During the problem state if I try to ssh, I can enter the login password but after hitting enter, the cursor moves to a new line and does not show any output (just stuck there). During this time I cannot access the GUI either, but ping works fine.

I think if I leave it for a while I have get access again, but I end up just reseting the VM. I checked the I/O write bandwidth and its showing 105MB (this is after the reset, would the historical info be lost after reset?). This is happening on multiple nodes.

What can try to troubleshoot this problem?

Re: Unable to ssh or access ISE GUI intermittently

anthonylofreso — Tue, 21 Aug 2018 11:34:40 GMT

What happens if you try to access the ISE VM via the hypervisor console?

Re: Unable to ssh or access ISE GUI intermittently

Madura Malwatte — Tue, 21 Aug 2018 12:38:07 GMT

Hi Anthony,

When trying to access via the hypervisor console I get the same issue - login prompt, after entering password and hit enter, cursor blinks on a new line and nothing happens.

Re: Unable to ssh or access ISE GUI intermittently

anthonylofreso — Tue, 21 Aug 2018 12:53:30 GMT

Are you seeing any alarms for failed login attempts or admin account locked on the dashboard page (screen cap attached)?

and are there any lockout policies enabled under:

Administration > System > Admin Access > Lock/Suspend Settings
Administration > System > Admin Access > Account Disable Policy

Reason I ask... We're running ISE 2.2 Patch 5, and my Admin account will continually get disabled. I receive syslog emails that say account was disabled due to failed login attempts. but when I look at the details of the message within the ISE gui, they read that the account was disabled due to inactivity.

There's a TAC case open. No resolution yet...

Also, this only happens with the local Admin account. I'm always able to login to the GUI with my AD credentials.

Re: Unable to ssh or access ISE GUI intermittently

Madura Malwatte — Tue, 21 Aug 2018 13:16:32 GMT

No, I don't see any such alarms. But I now have disabled the lock/suspend setting. Let's see if this helps.

Re: Unable to ssh or access ISE GUI intermittently

howon — Tue, 21 Aug 2018 14:58:03 GMT

Since this is a new deployment, I suspect that it may be due to the VM environment. Make sure to allocate enough CPU/RAM for the VM and confirm that the CPU/RAM resources are dedicated for it. For a small deployment you need to allocate 12vCPU and 16GB RAM. Also, make sure the snapshot is disabled for the VM.

Re: Unable to ssh or access ISE GUI intermittently

anthonylofreso — Tue, 21 Aug 2018 16:50:33 GMT

you might turn on Alarm Notifications here: Administration > System > settings > Alarm Settings > Alarm Notification (Make sure your SMTP Server settings are appropriately configured)

I've found more details come across in the emails sometimes.

Re: Unable to ssh or access ISE GUI intermittently

Madura Malwatte — Wed, 22 Aug 2018 00:07:03 GMT

Hi howon,

These VM's were deployed via OVA and I didn't get the option to change the resource allocation. And it's thick provisioned.

Re: Unable to ssh or access ISE GUI intermittently

Madura Malwatte — Wed, 22 Aug 2018 00:19:43 GMT

Hi howon,

You may be right and could be related to snapshots. I found a similar thread, and I think there is netapp or similar running on top.

https://community.cisco.com/t5/identity-services-engine-ise/ise-2-3-hangs-every-4-hrs/td-p/3543969

Re: Unable to ssh or access ISE GUI intermittently

Madura Malwatte — Thu, 23 Aug 2018 03:37:47 GMT

So I have some good news. We did have netapp running which was taking backups (using snapshots) of the ISE VM's. We disabled the ISE VM's from the netapp backup and so far so good, haven't lost gui or console/ssh access. Thanks for the suggestion Howon.