ISE 3.1 - Can't Access GUI - ISE API Gateway Service: not running

[mattw]

Hello,

Yesterday I attempted to upgrade a distributed deployment from ISE 3.0 patch 3 to ISE 3.1 patch 1 using the automated GUI method. The cluster is not yet in production so I'm trying to avoid a TAC case if I can fix this myself.

The primary PAN upgraded first and appeared to successfully upgrade to 3.1 and apply patch 1 but the other nodes did nothing.

I couldn't log into the GUI despite the Application Server process being in the running state.

Turns out that the two ISE API Gateway services were down and the appliance was not listening ports tcp/80 or tcp/443 (nothing listed in "show ports").

Some research uncovered bug https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa08018 whereby the GUI is inaccessible with ISE 3.1 unpatched or with patch 1 if IPv6 is globally disabled. The issue is apparently fixed in patch 2.

The fix is to re-enable IPv6 globally, stop and restart the application. This got me into the primary PAN GUI but all the other nodes were kind of stuck. The PPAN thought they were upgrading but they where doing nothing.

I had to break the cluster into multiple standalones, upgrade them manually and attempt to re-create the cluster.

5 out of the 8 nodes are back in the cluster but 3 are not playing ball.

IPv6 is enabled so the bug is no longer in play, the Application Server process is running but the ISE API Gateway Service service is in the "not running" state. When stopping and starting the application you see the API gateway service go into the "Initializing" state but it never goes into "running". After a while it drops into the "not running" state.

I've tried stopping and starting ISE - no joy.

I've tried stopping and starting ISE in safe mode - no joy.

I've tried a reboot - no joy and no errors on the console.

Any ideas on where to go next before I reimage the appliances?

Thanks in advance,

Matt.

[Arne Bier · Accepted solution]

sounds like a production cluster to me - best to get TAC involved.

[mattw]

It's not yet in production which is why I wanted to see if it was fixable without TAC as there was no real urgency. I've reimaged the server now so we're all good but I was curious to know if there might be something else that can be done.

Thanks,

Matt.

[Yordan1]

hi

did you find the solution? i have the same issue , i have to deregister the ise, did application reset-config, since then the gui is no longer working and ISE API Gateway Service not running.... 

[ditti.tech]

Hello, I have the same issue, any tips from TAC to resolve the issue ?

I also open a TAC case but it could speed it up a little.

Thanks

[mattw]

Hi,
You're not going to like this: I had to rebuild the entire deployment. In hindsight I wish I had done that earlier rather than faffing about resetting the application and trying other things to restore service. For me it wasn't too bad as it was a new deployment but I had to re-image all 8 nodes!
Good luck!

[J. H.]

We have the same exact problem upgrading from 3.0 to 3.1. But this cluster is in production. I've enabled IPv6 but it didn't help.

All 4 nodes except 1 of 2 PSN's works now. I'm not going to use Upgrade via GUI again. Had to do upgrade manually for 2 of the nodes. I did everything recommended, I even ran the Upgrade Readiness Tool. There was no way to know that this would happen.

ISE API Gateway Service is initializing when I do a application start ise, but then says "not running" after a while.
I don't know which logs to check (there's no kong.log which somebody referred to). 

This PSN node is still responding to RADIUS requests without being part of the cluster (fortunately!!).

I just wish to know which logs I can check as for why the API Gateway Service isn't starting.

[Greg Gibbs]

If the node is out of sync, this could be a much bigger issue than the API Gateway Service not starting. With a Production issue, don't waste time trying to troubleshoot this yourself. Open a TAC case and get them investigating the issue!

[Yordan1]

i think we are hitiing this bug - https://bst.cisco.com/bugsearch/bug/CSCwf50584