Hello,
Yesterday I attempted to upgrade a distributed deployment from ISE 3.0 patch 3 to ISE 3.1 patch 1 using the automated GUI method. The cluster is not yet in production so I'm trying to avoid a TAC case if I can fix this myself.
The primary PAN upgraded first and appeared to successfully upgrade to 3.1 and apply patch 1 but the other nodes did nothing.
I couldn't log into the GUI despite the Application Server process being in the running state.
Turns out that the two ISE API Gateway services were down and the appliance was not listening ports tcp/80 or tcp/443 (nothing listed in "show ports").
Some research uncovered bug https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwa08018 whereby the GUI is inaccessible with ISE 3.1 unpatched or with patch 1 if IPv6 is globally disabled. The issue is apparently fixed in patch 2.
The fix is to re-enable IPv6 globally, stop and restart the application. This got me into the primary PAN GUI but all the other nodes were kind of stuck. The PPAN thought they were upgrading but they where doing nothing.
I had to break the cluster into multiple standalones, upgrade them manually and attempt to re-create the cluster.
5 out of the 8 nodes are back in the cluster but 3 are not playing ball.
IPv6 is enabled so the bug is no longer in play, the Application Server process is running but the ISE API Gateway Service service is in the "not running" state. When stopping and starting the application you see the API gateway service go into the "Initializing" state but it never goes into "running". After a while it drops into the "not running" state.
I've tried stopping and starting ISE - no joy.
I've tried stopping and starting ISE in safe mode - no joy.
I've tried a reboot - no joy and no errors on the console.
Any ideas on where to go next before I reimage the appliances?
Thanks in advance,
Matt.
