Can't access ISE secondary node via CLI / GUI after joining deployment

[Ryan H]

Hey gang!  I'm running into a strange deployment issue in my lab.  Using ISE 3.2 Patch 4.  The primary server is running fine as PAN/PSN/MNT.  When I try to add a secondary server to the deployment, it is added successfully and the status of the new server shows up as green on the deployment page.  However, after that point I can no longer login to the secondary via GUI or CLI.  When I try via GUI, there is no web page presented and I just get TCP RST from the server.  When I try to log into the CLI, it accepts the credentials but immediately logs me out.  It also will not process AAA requests from NADs.  I've tried rebuilding the secondary and repeating the whole process, and got the same results again.  Any ideas?  Thanks!

[Greg Gibbs]

You might be hitting this bug - https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwi33361

Your best bet would be to open a TAC case to confirm if this is the issue and, if so, see if they have a hotfix available (since there is no patch available yet with the bug fix).

[jyla]

This actually turned out to be the issue. So after regenerating the kong certificates, we could login to the ISE CLI/console again and the webgui started up.

[Ryan H]

Thanks Greg.  TAC support would be tricky as this is a lab environment. However, the description of this bug doesn't quite fit... it suggests the GUI is accessible (which in my case it is not,) and also the specific error wording for the bug, "Failed to connect to ConfD: Connection refused" suggests a flat-out rejection of the SSH connect attempt.  In my case the SSH/console session connects fine, but it is immediately disconnected after successful authentication.  Interestingly, if I intentionally supply the wrong password upon connection attempt, I'm re-prompted to put in the pw multiple times.  It's only when I put in the correct pw that the session is established and then immediately disconnected again. 

[jyla]

Hi Ryan,

Sorry for waking up this old thread - but we might have hit the same issue as you, and I wonder if you ever got it fixed (and found the rootcause) ?

We upgraded from 3.2p6 to 3.3p4.

Issue shows up clearly using securecrt as ssh client, here the ssh session is disconnected when trying to login, but you can actually see the reason for the disconnect stated:

I booted up a centos rescue image and mounted the ISE disk to try to see what happens.

Looking at the /etc/passwd file I can see that our static user (acsadmin) has the UID 500

But the homedirectory is for some reason assigned to a user with UID 1000, and the same ownership is set for all files within the folder:

The UID 1000 is non-existing on the unix side of this deployment, and it prevents the user from changing its work directory to its homedir if I understand correctly.

I have TAC involved in troubleshooting, to find the reason why the update would change the ownership of the folder and content.
I hope there is a log somewhere detailing the upgrade/patch process/progress which can hopefully give us the cause. We are hesitant to continue upgrading other deployments until then.

Any inputs are more than welcome.

[Marcelo Morais]

Hi @jyla ,

 please use the Backup and Restore upgrade method, i.e. install an ISE 3.3 P4 from scratch and Restore the ISE 3.2 P6 backup on it.

Hope this helps !!!

[jyla]

Hi Marcelo,

Thanks for the feedback. Yes, that probably ends up being the solution - but I surely hope Cisco/TAC are interested in finding the rootcause to prevent other users getting hits by the same issue

The deployment is only used for tacacs and that part is still working.
So the only issue we see is that we cannot login to ISE, so we have time to wait for Cisco to find the rootcause.

[Marcelo Morais]

Hi @jyla ,

 what you said makes sense to me ... let's try to "dig a little deeper" ...

 This kind of weird stuff reminds me of ISE 2.7 P8, a very good patch that fixes a bizarre Field Notice (Field Notice: FN74005 - Identity Services Engine: Java Heap Size May Significantly Impact System Performance - Software Upgrade Recommended), but at the same time has issues whenever you upgrade from ISE 2.7 P1 or P2 to it, and that's why it became a Deferred Release !!!

You said that you upgrade from ISE 3.2 P6 to ISE 3.3 P4: 

• Did you notice the issue when you reach to ISE 3.3 or only when you update to ISE 3.3 P4 ?
• Have you tried updating first from ISE 3.2 P6 to P7 and then to ISE 3.3 P4 to check if the issue exists ?

Regards