ISE 2.4 can not send operation to node errors

[Jay Stants]

Hey everyone, 

I've been running into an odd issue lately with a single node lab setup (ise 2.4 patch 14). 

To start with i'm trying to rejoin the node to AD, but keep getting failures that the operation can not be sent to the node. 

Looking at DNS from the node I don't see any issues with SRV records and since this node was joined at 1 point to AD i'm not sure why it's failing now. the below results return both DC's in my lab aswell Time is in sync on the Ise node and across the DC's and other systems in my environment. 

SRV query from node

When I attempt to troubleshoot [Operations -> Troubleshoot -> Download Logs]

I get an error after selecting the node "Node is not reachable Please check node status

I'm baffled as to what may be causing this. i've tried restarting ise application and even a reboot with the same results. 

anyone else out there see or have this issue?

[Milos_Jovanovic]

Hi @Jay Stants,

Have you tried confirming that you actually have connectivity between ISE and AD? You can run TCP Dump from ISE (Operations / Diagnostic Tools), and filter it based on IP address of your AD server.

If you do see traffic being exchanged there, you can try looking at log file related to AD (Operations  / Download Logs / Debug Logs / ad_agent.

BR,

Milos

[Jay Stants]

Hey @Milos_Jovanovic

So from what i can tell in the reports the node is actively connecting to AD in the background to refresh TGT tickets and is discovering the AD servers but trying to enable debugs or even traces on Active Directory within Ise doesn't help. i'm not able to actually download the logs. when i try i get the above error that was stated "the node is not reachable" which leads me to think there's some sort of disconnect between the Gui and the backend underlying system but i'm not sure how to actually troubleshoot that or even better fix it.

AD Connector Operations Report

[Milos_Jovanovic]

Hi @Jay Stants,

Going again through your post I see what you mean. I would advise reboot of the device, but you already did that. Since you are experiencing issues with AD connectivity and also unable to download logs at the same time, I believe you do experience some ystem issue with your deployment.

Regarding downloading logs, you could use CLI command 'show logging application ad_agent.log' which will show you logs relevant to AD connectivity. But, as I said, I don't think you have issue only with AD section.

If you have backup, I would advise to install new node and perform restore.

BR,

Milos

[Jay Stants]

Hey @Milos_Jovanovic ,

Here is something interesting I've noticed. I stood up a new 2.4 node (unpatched) and was able to join AD without issue. I then applied patch 14 to the node and that completely broke the AD integration and the node become un-joined from AD after the patch was installed. The node could not rejoin AD and presented the same issue as above. If i roll back the node to pre 14 patch, after that completes and restarts the node rejoins AD and all is fine. (very odd)

now back to the original node, I attached the output from the logging command, i added a tail so that i could capture as i was attempting to join the node back to AD. Take a peak and if you have any suggestions let me know. i did pick out a few things but i really don't know how to possibly remove any previous info that the node has stored in config files or cache to make the node think it's never been joined to AD post applying patch 14. 

[Milos_Jovanovic]

Hi @Jay Stants,

Maybe you can try with patch 13, just to test that scenario as well? Usually, patches are fixing things, but, sometimes, it happens that they break things as well. Although, I have patch 14 installed on multiple deployments, all of which are AD integrated, and I'm not facing any issue.

Unfortunatelly, this file is not very readable and I can't conclude anything from it There is no command similar to what you see in GUI, so nothing to correlate with. Should you continue to face the issue, your way forward must be Cisco TAC.

BR,

Milos

[Jay Stants]

Thanks for trying to help, I did open a TAC case and once I have some sort of indication on what the issue is I will post incase someone else possibly runs into this issue as well. 

[jacob_parker]

Hello Jay, 

i stumpled over the same issue as you.
We use an WMware environment with 2 nodes. 

1. Software Version: 2.4.0.357 
2. Patches: 5,7,10,11,12

After installing patch 13 i lost my deployment and the backup node is not more reachable. 
And in addition i'am not able to start 'Download Logs'. Same behaviour as your case. 

Did you get some information from the Cisco TAC engineers? 
I also opened a Cisco TAC case, but Cisco is at the moment not able to repair respectivelly allocate the root cause.

If you have some updates about this, i would appriciate your feedback.   

Kind Regards
Jacob

[pzkqx6000]

We have a similar issue and it looked someway related to a auto signed admin certificate expired. After renewed it (ise service will be restarted) all started working again.

ISE 2.4 version.

[Yordan1]

hi

did you find solution on this issue. i have the same problem with 2.4 / patch 12. already tried all announced solutions, no luck

br

Yordan

[Jay Stants · Accepted solution]

The only solution that worked was to redeploy all of the ISE nodes and rebuild the environment.