Cisco ISE - TACACS stopped working?

[dazza_johnson]

Hey guys, I already have a TAC case open but sometimes I solve issues quicker here......

We have an ISE 2.1 deployment that has been ticking over nicely for about 12 months when TACACS (on 2 nodes) just stopped working - no changes were made at this time. Has anyone experienced this before and if so what was the fix?

The node is listening on TCP port 49, a 3 way TCP handshake is established but then gets torn down? Same result on IOS, Nexus, WLCs etc.

Anyone seen this before?

[Cory Peterson]

One thing I have been doing if I have issues that are only on some of the PSNs if a full-sync. I have seen this fix many random issues on the PSNs. 

Keep in mind doing a full-sync will cause a restart of the services on the Node that is being synced to. So you want to do one at a time and during a maintenance window if required.

To do a full-sync:

Navigate to "Administration" --> "Deployment"

Click the Checkbox next to the node with the problem

Click "Syncup" on the top toolbar

[Arne Bier]

ha ha. Yes this reminds me... Sadly, even in ISE 2.4 there are times when a PSN just stop syncing for no apparent reason and then you have to manually sync as @Cory Peterson mentioned.  Just the other day I was banging away at the keyboard writing a really cool Policy Set but the testing was just not working out as I expected.  I hammered away another hour or so and finally came to the conclusion that I am going insane because nothing I did had any effect.  LIGHTBULB MOMENT - the PSN has gone AWOL again.  Right! Do a manual sync!  Issue sorted. 

How can you tell if the PSN has gone AWOL?  You monitor the message count.  Even though the icons are green to tell you that the node is (apparently) well, the counters are still working and tell you that the PSN has not acknowledged them. 

This kind of situation ought not to happen very often, especially if the user is not doing anything to provoke it.  But be warned:  it can strike when you least suspect it.

[dazza_johnson]

Thanks guys, I appreciate you taking the time to comment. We have already tried a full sync of the PSNs and a reboot prior to opening the TAC case - neither made a difference......

[Arne Bier]

just a thought - you said the TCP 3-way handshake was established and then torn down.  Are there any firewalls in between the PSN's and the NAD?  Idle timers on a stateful firewall can do that.  Do you have a tcpdump of such an event?

[williams_t82]

This fixed the issue for me. Thanks.

[thomas · Accepted solution]

Please call TAC for production issues like this.