TACACS Failure

[s1nsp4wn]

ISE 2.4 Patch 9

I've successully used tacacs with no issues until all of a sudden I can to work one day and tacacs wouldn't work on any device I tried accessing and logs no longer appeared in operations>tacacs live logs. The only changes made prior to me noticing this was my primary psn was out of sync when I came in, so I just manually synced it back to my deployment AND I installed new product licenses because my existing ones were about to expire in a few days. I deleted the old licenses after installing the new.

Would any of the above cause TACACS to stop working? I did a packet capture and plain as day saw my network devices still sending tacacs requests, but not getting any response from ISE. I ended up having to reload the PSN to get it working again but my concern is finding out what caused the issue. Would a license update or deletion (even though new ones are installed) to the device admin license cause it? Would a sync issue break it?

[s1nsp4wn]

Does this ring any bells?
https://www.cisco.com/c/en/us/support/docs/security/identity-services-engine/214136-recovering-disabled-and-out-of-sync-node.html

[Damien Miller · Accepted solution]

There is a TACACS bug in 2.4 p9 that can cause all radius and tacacs logs to stop working, but it does not cause authentication to fail. It is a regression of a previous issue reintroduced in patch 8/9. There is a hotfix I am running with a client that does fix it. Pending roll in to a public future patch.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvq73457

Not saying this is the issue you are facing, but it could be just one of the issues. Without logs it is hard to say what the issue is on failing authentication. If you haven't done so already, you should really open a TAC case.

[s1nsp4wn]

Hi Damien,

Got a tac case open.  In my case authentication, authorization, and accounting stopped working.  You didn't even see failures pop up in the logs.  That's when I got real nervous.  Then, I did debugs on various network devices that used ISE for tacacs and they all showed the same.  TACACS request sent, no response back.  So in short, I never even got as far as authentication failure because no logs showed and no response ever came back to the devices.

[Damien Miller]

Point the engineer to that bug, the hotfix they gave us has resolved the logging issue at least. It had been earmarked for patch 10 but until it releases there is no guarantee the fix will be included. Takes some time to QA fixes and commit them. Could possibly provide a quick fix for part of your issue if you are hitting it.

[hslai]

ISE 2.4 Patch 10 has been posted on Sept 27, 2019 and the bug fix for CSCvq73457 is included.

[Colby LeMaire]

My guess would be that during the sync, the services were restarted and the TACACS+ service didn't start up completely or got hung.  Next time (hope it doesn't happen again), try to check to see if all of the services are running before reloading.  At this point, all you can really do is work with TAC to see if they can find anything in the past system logs.

[s1nsp4wn]

Are you referring to the ise application service? If so that was running before the reload. Not sure if it was while out of sync.