PSN not Authenticating Radius and TACACS

[mpbaker82]

I have a ISE distributed deployment:

1 PAN (admin, monitor, and psn) geographic location 1 (primary)

1 SAN (admin, monitor, and psn) geographic location 2 (backup to primary)

5 PSN ( psn persona only) geographic location 3-5 (the only service running on these psn's is the "enable device admin". Everything else is deselected

No node groups are created

I'm able to point any device on my network regardless of the geographic location to my primary PAN and the device authenticates using the polices i have setup (radius/tacacs). However, when i attempt to point a device to its local ISE psn node, it fails. In my Admin>System>Deployment screen, all my nodes give a status of connected. 

I've tried to use some logs to verify the sync, and to see if the psn's are even seeing the authentication attempt. but im not seeing no such thing. its possible im not even looking at the right log. 

Any ideas as to why? I'm hoping the great people on this forum can be of more assistance then the TAC support team. I can honesty say I've have three separate and individual experiences with TAC and its much less desirable.

Thanks :)

Michael

[paul]

So you are testing with TACACS only right now?  Your title says RADIUS and TACACS but you said you have turned off RADIUS and only have Device Admin enabled under Policy Service.  You can go to the debug menu and do packet captures on any of the nodes to verify packets are being received on the PSN you are testing against.  You could also force a sync on the PSNs from the deployment screen, but if you are testing TACACS you should see attempts even if the PSNs are out of sync.  Also make sure you are looking at the TACACS live logs and not the RADIUS live logs.

You can also run TACACS debugs on the network device.

[mpbaker82]

Paul,

Thanks for your help

Which service runs Radius?

Enable Device Admin = TACACS

 ? = Radius

My secondary admin node has every service enabled that the primary admin node does but when i point my radius device to that san, it still doesn't authenticate. 

As i mention before, we have several geographically separated areas which hold their own ISE. Devices in those locations will point their their respective ISE nodes. It just so happens that two of those locations run the PAN and SAN.

[paul]

Session Services = RADIUS

Profiling goes with Session Services if you are planning to use the profiling feature for MAB devices

[mpbaker82]

Hmm, thats odd because we are only using ISE for network devices. i.e. switches, routers, firewalls, etc. We dont have end users devices authenticating with ISE. Its more of a network device management over the oob connection.

I dont have session services enabled on on the pan ise node and im able to point a test switch to the pan and it authenticates with both tacacs and radius on the pan node. if i take that same test switch and point it to the san or even a psn, it fails. no workie.

I dont have node groups enabled because im not looking for the failover between psn's. In my network devices, i have the local psn node 1st, then my pan as 2nd should the local ise node fail or become disconnected.

I mean we are talking about the same session service setting right? Admin> system > deployment > Policy > Session Service

thank you, you've been a great help :)

[hslai]

This is happening even after you resolved Non Responsive Cisco ISE. Correct?!

Like Paul mentioned, you may turn some debug on the network device (NAD) to check the communication. Or, use TCPdump or similar packet capture tool to verify packet flows between NAD and PSN. Once that verified, we may go further in debugging on the ISE side.

Sorry to hear your less-than-desirable experience with TAC. If this happens again, please ask to speak to the duty TAC manager.

RADIUS is part of the core of ISE policy services so it's probably got turned ON as soon as one of the policy services enabled.

[mpbaker82]

Thank you hslai, I will def be contacting the duty tac manager for sure. I didnt know there was such a thing but now that i do, i hope to get some more help moving forward.

im unable to provide any debug information due to the sensitivity of my environment but ill follow your instructions and see what else i can figure out. Ill keep this thread posted.

Thanks :)

[Damien Miller]

So I have no reason to believe that this is the cause of your issue, but just a general comment on the design.  With the PAN and MNT's collocated in a hybrid deployment, you should really only have sessions services enabled on the 5 dedicated PSNs, not all 7 nodes. 

[mpbaker82]

Thanks damien

However, Im not quite following your comment. however, im very interested. The ISE deployment is new to me so i'm still learning as i go. trial by fire i guess.

are you saying that the pan and san would not have radius services enabled? assuming from a previous comment that session services enable radius, while device admin services enable tacacs.

[pan]

Please provide few detail. I will try to help you here.

Output of "show version" command

What do you seen in tacacs live logs?
Operations>TACACS> live logs> try to click on details report (a small icon under details column) Try to authentication and then see the logs for that authentication.

[mpbaker82]

Cisco Identity Services Engine

Version = 2.4.0.357

Build Date = Mar 22 2018

Install Date Oct 23 2018

Cisco Identity Services Engine Patch

Version 3

Install Date Oct 24 2018

As for the details report, i know what your talking about but there isnt any logs coming through if I point the radius device to the psn. If i point it to the pan, i can authenticate all day for both radius and tacacs. Im wondering if the psn nodes truly replicated to the pan. The node statue says complete but it doesn't make sense that they would preform like this if they did.

is there a log i can check out to verify the completion status of the SAN and PSNs? if im remembering right, we used show logging application.log tail command on each of the nodes to see the status. does this ring true with anyone?

I know the devices are configured correctly for authentication, they wouldn't authenticate with the pan if they weren't. So to make the change to the san or one of the psn's, i just go in and point the device at the new aaa server.

[pan]

Are you facing issue with RADIUS or TACACS?

1> Enable debugs. Administration> system > logging> debug log configuration > select the psn and enable runtime-aaa to debug.

2> Point one of the device to the affected PSN and then open two SSH session to the affect PSN and tail following two logs then authenticate:

show logging application prrt-server.log tail

show logging application localStore/iseLocalStore.log tail

Also take tcpdump operations>diagnostic tools> tcpdump> select the psn and then give "ip host x.x.x.x" in filter and see what do you see there.

If possible share the logs of the above two logs.

[mpbaker82]

Im facing an issue with both if i point the device to one of the PSN or the SAN node. I have no issues if i point the device to the pan node. I can authenticate using both tacacs and radius all day. the problem occurs only when i point the device to its local ise node.

thank you again for the help

ill run the commands as mention. However, i am unable to share any log information. if there is something of particular interest, i can filter just that and remove any thing that needs to be removed.

[pan]

I have updated my reply so please check again.

In logs check what you see for the authentication request, it will show why it is failing. For TACACS you need device admin license.

[hslai]

pan brought out an interesting point on Device Admin licensing. That in ISE 2.4 is done per ISE node that enabled for device admin.

If you are going to watch iseLocalStore.log, please the option of "Local Logging" for Passed Authentications.