topic Re: Tacacs authentication denied after Cisco ISE failover in Network Access Control

Tacacs authentication denied after Cisco ISE failover

jeremytetart — Fri, 21 Feb 2020 18:38:50 GMT

Hello guys,

Could you help me to resolve a problem please.

I use Cisco ISE cluster (version 2.3.0) with Active directory to verify which users are authorized to manage Cisco switch & Router.
However, when an automatic failover occur, I'm not able to connect to my devices with tacacs protocol

The error show :

login as: <user_x>
Using keyboard-interactive authentication.
Password:
Access denied

I didn't find anything in log of ISE after the failover. Before that, the authentication working fine.

Service on cluster :
TACACS1(Master) has : PAM, MNT & PSN => 192.168.0.1
TACACS2(Standby) has : PAM, MNT & PSN => 192.168.0.2

- Share key on both switch and ISE is OK
- Communication between AD and ISE is OK

Switch config :

switch1#
aaa group server tacacs+ TACACS_ADMIN
server name TACACS1
server name TACACS2
!
aaa authentication login default local
aaa authentication login AUTHENT group TACACS_ADMIN local
aaa authorization exec default group TACACS_ADMIN none
aaa accounting exec default start-stop group TACACS_ADMIN
!
tacacs server TACACS1
address ipv4 192.168.0.1
key 7 095841xxxxxxxxxxxxxxxx
tacacs server TACACS2
address ipv4 192.168.0.2
key 7 150604xxxxxxxxxxxxxxxx
!
line vty 0 4
login authentication AUTHENT
transport input ssh
line vty 5 15
login authentication AUTHENT
transport input ssh

I did few tests on switch1 before launch manual failover

Test on switch1 to reach TACACS1 (192.168.0.1) before failover

switch1#test aaa group tacacs+ <user account> <pwd> legacy
Attempting authentication test to server-group tacacs+ using tacacs+
User was successfully authenticated.

switch1#test aaa group TACACS_ADMIN server 192.168.0.1<user account> <pwd> legacy
Attempting authentication test to server-group TACACS_ADMIN using tacacs+
User was successfully authenticated.

switch1#ping 192.168.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

switch1#telnet 192.0.0.1 49
Trying 192.168.0.1, 49 ... Open

Result:
- Ping OK
- Port 49 open
- Authentication user OK with TACACS1 & AD (192.168.0.1)

Now try the same test on switch1 to reach TACACS2 before failover

switch1#ping 192.168.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

swrica05b#telnet 192.168.0.2 49
Trying 192.168.0.2, 49 ... Open
[Connection to 192.168.0.2 closed by foreign host]

switch1#test aaa group TACACS_ADMIN server 192.168.0.2 <user account> <pwd> legacy
Attempting authentication test to server-group TACACS_ADMIN using tacacs+
User was successfully authenticated.

Result :
- Ping OK
- Port 49 open but closed immediatly (maybe because it's a standby cluster for PAM)
- Authentication user OK with TACACS2 & AD (192.168.0.2)

Now, I stop TACACS1 and launch new test. Still Access Denied.
I don't see anything in Operation => Tacacs => live logs on TACACS2
I launch debug on the switch. In attachment the result (Tacacs Log.txt).

- debug tacacs
- debug aaa authentication

switch1#sh tacacs
Tacacs+ Server - public :
               Server name: TACACS1
            Server address: 192.168.0.1
               Server port: 49
              Socket opens:         24
             Socket closes:         24
             Socket aborts:          0
             Socket errors:          0
           Socket Timeouts:          0
   Failed Connect Attempts:          0
        Total Packets Sent:         26
        Total Packets Recv:         21

Tacacs+ Server - public :
               Server name: TACACS2
            Server address: 192.168.0.2
               Server port: 49
              Socket opens:          5
             Socket closes:          5
             Socket aborts:          0
             Socket errors:          0
           Socket Timeouts:          0
   Failed Connect Attempts:          0
        Total Packets Sent:          5
        Total Packets Recv:          0

I launch a tcpdump and I see this

Sequence 1 src: switch1 dst: 192.168.0.2 protocol:tcp paquet : [syn]
Sequence 2 src: 192.168.0.2 dst: switch1 protocol:tcp paquet : [syn, ack]
Sequence 3 src: switch1 dst: 192.168.0.2 protocol:tcp paquet : [ack]
Sequence 4 src: 192.168.0.2 dst: switch1 protocol:tcp paquet : [fin, ack]

Sequence 5 src: switch1 dst: 192.168.0.2 protocol:tacacs+ paquet : [authentication]
Sequence 6 src: 192.168.0.2 dst: switch1 protocol:tcp paquet : [RST]

I think TACACS2 close the TCP/IP session before the tacacs authentication frame is sent. After that, a TCP/IP Reset occur.

When I restart TACACS1 (which have secondary administration role now), I'm able to connect to my device. So I think I have a misconfiguration or problem on TACACS2.

Someone can help me please.

Regards.

Re: Tacacs authentication denied after Cisco ISE failover

Arne Bier — Tue, 14 Nov 2017 00:22:40 GMT

dumb question perhaps, but does your license include TACACS for both servers (both UDI's)? This can be a common thing to forget when creating the license - you have to include both UDI's when creating the license key.

Re: Tacacs authentication denied after Cisco ISE failover

jeremytetart — Tue, 14 Nov 2017 08:35:34 GMT

Hi Arme Bier,

The cluster is 90 days evaluation so all licences are god. (See attachment).

I'm still investigating.

Re: Tacacs authentication denied after Cisco ISE failover

jeremytetart — Tue, 14 Nov 2017 14:27:36 GMT

Update: Broken and re-create the cluster has resolved my issue.

Regards.

Re: Tacacs authentication denied after Cisco ISE failover

ajc — Wed, 09 May 2018 20:48:21 GMT

I am having the exact same issue. Looks like we have a bug here because I have valid base and tacacs license with entries for both ISE nodes (I have a similar deployment like yours). Authentication works on PrimaryMNT/SecPAN/PSN but not the same on PrimaryPAN/SecMNT/PSN.

I will take a look again on the licenses just to be safe but I suspect that breaking the deployment and rebuilding is the only option. I will keep it post.

Re: Tacacs authentication denied after Cisco ISE failover

ajc — Thu, 10 May 2018 15:27:12 GMT

Hi Jeremy,

After multiple changes on the deployment I got the same result as you. We need to break the deployment and rebuild it in order to make it work including redundancy scenarios when one of the PSN goes completely down. I tested each node individually and also shutting down each one at the time.

Proceeding to talk to cisco about this.

Re: Tacacs authentication denied after Cisco ISE failover

jeremytetart — Fri, 18 May 2018 17:27:22 GMT

Hi Abraham,

Any news about Cisco ?

Re: Tacacs authentication denied after Cisco ISE failover

ajc — Fri, 18 May 2018 18:09:59 GMT

Looks like we are hitting the following CSCvi18260 based on the information provided but NOT confirmed yet. I will keep you posted.

Re: Tacacs authentication denied after Cisco ISE failover

jeremytetart — Sat, 19 May 2018 14:50:57 GMT

I can't view the bug and the solution 😕

See the screenshoot in attachment.

Re: Tacacs authentication denied after Cisco ISE failover

Sri Harsha Dasari — Wed, 06 Dec 2023 22:04:32 GMT

If you are seeing resets on pcaps from ISE server, it might be Device Administration service is not enabled on the PSN.
In you case, you could have checked if device admin service is enabled under deployments page on you failover device.