Tacacs authentication denied after Cisco ISE failover

[jeremytetart]

Hello guys,

Could you help me to resolve a problem please.

I use Cisco ISE cluster (version 2.3.0) with Active directory to verify which users are authorized to manage Cisco switch & Router.
However, when an automatic failover occur, I'm not able to connect to my devices with tacacs protocol

The error show :

login as: <user_x>
Using keyboard-interactive authentication.
Password:
Access denied

I didn't find anything in log of ISE after the failover. Before that, the authentication working fine.

Service on cluster :
TACACS1(Master) has : PAM, MNT & PSN => 192.168.0.1
TACACS2(Standby) has : PAM, MNT & PSN => 192.168.0.2

- Share key on both switch and ISE is OK
- Communication between AD and ISE is OK

Switch config :

switch1#
aaa group server tacacs+ TACACS_ADMIN
 server name TACACS1
 server name TACACS2
!
aaa authentication login default local
aaa authentication login AUTHENT group TACACS_ADMIN local
aaa authorization exec default group TACACS_ADMIN none
aaa accounting exec default start-stop group TACACS_ADMIN
!
tacacs server TACACS1
address ipv4 192.168.0.1
key 7 095841xxxxxxxxxxxxxxxx
tacacs server TACACS2
address ipv4 192.168.0.2
key 7 150604xxxxxxxxxxxxxxxx
!
line vty 0 4
login authentication AUTHENT
transport input ssh
line vty 5 15
login authentication AUTHENT
transport input ssh

I did few tests on switch1 before launch manual failover

Test on switch1 to reach TACACS1 (192.168.0.1) before failover

switch1#test aaa group tacacs+ <user account> <pwd> legacy
Attempting authentication test to server-group tacacs+ using tacacs+
User was successfully authenticated.

switch1#test aaa group TACACS_ADMIN server 192.168.0.1<user account> <pwd> legacy
Attempting authentication test to server-group TACACS_ADMIN using tacacs+
User was successfully authenticated.

switch1#ping 192.168.0.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

switch1#telnet 192.0.0.1 49
Trying 192.168.0.1, 49 ... Open

Result:
 - Ping OK
 - Port 49 open
 - Authentication user OK with TACACS1 & AD (192.168.0.1)

Now try the same test on switch1 to reach TACACS2 before failover

switch1#ping 192.168.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

swrica05b#telnet 192.168.0.2 49
Trying 192.168.0.2, 49 ... Open
[Connection to 192.168.0.2 closed by foreign host]

switch1#test aaa group TACACS_ADMIN server 192.168.0.2 <user account> <pwd> legacy
Attempting authentication test to server-group TACACS_ADMIN using tacacs+
User was successfully authenticated.

Result :
- Ping OK
 - Port 49 open but closed immediatly (maybe because it's a standby cluster for PAM)
 - Authentication user OK with TACACS2 & AD (192.168.0.2)

Now, I stop TACACS1 and launch new test. Still Access Denied.
I don't see anything in Operation => Tacacs => live logs on TACACS2
I launch debug on the switch. In attachment the result (Tacacs Log.txt).

 - debug tacacs
 - debug aaa authentication

switch1#sh tacacs
Tacacs+ Server -  public  :
               Server name: TACACS1
            Server address: 192.168.0.1
               Server port: 49
              Socket opens:         24
             Socket closes:         24
             Socket aborts:          0
             Socket errors:          0
           Socket Timeouts:          0
   Failed Connect Attempts:          0
        Total Packets Sent:         26
        Total Packets Recv:         21

Tacacs+ Server -  public  :
               Server name: TACACS2
            Server address: 192.168.0.2
               Server port: 49
              Socket opens:          5
             Socket closes:          5
             Socket aborts:          0
             Socket errors:          0
           Socket Timeouts:          0
   Failed Connect Attempts:          0
        Total Packets Sent:          5
        Total Packets Recv:          0

I launch a tcpdump and I see this

Sequence 1 src: switch1 dst: 192.168.0.2 protocol:tcp paquet : [syn]
Sequence 2 src: 192.168.0.2 dst: switch1 protocol:tcp paquet : [syn, ack]
Sequence 3 src: switch1 dst: 192.168.0.2 protocol:tcp paquet : [ack]
Sequence 4 src: 192.168.0.2 dst: switch1 protocol:tcp paquet : [fin, ack]

Sequence 5 src: switch1 dst: 192.168.0.2 protocol:tacacs+ paquet : [authentication]
Sequence 6 src: 192.168.0.2 dst: switch1 protocol:tcp paquet : [RST]

I think TACACS2 close the TCP/IP session before the tacacs authentication frame is sent. After that, a TCP/IP Reset occur.

When I restart TACACS1 (which have secondary administration role now), I'm able to connect to my device. So I think I have a misconfiguration or problem on TACACS2.

Someone can help me please.

Regards.

[Arne Bier]

dumb question perhaps, but does your license include TACACS for both servers (both UDI's)?  This can be a common thing to forget when creating the license - you have to include both UDI's when creating the license key.

[jeremytetart]

Hi Arme Bier,

The cluster is 90 days evaluation so all licences are god. (See attachment).

I'm still investigating.

[jeremytetart · Accepted solution]

Update: Broken and re-create the cluster has resolved my issue.

Regards.

[ajc]

I am having the exact same issue. Looks like we have a bug here because I have valid base and tacacs license with entries for both ISE nodes (I have a similar deployment like yours). Authentication works on PrimaryMNT/SecPAN/PSN but not the same on PrimaryPAN/SecMNT/PSN.

I will take a look again on the licenses just to be safe but I suspect that breaking the deployment and rebuilding is the only option. I will keep it post.

[ajc]

Hi Jeremy,

After multiple changes on the deployment I got the same result as you. We need to break the deployment and rebuild it in order to make it work including redundancy scenarios when one of the PSN goes completely down. I tested each node individually and also shutting down each one at the time.

Proceeding to talk to cisco about this.

[jeremytetart]

Hi Abraham,

Any news about Cisco ?

[ajc]

Looks like we are hitting the following CSCvi18260 based on the information provided but NOT confirmed yet. I will keep you posted.

[jeremytetart]

I can't view the bug and the solution :/

See the screenshoot in attachment.

[Sri Harsha Dasari]

If you are seeing resets on pcaps from ISE server, it might be Device Administration service is not enabled on the PSN.
In you case, you could have checked if device admin service is enabled under deployments page on you failover device.