pwd cache not allowed btwn 1 psn in 1 city and a psn in diff city

[NatalieNDeGennaro72244]

I have small ISE cluster - one primary PAN and one primary PSN/MNT in one city and another secondary PAN and secondary PSN?MNT in a different city with much distance between. ISE cluster is running version 2.7.0.356, Installed patches 2,3,5,7,9  all 4 nodes are vm's.

I have an engineer that logs on to firewall on this cluster from admin ip address (he is on the primary node at this point), then changes context to one of the 8 firewalls in the FX (Its a 9300 FP running ASA code). When he types in context command, once he hits enter, it switches him to the secondary node (where the pswd cache is not there) and he hangs and his session is broken - every time).

What is a fix for this where the engineer will be able to change context and still use the pswd cache so his session won't be broken any longer? He says this has been going on for about a year. I do not know what ISE version was running when the behavior changed (he has always done this).

Is there a way to force ISE to stay at whatever node he logged in at, so that he can change contexts without changing to the secondary node and yet the secondary is still available if needed? I read where the pswd cache is not replicated to other nodes from where it is generated. 

I do have PAN failover turned on, so if anything happens then the AAA will keep working.

Any ideas?

I am going to upgrade to ISE 3.2 in the next 30 days, will I have the same problem on 3.2 or will this issue magically disappear after upgrading to 3.2? Our upgrade will be clean install then read in backups.

Thank you in advance for any help you can give. 

here is example of on of the logs:

13013 - Received TACACS+ Authentication START Request

15049 - Evaluating Policy Group

15008 - Evaluating Service Selection Policy

15048 - Queried PIP - TACACS.User

15048 - Queried PIP - DEVICE.Owners

15048 - Queried PIP - DEVICE.Device Type

15041 - Evaluating Identity Policy

15048 - Queried PIP - Network Access.UserName

22072 - Selected identity source sequence - Test

15013 - Selected Identity Source - Two_Factor

13044 - TACACS+ will use the password prompt returned by the identity store

13015 - Returned TACACS+ Authentication Reply

13014 - Received TACACS+ Authentication CONTINUE Request

15041 - Evaluating Identity Policy

22019 - Identity Policy was evaluated before; Identity Sequence continuing

15013 - Selected Identity Source - Two_Factor

24638 - Passcode cache is not enabled in the RADIUS token identity store configuration - Two_Factor

24609 - RADIUS token identity store is authenticating against the primary server - Two_Factor

11100 - RADIUS-Client about to send request - ( port = 1812 )

11101 - RADIUS-Client received response

24612 - Authentication against the RADIUS token server succeeded

24623 - User record was cached

24638 - Passcode cache is not enabled in the RADIUS token identity store configuration

22037 - Authentication Passed

15036 - Evaluating Authorization Policy

15048 - Queried PIP - IdentityGroup.Name

13015 - Returned TACACS+ Authentication Reply

[NatalieNDeGennaro72244]

added this log for the context change:

13005 Received TACACS+ Authorization Request
15049 Evaluating Policy Group
15008 Evaluating Service Selection Policy
15048 Queried PIP - TACACS.User
15048 Queried PIP - DEVICE.Owners
15048 Queried PIP - DEVICE.Device Type
15041 Evaluating Identity Policy
15048 Queried PIP - Network Access.UserName
22072 Selected identity source sequence - Test
15013 Selected Identity Source - Two_Factor
24629 Searching for user in the RADIUS token identity store - Two_Factor
24626 User record not found in the cache - Two_Factor
15013 Selected Identity Source - Internal Users
24210 Looking up User in Internal Users IDStore
24212 Found User in Internal Users IDStore
24629 Searching for user in the RADIUS token identity store - Two_Factor
24626 User record not found in the cache - Two_Factor
22016 Identity sequence completed iterating the IDStores
22056 Subject not found in the applicable identity store(s)
22058 The advanced option that is configured for an unknown user is used
22061 The 'Reject' advanced option is configured in case of a failed authentication request

[Arne Bier]

Hi @NatalieNDeGennaro72244 

You wrote:

"Is there a way to force ISE to stay at whatever node he logged in at, so that he can change contexts without changing to the secondary node and yet the secondary is still available if needed? I read where the pswd cache is not replicated to other nodes from where it is generated." 

I can't see your ASA configuration, but I would assume that if you were to align all the Primary/Secondary ISE server IPs in all the contexts to be the same (i.e. Primary is PSN1, Secondary is PSN2) then that should take care of the consistency.  I am no ASA expert, but it appears that one configures the AAA commands per context. Is that right?

I am not aware of any ISE RADIUS Token Server Password Cache synchronisation between PSNs.

In my experience with Token Servers, I always wait for a new token to be generated and I never re-use a token. Perhaps I have been missing a trick, but I thought that OTP (one time PIN) means "one time" only ?