ISE v1.2 patch 5 PSN down, endpoint identity deleted

[Sheraz.Salim]

Kindly please refer to the diagram. I shall make it simple and clear.

ISE version 1.2 patch  5

3xPOL (2xVirtual Appliances)

1 MON

1 Admin

Since Janauray the 8th we encounter issues with ISE. the issue encounter were Endpoint end profiling the devices as (Cisco 1140 AP)  but the devices is a Motorola hand held running Windows CE. Also the Motorola mac address deleted from endpoint identity every 4 to 6 hours and we have to manually put the mac address to start authentication in order to work.

we open a cisco with TAC. and TAC advise there is a bug in software and it needs to be upgrade to patch 17 or either upgrade the ISE to 1.4 as it more stable version than version 2.

later few days after one of the node POL3 (in cisco language PSN) went down. and one of our SSID Wi-Fi clinets lost the connection as they were unable to authenticate (WLC security are set to POL3 with ISE group created AD-HOC Network devices with MAC filtering.) to fix the problem we change the WLC security AAA to POL1 (PSN) to make it work. since than its working.

later the next day an other POL2 (flapping up/down) an other SSID (DATA) clients start reporting connection drop. we change again the WLC AAA authentication ip in direction to set it to POL1 since than its working fine.

now out of 3 only 1 POL is working and three SSID end clinet get authenciated by this POL ip address.

we reached to cisco for help they looked into this and said the POL node are not syn. therefore the ISE needs a reboot to fix this problem. we the management decided if this require a reboot to fix the problem theye why not we upgrade the ISE to version 1.4. Cisco TAC mention upgrade can take up to 3 to 4 hours or might more depends on the server. now we want to go for upgrade but our structure of network is complex that we do not want to lost the ise for 3 to 4 hours. we are an hospital and all patient checking devices/Doctor computers/hand held devices/Records are authenticated through ISE. we using ISE mainly for wireless.

Now this is the background story. now I have question can we reload the POL nodes 1 by 1 to fix this problem. I also notice there is another work around we are running another ISE node of another trust hospital in our data centre. it is a virtual appliances   (ise-psn.web.com) in our controller (WLC) one of our main hospital SSID authentication setting AAA two ip address are POL1 and next is ip address of ISE-PSN.WEB.COM if we reload our ise and on wlc we point the ip address of ISE-PSN.WEB.COM will this keep the SSID client keep connected.

please advise we are in desperate situation where we need some direction to minumis the down time of our patient critical application which are connected to wireless.

[nspasov · Accepted solution]

Hi there and so sorry you are in such a crappy situation. It is no fun!

To answer your questions:

#1. I would definitely recommend upgrading to a later version of ISE or at the very minimum get your current version on the latest patch!

#2. Yes, you can reload the PSNs one at a time with zero to no service interruptions. Your WLC will detect that your first PSN is down and then move to the second one that is configured under the SSID > AAA Servers. It is very important that your PSNs are in a Node Group. That way if PSN-1 goes down, any sessions that were in the middle of the AAA process will get absorbed by another node in your Node Group. If the PSNs are not in a node group then clients that were trying to authenticate to the network at the time of the reload will have to retry again.

#3. Once clients are authenticated and authorized their traffic no longer flows through the PSNs. Thus, reloading the PSN will not affect clients that are already on the network. However, if a client has to re-auth (due to inactivity, idle or re-auth timer) then a working PSN will be needed, otherwise the AAA session will fail. 

#4. You can definitely configure a third PSN under your SSID and utilize your PSN that is located in that other hospital. As long as that node is in the same ISE deployment and is in-sync with the PAN then you should be good to go. You can quickly test this by creating a temporary SSID > Make that PSN its primary Radius server > Test it with a test machine. 

I hope this helps!

Thank you for rating helpful posts!

[Sheraz.Salim]

Thank you for your advice.

[nspasov]

You are welcome! Good Luck!