ISE Secondary Node PSN did not take over when Primary failed

[Danny Dulin]

We had a situation where the Primary node became "unjoined" to the domain. This caused a widespread VPN connectivity outage because ISE couldn't verify credentials because it couldn't talk to Active Directory.

Active Directory still had the machine account for ISE, but ISE dashboard listed primary as unjoined.

We had to delete the machine account in AD then rejoin the primary to get everything working.

QUESTION: Why did our secondary node did not service Policy requests even though it is enabled?

[Dustin Anderson]

So, the thing to look at is the primary didn't fail in terms of ISE. System was online, just getting rejects form AD. ISE is usually active/active, so it's up the the NAD to pick what node to auth against. Since your primary was active and taking auth, no reason for the NADs to change to secondary.

Hope that helps.

[Danny Dulin]

Dustin this is so helpful. Thank you!!

That's a little frustrating. Almost what's the point in having multiple ones.

Is there no way to signal NADs to use another Node?

[Dustin Anderson]

unfortunately, no. Until the NAD marks it as down, it'll usually stick to one. So long as it responds to RADIUS, it'll say as active to a NAD. I think they would have to make some way for certain failures to have ISE stop responding to requests. But as it is it just responds with a deny. 

And yes, I've had similar, just we had one failing and a reboot fixed it, but devices kept trying to use it.

[ajc]

I think that if you have a load balancer for the PSN's and that LB checking for PSN health via AD authentication (LB health monitor config) then you could have that failing PSN declared as inactive in the LB so all the traffic then redirected to the working ones.

[Dustin Anderson]

Yeah, I was thinking something similar, but would need a RADIUS test set up so if it fails it would remove from the pool. But not every company has a load balancer, so not a fix for everyone. Also a lot of reconfig to change over to one.