Identity Services Engine Unusable Domain Caused By Windows Domain Controller Reboot

[Alex Pfeil]

We have had an issue a few times now  so it is becoming an emergency. We had a domain controller reboot cause an issue with the policy node saying that a domain is unusable. We are running ISE 2.4 patch 8. 

Here are some key points:

1. Domain Controller has updates applied and reboots, comes online within 30 seconds.
2. Identity Services Engine stops authenticating clients with the following error: 24367  Skipping unusable domain - "domain name", Server not found in Kerberos database. This causes issues for hundreds of users.
3. For one outage, we rebooted an ISE Policy node and it re-connected to a different domain controller and started working.
4. For the second outage, authentications started working after approximately 2 hours with no Policy node reboot.

Any information would be appreciated.

[Jason Kunst]

Please see this post for general guidance on using community,
https://community.cisco.com/t5/security-documents/how-to-ask-the-community-for-help/ta-p/3704356

[Alex Pfeil]

I have a TAC case open. I know that this is not Cisco TAC. I always share issues that I am having here as a way to let other people know about the issue and see if they have had it before. I do not see anything wrong with that. This was also informational and you definitely did not have the right solution.

[Jason Kunst]

Alex thank you, it would be good to state what your intent is so we can close out the solution. If its just information sharing then please post as a document or blog and not something needing an actual solution. We are trying to manage and make sure everything is covered.

[Alex Pfeil · Accepted solution]

I am working with TAC and we already had the debugs enabled on the ISE policy node. We were able to see some good logs. I am waiting to hear back if we have the exact cause. Here are some example logs in case somebody runs into a similar issue in the future.

TimeStamp VERBOSE,Transaction Log Number Hidden,AdIdentitySearcher::performSearch: domain=[Some.Domain], base=[dc=Some,dc=Domain,dc=com], filter=[(&(|(objectCategory=person)(objectCategory=computer))(sAMAccountName=Bob))],lsass/server/auth-providers/ad-open-provider/ad_identity_searcher.cpp:324

TimeStamp VERBOSE,Transaction Log Number Hidden,LsaDmLdapDirectorySearch: domain=Some.Domain, dn='dc=Some,dc=Domain,dc=com', scope=2, query=(&(|(objectCategory=person)(objectCategory=computer))(sAMAccountName=Bob)),lsass/server/auth-providers/ad-open-provider/lsadm.c:4393

TimeStamp VERBOSE,Transaction Log Number Hidden,LsaDmLdapDirectorySearch: attempt=1, error=40286(LW_ERROR_LDAP_SERVER_DOWN),lsass/server/auth-providers/ad-open-provider/lsadm.c:4420

TimeStamp VERBOSE,Transaction Log Number Hidden,LocatorLookup(0x7f5091806e90): dc=Domain-Controller1, x.x.x.x-IPA,netlogon/service_locator/service_locator.c:318

TimeStamp VERBOSE,Transaction Log Number Hidden,LocatorLookup(0x7f509176c190): dc=Domain-Controller2, x.x.x.x-IPB,netlogon/service_locator/service_locator.c:318

TimeStamp ERROR  ,Transaction Log Number Hidden,LsaDmConnectDomain: domain Some.Domain is offline,lsass/server/auth-providers/ad-open-provider/lsadm.c:5011

[Alex Pfeil · Accepted solution]

We ended up finding that it was a bug. It can happen when multiple domain controllers are rebooted at the same time.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvp73385

We verified that this error was being thrown: LW_ERROR_KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN.

The issue is resolved in ISE 2.4 patch 10.

[Alex Pfeil]

Jason,

I have found multiple discussions in the forums that are bugs that have helped me in the past. The purpose of my post was to get as many eyes on my problem as I possibly could. Sometimes, a person will reply immediately with the fix. Other times, it could be that TAC will be the final solution. And now, this thread will help somebody in the future.