cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

This community is for technical, feature, configuration and deployment questions.
For production deployment issues, please contact the TAC! We will not comment or assist with your TAC case in these forums.
Please see How to Ask the Community for Help for other best practices.

781
Views
35
Helpful
6
Replies
Alex Pfeil
Rising star

Identity Services Engine Unusable Domain Caused By Windows Domain Controller Reboot

We have had an issue a few times now  so it is becoming an emergency. We had a domain controller reboot cause an issue with the policy node saying that a domain is unusable. We are running ISE 2.4 patch 8. 

Here are some key points:

  1. Domain Controller has updates applied and reboots, comes online within 30 seconds.
  2. Identity Services Engine stops authenticating clients with the following error: 24367  Skipping unusable domain - "domain name", Server not found in Kerberos database. This causes issues for hundreds of users.
  3. For one outage, we rebooted an ISE Policy node and it re-connected to a different domain controller and started working.
  4. For the second outage, authentications started working after approximately 2 hours with no Policy node reboot.

Any information would be appreciated.

2 ACCEPTED SOLUTIONS

Accepted Solutions

I am working with TAC and we already had the debugs enabled on the ISE policy node. We were able to see some good logs. I am waiting to hear back if we have the exact cause. Here are some example logs in case somebody runs into a similar issue in the future.

 

TimeStamp VERBOSE,Transaction Log Number Hidden,AdIdentitySearcher::performSearch: domain=[Some.Domain], base=[dc=Some,dc=Domain,dc=com], filter=[(&(|(objectCategory=person)(objectCategory=computer))(sAMAccountName=Bob))],lsass/server/auth-providers/ad-open-provider/ad_identity_searcher.cpp:324

 

TimeStamp VERBOSE,Transaction Log Number Hidden,LsaDmLdapDirectorySearch: domain=Some.Domain, dn='dc=Some,dc=Domain,dc=com', scope=2, query=(&(|(objectCategory=person)(objectCategory=computer))(sAMAccountName=Bob)),lsass/server/auth-providers/ad-open-provider/lsadm.c:4393

 

TimeStamp VERBOSE,Transaction Log Number Hidden,LsaDmLdapDirectorySearch: attempt=1, error=40286(LW_ERROR_LDAP_SERVER_DOWN),lsass/server/auth-providers/ad-open-provider/lsadm.c:4420

 

TimeStamp VERBOSE,Transaction Log Number Hidden,LocatorLookup(0x7f5091806e90): dc=Domain-Controller1, x.x.x.x-IPA,netlogon/service_locator/service_locator.c:318

 

TimeStamp VERBOSE,Transaction Log Number Hidden,LocatorLookup(0x7f509176c190): dc=Domain-Controller2, x.x.x.x-IPB,netlogon/service_locator/service_locator.c:318

 

TimeStamp ERROR  ,Transaction Log Number Hidden,LsaDmConnectDomain: domain Some.Domain is offline,lsass/server/auth-providers/ad-open-provider/lsadm.c:5011

View solution in original post

We ended up finding that it was a bug. It can happen when multiple domain controllers are rebooted at the same time.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvp73385

We verified that this error was being thrown: LW_ERROR_KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN.

The issue is resolved in ISE 2.4 patch 10.

View solution in original post

6 REPLIES 6
Jason Kunst
Cisco Employee

I have a TAC case open. I know that this is not Cisco TAC. I always share issues that I am having here as a way to let other people know about the issue and see if they have had it before. I do not see anything wrong with that. This was also informational and you definitely did not have the right solution.

Alex thank you, it would be good to state what your intent is so we can close out the solution. If its just information sharing then please post as a document or blog and not something needing an actual solution. We are trying to manage and make sure everything is covered.

I am working with TAC and we already had the debugs enabled on the ISE policy node. We were able to see some good logs. I am waiting to hear back if we have the exact cause. Here are some example logs in case somebody runs into a similar issue in the future.

 

TimeStamp VERBOSE,Transaction Log Number Hidden,AdIdentitySearcher::performSearch: domain=[Some.Domain], base=[dc=Some,dc=Domain,dc=com], filter=[(&(|(objectCategory=person)(objectCategory=computer))(sAMAccountName=Bob))],lsass/server/auth-providers/ad-open-provider/ad_identity_searcher.cpp:324

 

TimeStamp VERBOSE,Transaction Log Number Hidden,LsaDmLdapDirectorySearch: domain=Some.Domain, dn='dc=Some,dc=Domain,dc=com', scope=2, query=(&(|(objectCategory=person)(objectCategory=computer))(sAMAccountName=Bob)),lsass/server/auth-providers/ad-open-provider/lsadm.c:4393

 

TimeStamp VERBOSE,Transaction Log Number Hidden,LsaDmLdapDirectorySearch: attempt=1, error=40286(LW_ERROR_LDAP_SERVER_DOWN),lsass/server/auth-providers/ad-open-provider/lsadm.c:4420

 

TimeStamp VERBOSE,Transaction Log Number Hidden,LocatorLookup(0x7f5091806e90): dc=Domain-Controller1, x.x.x.x-IPA,netlogon/service_locator/service_locator.c:318

 

TimeStamp VERBOSE,Transaction Log Number Hidden,LocatorLookup(0x7f509176c190): dc=Domain-Controller2, x.x.x.x-IPB,netlogon/service_locator/service_locator.c:318

 

TimeStamp ERROR  ,Transaction Log Number Hidden,LsaDmConnectDomain: domain Some.Domain is offline,lsass/server/auth-providers/ad-open-provider/lsadm.c:5011

View solution in original post

We ended up finding that it was a bug. It can happen when multiple domain controllers are rebooted at the same time.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvp73385

We verified that this error was being thrown: LW_ERROR_KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN.

The issue is resolved in ISE 2.4 patch 10.

View solution in original post

Jason,

I have found multiple discussions in the forums that are bugs that have helped me in the past. The purpose of my post was to get as many eyes on my problem as I possibly could. Sometimes, a person will reply immediately with the fix. Other times, it could be that TAC will be the final solution. And now, this thread will help somebody in the future.

 

Content for Community-Ad