ISE 2.4 Patch 4 Warning + Release Notes Feedback

[Damien Miller]

Hello All,

I had the opportunity to hit this Severity 1 Catastrophic bug while installing patch 4 last night and it got me thinking.  

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvm93698/

1. Why is an available patch still available for download, and not deferred, if it has a catastrophic bug open with no fix/work around. The conditions being a multiforest AD isn't some rare one off environment. The number of support cases attached to it can attest to this. 
2. Why do the release notes open caveats always trail one patch behind.  ex. When patch 5 releases, we will see patch 4 open caveats updated. It seems that serious and high impact bugs are sometimes available on the bug tracker or hidden from public view. It's almost like those of us in the field need to always trail a patch behind to be safe. It would be nice to at least see the more serious issues added/updated in more real time. 

[hslai · Accepted solution]

On 1, we did not get any report CSCvm93698 impacting ISE 2.4 Patch 4 until a couple of days ago. And, no issue in backing out the patch.

On 2, I will forward your comments to our teams to review. However, we do not generally update the release notes until new patch releases.

[Arne Bier]

@Damien Miller - sorry to hear about that - those are not fun times.  You'd think AD software in ISE should be rock solid by now so that we can concentrate on fighting bugs in NEW FEATURES :-)

I recently did two separate customer deployments where I built ISE 2.4 from scratch and in one case patched straight to 4. And in the other case I went from 3 to 4.  In both cases I have AD integration. I am not sure I understand this bug because I have not noticed any issues. Can you please expand on the exact trigger here?

e.g. In one customer case I have one join point, which reveals 4 domains.  I whitelist one of the 4 domains.  We are able to authenticate just fine against the whitelisted domain.

I don't have more than one join point - and I have not used scopes.

Would this be an issue if I used LDAP against an AD domain?

[Damien Miller]

To be honest I'm not entirely sure either. LDAP appeared unaffected, I was still able to log in to the GUI via the LDAP connector. When I went to go lookup my same user account, which is synced between AD and LDAP, the AD connector test utility if would fail. Within the whitelisted domains section of the AD connector there are 9 domains across 4 forests.

Test Username : abc123
ISE NODE :
Scope : Default_Scope
Instance : alphabet-world

Authentication Result : FAILED
Error : Identity Not Found; Some Of The Domains Were Not Available

Processing Steps:
05:14:24:269: Resolving Identity - abc123
05:14:24:270: Search For Matching Accounts At Join Point - world.abc.alphabet.com
05:14:24:270: LDAP Search In Forest Failed - abc.alphabet.com,ERROR_NO_SUCH_DOMAIN
05:14:24:270: Skipping Unusable Domain - def.local,Domain Trust Is One-way
Trimmed
05:14:24:271: Identity Resolution Detected No Matching Account
05:14:24:271: Identity Resolution Failed - ERROR_NO_SUCH_USER_SOME_DOMAINS_NOT_AVAILABLE


If I was to append the domain, ex. world\abc123 the lookup would work. It would also work fine if I was to use abc123@world.abc.alphabet.com. If I did not specify the domain it would always fail. The failed lookups weren't even being logged to the ad_agent.log file, just a silent failure.

[Arne Bier]

I will admit that most of the time I can't tell the difference between a domain and a forest (other than the technical definition) so when I first get introduced to a customer network, and they say, our users live in domain  mycompany.com, then all I do is create a joinpoint at mycompany.com and ensure that I only whitelist mycompany.com once ISE discovers all the other "linked/trusted" domains.  But whether or not I have joined a forest or not, I have no idea. I would like to understand that stuff a bit better.

There is one guy at Cisco (Chris Murray, Technical Leader) who gave a CiscoLive preso on the AD Connector and I think he also created the AD stuff back in ACS days - as far as engineering goes, for me it stands out as better than anything else in the code base (it's been very stable in the past, well documented and the debugging in the GUI is top stuff).  This guy might be able to explain this nicely. 

Pity that he doesn't appear on the forums :-(

His session is BRKSEC-2132

[anthonylofreso]

This behavior definitely needs to change. If there is a bug above a certain threshold, the release notes need to be updated with that information... 

Big Bold Red banner

If you look back to CSCvj53801, that memory leak existed in two patches. The delta between introducing the memory leak, and the fixed patch-9 was nearly 100 days. Even if Cisco found the leak 60 days after releasing patch 7, that left customers un-aware for over a month unless they did some Sherlock level sleuthing to find the bug ID.

What is the reasoning on holding Release Note revisions until a new version is published?

[walwar]

I also have the same problem with patch 4.

First time I hit the bug, I thought it was because I went directly to patch 4 so I rolled back and installed the patches one by one but still I had the error in patch 4 so I rolled back to patch 3 and opened a TAC case, hopefully they will find the root cause.