FMC 6.6.3 - ISE PIC 2.6 Integration. Working but FMC Health ISE Connectivity critical Error

[Erwan LE BIHAN]

Hi all.

I've opened a Case with TAC (# 691307739) and, as It seems to be handled at turtle speed (20+ days) , I'm wondering if someone will have a good hint - or perhaps a solution ?

(TLDR:

My FMC is displaying a cosmetic error about ISE integration. This FMC 6.6 was connected to a Full ISE 3.0 trial and working, but as ISE 3.0 is not supported on FMC 6.6 compatibility matrix and I had no license, I had to go back to ISE PIC 2.6 and since this reinstall the error is displayed, but all is working.)

Here we have:

1 Firepower Management Center v6.6.3, handling two FTD 6.6.3 devices (1x FPR1120, 1xFTD5508x)

1 ISE PIC 2.6Patch5 - (Well, now, it's an ISE PIC 2.6... More on that later)

MS AD (4 DCs) ,MS DNS, Microsoft CA PKI (Single level) for internal cert, and a DUOProxy for Radius VPN Auth.

* Internal Root CA Cert was imported in both ISE and FMC

* No problem for creating the realm, using LDAPS, and connect to the 4 DCs. LDAP Download is working.

* No problem for connecting ISE to FMC. Test is ok, Active sessions are correctly imported from ISE and displayed on dashboards.

but I've an error in Health, ISE Connection Status Monitor Display : "Check connectivity to ISE".

So far, what was tested by TAC:

* Discard all certs and subscriptions, use ISE PIC as a root CA, create FMC cert on ISE PIC Root CA and use all certs belonging to ISE PIC CA instead of AD PKI CA:

=> Same, integration works, test is ok, but error is still displayed.

=> I prefer to have certs belonging to our PKI, so I reverted 48 Hours later to a new batch of certs: no change (test ok, integration ok, error displayed).

This FMC was integrated multiple time with different ISE versions (PIC 2.4, full 2.4, 2.7,3.0 and now PIC 2.6) all sharing the same name and ip address.

(I had to try different versions because I was trying to use ISE Radius / Posture to get DUO working for MFA VPN access, and now it's working using DUOProxy directly as a radius server connected to AD and DUO so no need for a full ISE to get Radius server). 

If I use expert mode on FMC and start ADI with --debug, I can clearly see ADI connecting without problems to all 4 DCs, then to ISE-VM, all SSL Handshakes are ok, pxgrid_connection_connect ok, subscription &_on_connect called, callback fired...

Communications using PXGrid and then:

May 19 09:21:51  SF-IMS[32627]: [32677] ADI:adi.ISEConnection [INFO] adi.cpp:623:HandleLog(): ISEConnection queries find the following capability states: [sessionDirectory: 1, endpointProfileMeta
Data: 0, securityGroupTagMetaData: 0, EPS: 0, ANC: 0, SXP: 0]
May 19 09:21:51  SF-IMS[32627]: [32677] ADI:adi.Health [DEBUG] adi.cpp:620:HandleLog(): ISE Services is DOWN, as ISE Identityis DOWN
May 19 09:21:51 SF-IMS[32627]: [32677] ADI:adi.ISEConnection [INFO] adi.cpp:623:HandleLog(): Preparing subscription objects...
May 19 09:21:51 SF-IMS[32627]: [32677] ADI:adi.pxGridAdapter [DEBUG] adi.cpp:620:HandleLog(): pxgrid_capability_create(capability**:0x7f09f00085f8)...
May 19 09:21:51 SF-IMS[32627]: [32677] ADI:adi.pxGridAdapter [DEBUG] adi.cpp:620:HandleLog(): returns [OK|0x7f09f000b990]
[...]
May 19 09:21:51 SF-IMS[32627]: [32677] ADI:adi.ISEConnection [DEBUG] adi.cpp:620:HandleLog(): registered callback for capability SessionDirectoryCapability
May 19 09:21:51 SF-IMS[32627]: [32677] ADI:adi.Health [DEBUG] adi.cpp:620:HandleLog(): ISE Services is DOWN, as ISE Identityis DOWN

Only Session Directory Topic is checked in Identity Source - Not SXP Topic as ISE PIC is unable to handle them - 

Bulk Download start, and each entry is integrated without errors.

Then every new callback since bulk download is parsed...

So both ways are working ? 

I'm very open to any idea.

Thanks in advance

[#Mat]

Hello @Erwan LE BIHAN 

I had this problem and solved it by simply unchecking and checking a subscription topic (in my case, I've both options enable) and save it. In this way, the module will reload and then you can wait or force the health policy.

Hope this works for you too.

Regards,

[rubenmachado]

Hi,

Thank you for your suggestion. Having the same issue on 7.0.5 version. Unchecking and checking Subscription Topic solved. However opened a case to identify the root cause.

And yea completely agrre with the way Cisco TAC support has been handling the cases in the last two years. It's been an awful experience.

[Erwan LE BIHAN]

Thanks @#Mat. 

I've made it a try (check Session Directory / SXP - save - uncheck both - save - check only session Directory - Save - Health Monitor / ISE Connectivity run) and it failed.

So I took another route: I installed another FMC 6.6.4 from scratch on trial license, installed our internal PKI Root Cert, Created realm, downloaded users and Integrated it with ISE PIC.

Guess what ? Sessions integrations Working. No errors. At all.

So I looked at Health monitor and BY DEFAULT on FMC 6.6.4, ISE Health monitor is not enabled.

I exported initial Health Policy from this FMC and integrated it on my running FMC.

No more errors of course, as this Health module is not running.

I'm waiting for TAC's response.

Regards.

[#Mat]

Well done! But that's correct, TAC should have a more logical answer for your case.

Regards

[Erwan LE BIHAN]

Hi.

TLDR: Can someone using FMC 6.6+ and ISE PIC have a look at his FMC 6.6 Health policy and check if ISE Health Policy is disabled ? Thanks !

/TLDR

Am I the only one to sense that TAC is no more what it used to be ?

It's been 5 WEEKS since this ticket was created and I'm still struggling with incoherent requests.

Last(Yesterday) of them is a complete copy/paste of an already non working attempt we tested one month ago: ditch all certs, use the ISE PKI instead on both ISE and FMC. 

Of course, as FMC show both bulk downloaded (on ADI restart) and updated sessions flawlessly, we know there's no problem with certs. I however tested one more time this configuration, and, Of course, it exhibits exactly the same behavior: Sessions are working, ISE Health monitor fail.

I did one more test: On the brand new FMC 6.6.4, I changed the Health Policy to enable ISE Health Module. 

It fails because its ISE Attributes checker is down. Sure. Lack of Attributes is one of ISE PIC limitations.

Can someone with FMC 6.6 + ISE PIC have a look at FMC Health policy and verify ISE Health module is disabled ?

Thanks.

[Erwan LE BIHAN]

Hi.

We are near the end of august and guess what... Case is still open. Nearly 5 MONTHS.

I never had such a bad experience with TAC. In fact, I never had such awful experience with any tech support.

And I'm working in IT since '93 - and worked with Digital Equipment when they sold PC Gears -

It's the first time one of my case is supported by ESTARTA SOLUTIONS for Cisco.

As a customer, I'm asked to supply trace file I never heard of. With no information on how to get it.

"As per escalation team request, can you please collect pcap captures from both FMC and ISE while you are performing connectivity test and upload them to the ticket ?"

And so I'm supposed to know that FMC is linux based, and that on ssh, expert mode, I'm able to do a capture with tcpdump and dump it in /var/common ? (I knew it, so I remembered it quite easily, but it's not simple task and will never ask this to one of my customer).

Is there a way to get my case out of ESTARTA and get real answers from TAC ?

Thx.

[Erwan LE BIHAN]

Holy god !

Today, at 13:23 CET Time, the bug is acknowledged :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvz80535

So beware: don't connect your FMC to a PXGrid 2.0 ISE if you are not sure to keep it. There's no downgrade afterwards.