FMC -> ISE pxGrid Integration Issue

[scsc_tech]

My integrator set up our firepower HA pair and set up the pxGrid ISE integration. When I started testing SGT filtering in our access policies, I kept losing connectivity that contained any SGTs.

Doing some reading I think I sourced this to MNT update issues. I ran the ISE integration test, and it passes, but looking at the detailed logs, it only passes when connecting to ISE1, but fails with certificate errors for ISE2.

I have tried importing the self signed cert for ISE2 but that didnt resolve it. It doesnt appear that there is a single CA for the ISE cluster, but each node is serving as its own CA. Im looking for help in resolving this and getting FMC to communicate to both nodes.

Primary host: 
test: ISE connection.
Preparing ISE Connection objects...
Connecting to ISE server...
Beginning to connect to ISE server...
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: _reconnection_thread starts
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: pxgrid connection init done successfully
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: testing connecting to host scsc-ise1 timeout=3 ...
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: testing connection to host OK scsc-ise1:Will use ip=10.200.254.11
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: connecting to host scsc-ise1 ...
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: stream opened
Starting SSL Handshake, SSL state:before/connect initialization
Completed SSL Handshake, SSL state: SSL negotiation finished successfully
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: EXTERNAL authentication complete
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: authenticated successfully (sasl mechanism: EXTERNAL)
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: pxgrid_connection_connect: Connected. host=scsc-ise1
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: Controller version: 2.0.0.7
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: Account approved
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]:  CoreCapability successfully subscribed
Captured Jabberwerx log:2019-03-27T21:55:32 [    INFO]: _on_connect called
ISEConnection queries find the following capability states: [sessionDirectory: 1, endpointProfileMetaData: 1, securityGroupTagMetaData: 1, EPS: 1, ANC: 1]
Preparing subscription objects...
Subscribing to EndpointProfileMetaDataCapability.
Captured Jabberwerx log:2019-03-27T21:55:33 [    INFO]: EndpointProfileMetaDataCapability successfully subscribed
Subscribing to SecurityGroupTagMetaDataCapability.
Captured Jabberwerx log:2019-03-27T21:55:33 [    INFO]: TrustSecMetaDataCapability successfully subscribed
Subscribing to SessionDirectoryCapability.
Captured Jabberwerx log:2019-03-27T21:55:33 [    INFO]: SessionDirectoryCapability successfully subscribed
Subscribing to EndpointProtectionServiceCapability.
Captured Jabberwerx log:2019-03-27T21:55:33 [    INFO]: EndpointProtectionServiceCapability successfully subscribed
Subscribing to AdaptiveNetworkControlCapability.
Captured Jabberwerx log:2019-03-27T21:55:33 [    INFO]: AdaptiveNetworkControlCapability successfully subscribed
Done preparing subscription objects.
Queried 2 bulk download hostnames:SCSC-ISE2.MYDOMAIN.com:8910, SCSC-ISE1.MYDOMAIN.com:8910
...successfully connected to ISE server.
Starting bulk download
connectionHealthPollingThread starting.
Captured Jabberwerx log:2019-03-27T21:55:34 [    INFO]: curl_easy_setopt() for CURLOPT_URL: 'https://SCSC-ISE2.MYDOMAIN.com:8910/pxgrid/mnt/sd/getSessionListByTime'
Starting SSL Handshake, SSL state:before/connect initialization
Rejecting this certificate presented by foreign server: Certificate with Serial Number '0x5C2E4F49000000007D1B0F08072239FC', issued by 'CN = SCSC-ISE2.MYDOMAIN.com, C = US', to 'CN = SCSC-ISE2.MYDOMAIN.com, C = US'
...because SSL negotiation encountered error: self signed certificate
...while validating this entry in the certificate chain: Certificate with Serial Number '0x5C2E4F49000000007D1B0F08072239FC', issued by 'CN = SCSC-ISE2.MYDOMAIN.com, C = US', to 'CN = SCSC-ISE2.MYDOMAIN.com, C = US'
Sending SSL alert:unknown CA
Captured Jabberwerx log:2019-03-27T21:55:34 [   ERROR]: curl_easy_perform() failed: (60) Peer certificate cannot be authenticated with given CA certificates at file build/gcl/src/pxgrid_bulkdownload_curl.c line 241
bulk download iter next failed REST errorPeer certificate cannot be authenticated with given CA certificates
Captured Jabberwerx log:2019-03-27T21:55:34 [    INFO]: curl_easy_setopt() for CURLOPT_URL: 'https://SCSC-ISE1.MYDOMAIN.com:8910/pxgrid/mnt/sd/getSessionListByTime'
Starting SSL Handshake, SSL state:before/connect initialization
Completed SSL Handshake, SSL state: SSL negotiation finished successfully
bulk download can fetch entries.
Sending SSL alert:close notify
Captured Jabberwerx log:2019-03-27T21:55:34 [   ERROR]: curl_easy_perform() failed: (23) Failed writing received data to disk/application at file build/gcl/src/pxgrid_bulkdownload_curl.c line 241
connectionHealthPollingThread interrupted.
connectionHealthPollingThread ending.
disconnecting pxgrid
Captured Jabberwerx log:2019-03-27T21:55:34 [    INFO]: _reconnection_thread exits
Captured Jabberwerx log:2019-03-27T21:55:34 [    INFO]: stream closed; err_dom=(null)
2019-03-27T21:55:34 [    INFO]:  destroying client ... 
Captured Jabberwerx log:2019-03-27T21:55:34 [    INFO]: _on_disconnect called
Captured Jabberwerx log:2019-03-27T21:55:34 [    INFO]: Event loop exit. status=1
Captured Jabberwerx log:2019-03-27T21:55:34 [    INFO]: pxgrid_connection_disconnect completes

[Rob Ingram]

Hi,
Sounds like you are on the right track.

The FMC needs to obviously trust the certificates issued on the ISE nodes (pxgrid and mnt). Can you check which CA the FMC is configured to trust and then check the system certificates for each ISE node for each role ("Used By" column) and ensure the same CA ("Issued By" column) issued the certificates. You possibly need to generate a certificate for the 2nd node and ensure you bind this to the pxgrid role (this will be the "Used By" column).

HTH

[scsc_tech]

Currently the FMC is set to the ISE1 node's CA certificates (even though I have imported both into the trust store)

In ISE certificates, the pxGrid certs are both issued by their own node, ISE1 and ISE2 respectively. 

[Rob Ingram]

Ok, I assume you mean you are currently using a self-signed certificate by each ISE node, not the ISE Cluster Internal CA.

When configured in a cluster, ISE has an Internal CA which can issue the pxgrid certificate or you could use an External CA (Windows CA). Here and here are cisco guides for configuring pxgrid certificates, with internal or external CA.

HTH

[scsc_tech]

Thanks

From what I am seeing in ISE, there are two CAs under "Internal CA Settings" one for each node.

But in CA overview there is just ISE1. Assuming you are saying to generate a new pxGrid certificate for ISE2 that is issued by ISE1 CA?

[Rob Ingram]

I assume you are using ISE 2.2 or greater? Within ISe in the "pxGrid Services" tab you have "certificates" from there you can sign certificates for use by the pxgrid service on ISE and by the FMC.

HTH

[scsc_tech]

2.3

Thanks RJI.

I performed that task, and the output certificates didnt seem to have a certificate path.

When I open the node and endpoint certs in Windows and go to the Certification Path tab, Certificate Status: "The issuer of this certificate could not be found." 

The root ca reports "This CA Root certificate is not trusted because it is not in the Trusted Root Certification Authorities store."

[Rob Ingram]

Last time I used it, it created a zip file with the certificate + root, sub CA etc

Windows would not nativately trust that CA because it would not have the Root certificate in it's trusted root store.

ISE should have those Root/Sub Root certificate in it's trusted certificate store. FMC probably won't (not 100% sure how yours is configured) have the Internal CA Root/Sub Root certificates in it's trusted root store, you'll have to import and create a CSR and sign a new certificate.

[scsc_tech]

I worked with TAC and got the certificates reissued with a common CA. 

Now when I run the integration test, it successfully connects to both pxGride nodes.

There is, however, one more error and I cant seem to find any reference to it online

Starting SSL Handshake, SSL state:before/connect initialization
Completed SSL Handshake, SSL state: SSL negotiation finished successfully
bulk download can fetch entries.
Sending SSL alert:close notify
Captured Jabberwerx log:2019-03-29T19:06:46 [   ERROR]: curl_easy_perform() failed: (23) Failed writing received data to disk/application at file build/gcl/src/pxgrid_bulkdownload_curl.c line 241
connectionHealthPollingThread interrupted.
connectionHealthPollingThread ending.
disconnecting pxgrid

[fedor.solovev]

Hi, it's been a couple of years, but
Could you share, what exactly did you do with TAC together ?

[JohnLima]

Today I had a TAC and to solve this issue we went like this:

1. The pxGrid Server CA was created on ISE  and signed by our Windows Root CA, after that it was imported as Truted CA in FMC and it was in System Certificate on ISE.
2. The MNT Server CA was the our Third-party Windows Root CA and was imported in FMC and ISE as a Trusted CA.
3. The FMC Server Certificate was requested by FMC and signed by Windows Root CA,and then imported into ISE Trusted Certificate.

I think our biggest problem was identifying the correct order of the certificates and signing them properly.