Cisco TAC Recommends Moving pxGrid to Mnt node in Distributed Deployment

[Evanjrosado]

Hi everyone,

Our ISE 2.4 patch 9 environment consists of (7) nodes that are dedicated to a unique persona to make a distributed deployment. (2) of our PSN's have the pxGrid service enabled. Long story short, we just integrated our pxGrid servers with (2) WSA's and when we ran the test to verify the output in the log window shows they WSA is unable to retrieve user-sessions.

*****Beginning is WSA log********

Checking DNS resolution of ISE pxGrid Node hostname(s) ...

Success: Resolved 'psn-1' address: x.x.x.x

Success: Resolved 'vpsn-1' address: x.x.x.x

Validating WSA client certificate ...

Success: Certificate validation successful

Validating ISE pxGrid Node certificate(s) ...

Success: Certificate validation successful

Success: Certificate validation successful

Checking connection to ISE pxGrid Node(s) ...

Trying secondary PxGrid server...

Preparing TLS connection...

Completed TLS handshake with PxGrid successfully.

Trying download SGT...

Able to Download 27 SGTs.

Trying connecting to primary ERS service...

Trying download user-groups...

Able to Download 29 user-groups.

Trying connecting to secondary ERS service...

Trying primary PxGrid server...

Preparing TLS connection...

Certificate validation error Certificate validation error: Unacceptable certificate from psn-1: application verification failure.

Failure: Connection to ISE pxGrid Node failed.

Trying download user-sessions...

Failure: Failed to download user-sessions.

Trying download user-groups...

Able to Download 29 user-groups.

Failure: Connection to ISE pxGrid Node failed.

Test interrupted: Fatal error occurred, see details above.

*****End is WSA log********

We have a TAC case open on this and they pointed us to bug id: CSCvq03494

WSA Failed to download user-sessions from ISE pxGrid

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvq03494/?reffering_site=dumpcr

The TAC engineer is recommending that we move the pxGrid roles to the MnT services. I wanted to get everyones opinion on this one. Any thoughts would be helpful.

[jeppich]

Hi,

Yup, this is a known bug and will be fixed in "CSCvr10059 ISE 2.4 P9 pxgrid with WSA: WSA not retrieving AD groups from ISE".  Please provide your customer information to TAC, if you have not done so already.

Thanks,

John

jeppich@cisco.com

[Evanjrosado]

Are there any drawbacks to doing this? Any concerns or considerations? I don't want to mess around with the MnT servers. I was hoping to have them separate considering their function.

[jeppich]

Hi,

This would require a certificate change, so it would affect other pxGrid deployments.  You will need to re-configure your pxGrid clients to use the proper certificate.

Thanks,

John

jeppich@cisco.com

[jj27]

This.  And, you will likely need to reconfigure pxGrid integrations to utilize the new IP addresses if they were originally established that way.

[Evanjrosado]

Thanks for the feedback. The pxGrid subscribers that we have such as FMC, Stealthwatch, QRadar, DNAC, etc are pointing to ISE using dns name. And I understand the process of renewing the certificates. I just don't want to co-mingle pxGrid with our MnT honestly. If i had my choice they would be on a dedicated VM's. 

[jeppich]

Hi,

Yup, understood.  Probably best to wait for the patch or setup a stand-alone instance of ISE 2.4 and WSA.

Thanks,

John

jeppich@cisco.com

[Damien Miller]

Take this with a grain of salt. Do not implement the "workaround" for two reasons. The first, it's not an officially supported deployment model that has been tested, dedicated nodes are meant to be dedicated. The next, my customer who has run in to the same issue during testing has told me that a bug fix is coming soon but from the WSA code side.

If it's not critical, I would hold station like John mentioned. I for one can't wait to start using pxg v2 for this integration, come on WSA team *fingers crossed*.

[Evanjrosado]

Hi Damien,

Thanks for your response and I 100% agree with you and have recommended to my CISO and team that we should vet this recommendation out first before taking action right away. Not that I want to commingle our dedicated MnT services with pxGrid services but to play devil's advocate, would it hurt the MnT's considering their vital function and role that they play in the deployment? We do have the pxGrid services running on (2) of our dedicated PSN's.

[jeppich]

Hi,

It is recommended that the (2) pxGrid node are dedicated. However, I have seen deployments, where they are part of the MNT, PSN persona.
If you have additional questions, please email me directly.

Thanks,
John
jeppich@cisco.com

[Evanjrosado]

Hi John,

thanks for your response. I'm going to recommend to keep our nodes dedicated and submit a request through TAC for a patch request if one hasn't been submitted. We will create new VM's and move our (2) pxGrid nodes there. 

Just a heads up for everyone out there, we upgraded our WSA deployment from 11.8.0-440 to 1.8.0-414 to fix a pxGrid issue that prevented WSA from pulling AD groups from ISE. BUT after upgrading to this version it caused our WSA appliance to lose connectivity because the M1 interface no longer support 1 gig interface due to a defect. We could not manage the appliance through the M1 at that moment but thankfully we had remote console access. We were told by TAC that we had to either roll back to use the 1 gig interface or to keep the upgraded version and upgrade the M1 interface to a 10 gig sfp connection. Even though these seems wasteful because its a management connection, we ended up meeting in the middle of the road to keep the upgraded version so that WSA could pull AD groups from ISE and upgraded the M1 interface to 10 gig sfp. But now we found that WSA can pull AD groups but not user-sessions. 

[Evanjrosado · Accepted solution]

Update: TAC recommended temporarily pointing the WSA's to the MnT's instead of pointing them to the pxGrid servers directly for pxGrid to work. We have verified and tested this solution. This is temporary until a fix on the WSA side is released. 

[Damien Miller]

Traditionally, the messaging is that you don't run anything on the PAN/MNT once you get in to a hybrid/dedicated design. If you were to run a 1 or 2 node standalone deployment, then all roles run on the same two VM/appliances including pxgrid. So from the technical perspective it will run.

From the scaling guide; 160 connections
PAN+MnT+PXG on same node and dedicated PSNs
-OR-
PAN+MnT and dedicated PSN & PXG
Minimum 4 nodes redundant
Source: https://community.cisco.com/t5/security-documents/ise-performance-amp-scale/ta-p/3642148#toc-hId--290198031

The ISE scaling guide lists running pxgrid on its own PSN or the PAN/MNT with the same number of connections. It will work but it goes against the general recommendations. You can turn up the pxgrid persona on your pan/mnt and test, having other integrations already in use could complicate that since you may have to set them up again.

The 11.7 WSA integration was tested using a standalone deployment hosting all roles on the same nodes, which is why it's causing us an issue now. I for one am just waiting for the software fix.

[Evanjrosado]

Thanks again for the feedback Damien. I agree and hope for a software fix soon.