Issue installing wildcard cert back on Primary PAN (after RMA)

[tachyon05]

I have an ISE deployment with 2 PANs and 2 PSNs.  Primary PAN, node A, had to be replaced due to hardware failure.  node B was promoted temporarily so I can continue to manage ISE.  I need to install the wildcard cert back on the new node A and promote it to be the primary PAN again.  Issue is the this wildcard cert won't install on the new server.

Node A will take the cert while it is in standalone mode, but cert will disappear or change after node is joined to the deployment.  If I export the cert from node B and reinstall it on node A, it will disappear, not install, or change after it is installed.  Specifically, the "issued to" field of the cert will change making it unable (e.g. change to FQDN of node A instead of what it shows on nodes B, C, and D).  I worked with 5 TAC Engineers over the last 2 month, and tried various methods, some repeatedly, but all failed so far.

I do have 4 "spare" servers I can repurpose.  I have configuration backup and the cert and key.  One solution maybe to standup a new deployment using these 4 spare servers, and cutover to them from the existing deployment.  What is the best way to do that?

[Marcelo Morais]

Hi @tachyon05 ,

 please try to:

1st restore the backup of Node A including the ADEOS:

ise/admin# restore CONFIG-DATA.CFG10-<file name>tar.gpg repository <repository> encryption-key plain <password> include-adeos

2nd check not only if the Certificate is OK, but also if the Node A is working.

Note: at this point if Node A is "back on business", then try to deregister Node C (PSN) from Cluster and register it to Node A ... if everything is OK, continue the process !!!

IMPORTANT: when restoring ADE-OS you would be restoring OS Level Configuration. This would include ALL of the OS Configuration data that is configured when setting up the ISE Node (like hostname, IP Addr, NTP, enabling SSH, default gateway and name servers). Restoring the ADE-OS configuration would be used if you want an exact duplicate of the ISE server the backup was taken from !!!

Hope this helps !!!

[tachyon05]

Thanks Marcelo.  Three questions.

Node A has been non-operational due to cert issues for 2 month even though it has been joined to the deployment for much of that time.  If Node B (current primary PAN) dies, we would be in a bad shape without a working secondary PAN.  Can I deregister node A, add ADMIN and MONITOR roles to one of the PSNs (node C or D) so they become a secondary PAN to take that pressure off? 

All recent backups were taken from current primary PAN node B.  Does this mean backups contain ADE-OS info (such as IP and hostname) for node B and can't be used to restore a new node A?

I have the hardware to build up another deployment and would prefer not touching the production deployment unless I have too.  Would it be possible to get a new deployment up and running and then cutover to it with the following steps?

1. Shutdown network ports on node A from production deployment - node A is not working anyways.

2. Restore a configuration backup onto new standalone node A, with the same IP and hostname as production node A.

3. Install certs on new node A in new deployment.

4. Shutdown network ports on node C (PSN) in production deployment.

5. Install ISE on new node C in new deployment using the same hostname and IP as production node C and join it to new deployment.

5. Repeat for nodes B and D.

Thanks

[ahollifield]

What you describe is not a supported ISE configuration.  It is not supported to have Admin+Monitor+PSN on the same ISE node unless it is a two node HA deployment.  Although, it might technically work for this particular issue.  

That is correct, you would want to restore any backup without including the ADE-OS information (the default behavior anyways).

[Marcelo Morais]

Hi @tachyon05 ,

 1st " ... Can I deregister Node A, add ADMIN and MnT roles to one of the PSNs (Node C or D) so they become a SPAN to take that pressure off? ... ", the straight answer is yes, but always have in mind the Performance and Scalability Guide for ISE, search for Different Types of Cisco ISE Deployment.

 2nd " ... Does this mean backups contain ADE-OS info (such as IP and hostname) for Node B and can't be used to restore a new node A? ... ", 

 a. whenever you generate a Backup the ADE-OS is included

 b. you can use the "Node B Backup" to Restore the CONFIG-DATA to Node A not the CONFIG-DATA and ADE-OS:

ise/admin# restore CONFIG-DATA.CFG10-<Date-Hour>tar.gpg repository <repository> encryption-key plain <password>
ise/admin# restore CONFIG-DATA.CFG10-<Date-Hour>tar.gpg repository <repository> encryption-key plain <password> include-adeos

  3rd " ... Would it be possible to get a new deployment up and running and then cutover to it with the following steps? ... ", yes !!!

 Note: just adding the following:

3.1 register Node A to the Cluster as a SPAN

5.1 register Node C to the Cluster as a PSN

Hope this helps !!!