ISE intermediate CA certificate replacement used for admin certificate

[Andrej Sumak]

Hi Team, 

I need to replace an intermediate CA (subordinate CA certificate) in the chain, which is simultaneously used for admin and eap roles. The intermediate CA certificate was unfortunately renewed in a way, that the existing certificate with the original private key was simply extended (MS allows this). This means we now got a new intermediate CA cert, which is identical to the old one but has differences in the serial number and expiry dates fields. 

How to proceed in such a case in a large distributed deployment. Following the workaround documented in Further Problem Description section for https://bst.cisco.com/bugsearch/bug/CSCut10928 is simply not feasible as it causes to much downtime of the service. 
Experts, how would you tackle this problem? 

thanks and cheers,

 Andre

[Arne Bier]

Hi

You will install the new intermediate CA cert on the Admin. That automatically copies it to all secondary nodes.

Then starts the fun. You'll create a CSR for each node (I assume you don't want to use the same Admin/EAP cert for all nodes). You can decide at this point to only create new EAP certs for each ISE Node that is a Services Node (PSN).  Submit the CSRs to the CA and then bind the certs. When you bind the EAP certs you will have no service disruption. That's the good news. At least then your EAP Service will be updates to use a new cert that includes the new Intermediate CA.

For the not-so-fun part .. the Admin cert renewal. Choose any node that is not the active Admin node and then create a CSR for the Admin role. When you bind that cert, it will restart services on the node - causing a 10-15 downtime. This is unavoidable. For PAN and MNT this does not present any end-user challenges. When you restart a PSN, then I hope your network devices can failover to a secondary PSN.

I have also recently read that when you get to the last node (active Primary PAN) then you run the risk of restarting ALL nodes at the same time - like a simultaneous restart of ALL nodes. I think this may be an ISE 3.1 "feature". In any case, if you want to be 100% sure to avoid this melt-down, then promote your Secondary PAN and wait for the Admin nodes to settle down again (20-30 min). Now the new Secondary node's Admin cert can be renewed without that spontaneous reboot issue.  Once that has been done, promote the Secondary again, and then you're back to your original PAN Active/Standby scenario.

When you look at the final certificate status of each ISE node, you can generate self-signed certificates for all the roles that you don't use - e.g. EAP on Admin/MnT nodes, or RADIUS DTLS etc.  I tend to make 10 year self-signed certs. That means I won't be bothered by any expiration alarms for a while.

SAML is a bit special - I think you only generate one cert for the entire deployment.

Any certs marked as "Not in use" can be deleted.

[Andrej Sumak]

hi, 

tnx for the reply, but i already stumbled on an issue in step 1.  
When i try to import the new intermediate cert, ISE of course sees that nearly everything in the cert is the same and does not allow to import it, but wants to replace the current intermediate with the new one.  After discussing with my colleagues, the chance is almost 100% that this will fail, as ISE´s server certificate are signed by the old intermediate CA, which needs to be replaced. This is actually the issue, that currently poses a showstopper on my side. To get this more clear i also uploaded a small diagram. 

How i see it, is, that no matter what i do, I need to move the current EAP and Admin roles to some new certificates, node per node.

And yes, you are correct with the PAN admin cert replacement. It triggers a rolling restart of services on all of the nodes, but it never disrupted anything in our deployments, as it is a rolling restart, meaning node after node. 

The rest is clear and i agree with you  

tnx,

 A

[Arne Bier]

One workaround for that duplicate cert during import, is to create a CSR where some part of the cert is unique - e.g. you can make the Organization to be slightly different to the current one. That is enough for ISE to tell them apart. it certainly works when importing system certificates in ISE (e.g almost identical cert used for Admin and EAP, but where one of the certs has a different value in the "Organisation" field)

[Milos_Jovanovic]

Hi @Andrej Sumak, @Arne Bier,

I don't think this will be possible without re-issuing new Intermediate cert, with new pair of keys this time.

Let's assume for a moment that it is even possible to import new Intermediate CA with same key (which is not). From this moment on, I believe that all certificates issued by old Intermediate CA on client machines or users would not be considered as trusted anymore (and for a period of time, you must have overlapping between certs issued by old and by "new" Intermediate CA). ISE is having hardcoded certificate, with its public key and serial number, so I believe client cert validation would fail, because is simply has different Intermediate from what is with client.

From my standpoint, proper way should be re-issuance of new Intermediate CA, followed by dependent tasks on remaining components (e.g. same cert is distributed to all clients as trusted one, potentially you add it to GPO configs, if you are filtering is somewhere, and lastly you replace certs for Admin/EAP on ISE)/

I would like to hear back from someone from Cisco about this, as this is quite an exotic task (that I personally just dodged recently).

Kind regards,

Milos

[Andrej Sumak]

Hey Milos, 

tnx for your valuable input. I totally agree with you, that only setting up a separate new intermediate CA will be a feasible solution and then issue new ISE certs one by one and also replace them one by one. Of course the new intermediate CA needs to be imported first into the trusted root store of ISE and endpoints first. 
The whole thing is actually a cat and mice game. As soon as you need to replace your intermediate CA cert this automatically means, that you need to also replace your ISE identity certs. It´s even more complicated if you use the same cert for multiple roles. I generally tend to use public certs (globalsign, digicert, etc...) for admin role and an internal PKI cert for EAP role. 
But this is a new customer, where I got ambushed...  
And as many sources suggest, I would never go with self signed certs. Not even for a temporary setup. Using self signed certs for EAP means, importing the ISE Issuing CA cert to all endpoints via GPO, then altering the 802.1x config for trusting this new CA, etc... Much to much work and to many mistakes that can happen if you run a big shop, where there is a separate department for everything in IT...

BR,

 Andrej

[hslai]

@Andrej Sumak and team: Without getting the same or similar in our lab, I am not certain what would happen.

In the past, I ran into some situation that ISE did not allow the certificate replaced. Then, I had to workaround it by using some other certificate (e.g. a new self-signed), deleting the certificate being replaced, and then importing the new certificate.

[REJR77]

Hi Andrej,

Quite an old post, but I am quite in the same situation. Intermediate CA certs were renewed with same private keys (only expiration date and Serial are changed). How did you finally proceed with this? Did you have any issues with EAP authentication or with ISE deployment?

Did you renew the Admin and EAP certs and what was your procedure?

Regards

[Andrej Sumak]

Hi @REJR77, 
As a matter of fact, i had to get new certs for all ISE nodes, signed by the new intermediate CA cert to use those new certs then for EAP and Admin roles... 
Hope this helps. 

[LRATS]

Hi @Andrej Sumak,

thanks for your last comment! 

I am currently experiencing the same problem.
Yesterday I replaced the intermediate certification authority certificate with a new one and used the same key.
I then imported the new intermediate CA certificate into the ISE and ignored the “replace old certificate” warning. What could possibly go wrong?
This has resulted in all clients receiving a warning from Windows this morning that they no longer trust the “old” EAP certificate - presumably because the ISE has now sent the new certificate instead of the old intermediate certification authority certificate and the chain then of course no longer fits exactly.

Now my thought was to create the EAP certificate with the new CA certificate and I followed your blog post carefully.

Your last comment confirms that I need to recreate the EAP & admin certificate with the new intermediate CA certificate.

Now my important question: Was there a problem with the “old” client certificates afterwards? Most of our clients still have a certificate that was issued by the “old” intermediate certification authority for at least 1 year. Was there still a trust problem when the clients connected to ISE?

Thank you very much!

[Andrej Sumak]

HI @LRATS , 

No, indeed there was no problem with the clients, which got the certs issued by the "old" intermediate CA. But i only have a clean Windows environment where the clients are sending the complete chain. I assume you could run into an issue with clients, which only send their identity certificate. Then it could get to a problem imo. 

I luckily also dodged the bullet with the clients and the warning where they are complaining, that they don´t trust the ISE EAP certificate anymore. My customer already pushed both intermediate CA certs into the trusted store on all clients upfront. 

Hope this helps you.

BR,

 Andrej