DNAC/ISE ERS Integration issue after changing ISE password

[Mike.Cifelli]

I recently built out a new ISE cluster that we migrated to, same IPs and hostnames, but 2.7p2 version.  During this rebuild I updated the ISE password for the account used to integrate DNAC with ISE.  After a successful ISE cluster migration I noticed the ERS communication between the two was not working.  Note that the pxgrid integration was functional though.  After attempting to re-trigger the integration by entering the new updated password in DNAC UI under Authentication and Policy Servers the status would move to 'Active' and then immediately move to inactive.  Under system360 the DNAC error would complain about an incorrect password.  After working with TAC we identified the issue was due to this bug: CSCvt27360 (https://quickview.cloudapps.cisco.com/quickview/bug/CSCvt27360).  The workaround included updating the DNAC db manually via encrypting the new shared secret since the db decrypted password depicted the old password.  Lastly, DNAC developers mentioned that if the ISE password is changed again, the bug will re-occur. They are still working on the fix as it has been hard to get the root cause of this bug.  HTH!

[Arne Bier]

@Mike.Cifelli  - Do you know if this is fixed in any DNAC release? When integrating ISE/DNAC for the first time it all seems quite innocuous ... but changes to ISE (upgrades/rebuilds/password) have also plagued me in the past. Yours is another trick to keep in mind ...

[Mike.Cifelli]

@Arne Bier - Last I heard was this: 

Unfortunately, there is still no fix integrated with a DNA Center version at the moment. This is a hot topic and there is a going discussion between engineering and developers teams regarding this bug, the potential cause and the fix, but no resolution yet.

My assumption is that it will get fixed in a 2.x version.  Until fixed, my advice would be to not change passwords.

[matty-boy]

We're currently working the issue right now with TAC.

Customer had to upgrade to larger ISE, DNAC and WLC appliances to cope with an increase of APs and clients.

We swapped out the WLCs and ISE with no (apparent) issues.

Yesterday we swapped over to the new DNAC cluster (we backed up the old cluster and restored onto the new cluster after ensuring the system and application packages were all the same).

Restore was successful and I can see all the switches, WLCs and APs in the inventory and command runner works.

However, DNAC cannot fully communicate with ISE.

In System360, both ISE servers show as green/available but when I go into the Authentication and Policy Servers settings page it shows ISE as INACTIVE.

If I edit the entry, re-enter then password and hit save, after about 30 seconds I see an error message in the corner of DNAC with a red warning triangle:

#########################

Decryption failed : Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

#########################

Each time I try to re-integrate via the DNAC GUI, a new alarm is generated in ISE:

#########################

Alarms: Service Component Error

ERS identified deprecated url.The request url is deprecated and recommended avoid using it

(No further details available)

#########################

I had a Webex session with TAC (AAA team and DNAC team on call) but we're no further other than the fact that both TAC engineers are reviewing the respective logs for DNAC and ISE.

I hope this is resolved soon as we are unable to make any changes in DNAC at the moment!

Cheers,

Matt.

[Mike.Cifelli]

FYSA, Per TAC:

So far the only thing has been told about a release is that it won’t be fixed in 1.3.3.9. The fix will be applied in a Wolverine patch (2.1.2.x), however, developers are still working the root cause of the issue so they can apply a fix later on, but no fix yet.

@matty-boy did you receive any other info on this? 

[matty-boy]

Hi @Mike.Cifelli,

It was a fun one for sure! Somehow DNA had got upset and a bunch of encrypted credentials stored in the DB had become corrupted.

Case was escalated to the BU where an excellent BU engineer diagnosed and resolved the issue in very little time. She extracted the encrypted password and manually decrypted it where we saw that it was nonsense. So we took the clear text password, encrypted it manually and inserted it back into the DB. We had to do the same thing in a few different parts of the DB and all is now good.

So unfortunately it's not something a mere mortal would be able to do. Definitely one for the BU!

Hope this helps,

Matt.