Best approach to updating TACACS/RADIUS share secrets on ISE and network devices

[pn2020]

We have an automation (using python) project where we have to update shared secrets on network devices (Cisco IOS/IOS XR and also other non-Cisco platforms, via netmiko).  Of course, we also have to update (via ERS REST API) the Cisco ISE server with the same shared secrets.

What is the best approach to updating, with minimum connectivity outage?  One device at a time, ie, updating the ISE, then updating the device, check for connectivity, then move on to another device?  Or bulk update, ie, updating the shared secrets on the ISE for a small group of devices, then updating the secrets for the same group of devices?

I assume it would be one at a time, but like hear additional feedbacks.  

Also, is there a solution whereby we can dictate/direct the Cisco devices (or any network device platform) and Cisco ISE to try to check the authenticate using the new shared secret.  If check is good, then flip over to the new shared secret.  That way, we can get a minimum connectivity disruption. Is that possible?

Thanks,

Peter

[balaji.bandi]

Do you have a fall back Local Account? if yes

I will change all the Secret at end Device First and later on ISE Side. and test

best practice, test 1 or 2 devices all working as expected, then deploy mass device config change. (even it fails you have fallen back to Local Account to change as required)

[pn2020]

Thanks BB.  

We have quite a few devices, so likely that some won't have a local account to fall back.  Especially, a number of them aren't Cisco platforms.  

So, I assume the safest plan is to do one by one?  

BTW, is this how to check if Cisco IOS routers/switches are configured with local tacacs/radius authentication fallback when communication ISE/ACS server is down?

aaa authentication login default group tacacs+ line

aaa authentication login console group radius local

[balaji.bandi]

The device does not have control like Cisco devices if not many suggest changing manually and test it.

as per the cisco devices - you should ( as per my interested) have local account some point you may need if any disaster of ISE or network connection loss.

below command fall back to Local - only if you have a local username and enable password enabled. 

aaa authentication login console group radius local

Note: if not your device is locked and you need to go with password recovery.

[Mike.Cifelli]

Agree with @balaji.bandi .  Another thought to ensure clients connected to certain NADs that onboard via 8021x or mab is to change the reauth timer to a greater time setting than you have configured now.  Essentially what you could do is bump the reauth timers to 8-12 hours in your authz profiles to buy you time to conduct your changes, and avoid having client onboarding issues.  Always test on one or two devices first before mass rollout.  Good luck & HTH!

[howon]

Aside from the tips provided, you can also leverage second shared secret feature on ISE. This allows two shared secret to be active at the same time for migration (Only available for RADIUS):

[pn2020]

Thanks everyone for your feedbacks!

I will look more into this.

[balaji.bandi]

@Mike.Cifelli yes that would be a nice idea, original post does not mention any BYOD feature using with ISE, But your point needs to consider one another aspect of dependency, good point.

let us know how it goes.

[pn2020]

Hi Mike,

Your idea sounds good, but I am not well tracking.  If you can further expand to help me understand that would be great.  

Specifically, on these:

- NADs: what do you mean by NADs?

- that onboard via 8021x or mab 

- reauth timer: where do I change that?  On the devices, ISE server, or both?

Thanks.

[balaji.bandi]

Do you use ISE for dot1.X authentication?  or BYOD in your environment? 

check other screenshot posted on this post - you can have seconds shared key, (not tried myself)

[pn2020]

We are not using ISE for dot1.X authentication, no BYOD.

The TACACS doesn't have the 2nd shared secret, but it has the retired secret feature.  I like to explore that feature more. If anyone has good experiences with the retired secret feature in production or in lab, please share.  Details on how that feature works would be great.

Thanks in advance!

[hslai]

...

The TACACS doesn't have the 2nd shared secret, but it has the retired secret feature.  I like to explore that feature more. If anyone has good experiences with the retired secret feature in production or in lab, please share.  Details on how that feature works would be great.

...

---

See Change your shared secret without network disruption 

This can only be configured in ISE admin web UI.

[pn2020]

Thanks  hslai.  So the retire secret feature can't be configured via ERS REST API?

[hslai · Accepted solution]

---

@pn2020 wrote:

... So the retire secret feature can't be configured via ERS REST API?

---

That is correct. Please voice your feedback through New Features and Feedback 

[pn2020]

That's definitely disappointing and a major deficiency, especially RADIUS supports the secondary shared secret. 

I did leave a request on that forum channel.

Thanks.