This blog is going to cover a work around for a current issue with "aaa command authorization" which leads to the "ERROR_HEALTH_CHECK_TIMER_EXPIRED" error for a network-device.
The fix for this is in 1.3 release of PnP, but will require IOS upgrades to support it.
Here is a work around using an Embedded Event Manager (EEM) script built into the configuration pushed to the device during Plug and Play.
I am going to assume you have seen my other blogs on PnP, so I am not going to cover the basics, just the issue and the solution.
If you have "aaa command authorization" in a configuration file you will seen the screen shot below.
Strangely, although APIC-EM thinks the process has failed, you will see the configuration file looks correct (which it is).
Why is this happening?
Once you enable command authorization, the switch needs a username to authorize the commands it is running.
After the configuration file is downloaded to the network-device, the final step is to "write mem" and save the configuration.
Of course, this command will be authorized.
As the PnP process originates from the network-device, it has never been logged into, so does not have a valid username to authorize any exec commands. (If you look at the ISE logs, you will see a user called "async" is being used for the commands).
The solution is quite simple. Simply leave out the "aaa command authorization" commands, and place them in an EEM script to run after the PnP process completes.
The EEM script will remove itself after it has run.
The EEM script needs to also be authorized. In this example, I have provided and user "sdn2" and a privilege level "15". This user is not a local user, it is only defined on my tacacs server as that is where authorization is occurring.
action 2.3 cli command "no event manager applet POST_PNP"
action 2.8 cli command "end"
action 2.9 cli command "wr mem"
action 3.0 cli command "end"
UPDATE: May 2017. I have found situations where the EEM script fires too quickly, mainly if you are doing a management VLAN switchover, rather than pnp startup vlan. You might need to change this to a longer countdown time. Change from 30 to 180 seconds.
After the switch boots up, you can see that the status is successful in PnP application.
Looking at the startup configuration, the "aaa authorization" commands are present.
3650-dns#show start | inc autho
aaa authorization exec default group ISE-T if-authenticated
aaa authorization commands 1 default group ISE-T if-authenticated
aaa authorization commands 15 default group ISE-T if-authenticated
This blog covered a workaround for the "aaa command authorization" issue. It has already been resolved in APIC-EM 1.3 and will be resolved in IOS network-device software images soon.
This solution is only for those people who would like to deploy now, on current versions of IOS.
This workaround highlights the power of EEM scripts.
In the meantime, if you would like to learn more about this, you could come hang out with us in The Cisco Devnet DNA Community. We’ll have a continuous stream of blogs like this and you can ask questions and we’ll get you answers. In addition, we have a Github repository where you can get examples related to PnP.