Change of focus
This blog is going to cover a work around for a current issue with "aaa command authorization" which leads to the "ERROR_HEALTH_CHECK_TIMER_EXPIRED" error for a network-device.
The fix for this is in 1.3 release of PnP, but will require IOS upgrades to support it.
Here is a work around using an Embedded Event Manager (EEM) script built into the configuration pushed to the device during Plug and Play.
I am going to assume you have seen my other blogs on PnP, so I am not going to cover the basics, just the issue and the solution.
Problem
If you have "aaa command authorization" in a configuration file you will seen the screen shot below.
Strangely, although APIC-EM thinks the process has failed, you will see the configuration file looks correct (which it is).
Why is this happening?
- Once you enable command authorization, the switch needs a username to authorize the commands it is running.
- After the configuration file is downloaded to the network-device, the final step is to "write mem" and save the configuration.
- Of course, this command will be authorized.
- As the PnP process originates from the network-device, it has never been logged into, so does not have a valid username to authorize any exec commands. (If you look at the ISE logs, you will see a user called "async" is being used for the commands).
Solution
The solution is quite simple. Simply leave out the "aaa command authorization" commands, and place them in an EEM script to run after the PnP process completes.
The EEM script will remove itself after it has run.
- The EEM script needs to also be authorized. In this example, I have provided and user "sdn2" and a privilege level "15". This user is not a local user, it is only defined on my tacacs server as that is where authorization is occurring.
- event manager session cli username sdn2 privilege 15
- Move the "aaa" commands inside the EEM script that runs 30seconds after being created.
- action 2.1 cli command "aaa authorization commands 1 default group ISE-T if-authenticated"
- action 2.2 cli command "aaa authorization commands 15 default group ISE-T if-authenticated"
- The EEM script then removes itself.
- action 2.3 cli command "no event manager applet POST_PNP
- Issue another "wr mem" to save the configuration. This command will be authorized, so if you do not have a valid username for step #1, the write will fail.
- action 2.9 cli command "wr mem"
- The two debug commands are optional. I included them in case you want to look deeper into aaa and eem scripts. Make sure you remove these and do "undebug all" after you go live.
- action 1.1 cli command "debug event manager action cli"
- action 1.2 cli command "debug aaa authorization"
hostname 3650-dns
aaa new-model
!
tacacs server ISE
address ipv4 10.10.10.130
key xxxxxx
!
aaa group server tacacs+ ISE-T
server name ISE
!
aaa authentication login default group ISE-T local
aaa authentication enable default group ISE-T enable
aaa authorization exec default group ISE-T if-authenticated
aaa accounting exec default start-stop group ISE-T
aaa accounting commands 0 default start-stop group ISE-T
aaa accounting commands 1 default start-stop group ISE-T
aaa accounting commands 15 default start-stop group ISE-T
enable password xxxxx
!
user sdn privilege 15 password xxxxx
!
ip http server
ip http secure-server
snmp-server community xxxxxx RO
!
line con 0
line vty 0 4
transport input ssh telnet
line vty 5 15
transport input ssh telnet
!
event manager session cli username sdn2 privilege 15
event manager applet POST_PNP
event timer countdown time 30
action 1.0 cli command "enable"
action 1.1 cli command "debug event manager action cli"
action 1.2 cli command "debug aaa authorization"
action 1.8 cli command "config t"
action 2.1 cli command "aaa authorization commands 1 default group ISE-T if-authenticated"
action 2.2 cli command "aaa authorization commands 15 default group ISE-T if-authenticated"
action 2.3 cli command "no event manager applet POST_PNP"
action 2.8 cli command "end"
action 2.9 cli command "wr mem"
action 3.0 cli command "end"
end
UPDATE: May 2017. I have found situations where the EEM script fires too quickly, mainly if you are doing a management VLAN switchover, rather than pnp startup vlan. You might need to change this to a longer countdown time. Change from 30 to 180 seconds.
After the switch boots up, you can see that the status is successful in PnP application.
Looking at the startup configuration, the "aaa authorization" commands are present.
3650-dns#show start | inc autho
aaa authorization exec default group ISE-T if-authenticated
aaa authorization commands 1 default group ISE-T if-authenticated
aaa authorization commands 15 default group ISE-T if-authenticated
What Next?
This blog covered a workaround for the "aaa command authorization" issue. It has already been resolved in APIC-EM 1.3 and will be resolved in IOS network-device software images soon.
This solution is only for those people who would like to deploy now, on current versions of IOS.
This workaround highlights the power of EEM scripts.
In the meantime, if you would like to learn more about this, you could come hang out with us in The Cisco Devnet DNA Community. We’ll have a continuous stream of blogs like this and you can ask questions and we’ll get you answers. In addition, we have a Github repository where you can get examples related to PnP.
Thanks for reading,
5 Comments