How to clear hanged EEM jobs on Cisco Nexus 5596

[gpn-spb-cisco-access]

Hello. 

I have Cisco Nexus 5596 running NX-OS 7.1(2)N1(1) that faced problem with hanged EEM jobs that occupied all available threads.

EEM applet is very simple:

event manager applet Schedule_Backup
  event timer cron cron-entry "0 1 * * *"
  action 1.0 cli command "backupconfig"

"backupconfig" is alias to command copy startup-config scp:/<...>.

Since last reboot of the switch (24 november 2018) all available EEM threads were occupied with hanged jobs so for now output of "sh event manager policy" looks like:

Key: p - Priority        :L - Low, H - High, N - Normal, Z - Last
     s - Scheduling node :A - Active, S - Standby
 default  class -5 applet events
   no.      job id p s  status            time of event          event type name
    1          33 N A running  Sun Dec30 01:00:00 2018               Timer Schedule_Backup
    2          46 N A running  Sat Jan12 01:00:00 2019               Timer Schedule_Backup
    3         645 N A running  Wed Sep09 01:00:00 2020               Timer Schedule_Backup
    4         729 N A running  Mon Nov30 01:00:00 2020               Timer Schedule_Backup
    5         730 N A running  Tue Dec01 01:00:00 2020               Timer Schedule_Backup

So applet stopped to run after December 01 2020.

event manager scheduler clear policy 33 and event manager scheduler clear all have no positive effect. All five jobs remain in status "running".

Can somebody advise how to clear scheduler threads without restarting whole switch?

[balaji.bandi]

as per the ouput they are not pending, they are scheduled to run.

you can view pendings :

show event manager policy pending

you want to clear  5

event manager scheduler clear policy 5

[gpn-spb-cisco-access]

For now the output of "sh event manager policy pending" is:

Key: p - Priority        :L - Low, H - High, N - Normal, Z - Last
     s - Scheduling node :A - Active, S - Standby
 default  class -2 applet events
   no.      job id p s  status            time of event          event type name
    1         784 N A    pend  Mon Jan18 12:00:00 2021               Timer Schedule_Backup2
        maxrun = 20
    2         785 N A    pend  Tue Jan19 01:00:00 2021               Timer Schedule_Backup
        maxrun = 20

I succeeded to clear both of it by executing event manager scheduler clear policy 784 and event manager scheduler clear policy 785

But i still have five active jobs listed in sh event manager policy pending that prevent the new jobs from executing:

Key: p - Priority        :L - Low, H - High, N - Normal, Z - Last
     s - Scheduling node :A - Active, S - Standby
 default  class -5 applet events
   no.      job id p s  status            time of event          event type name
    1          33 N A running  Sun Dec30 01:00:00 2018               Timer Schedule_Backup
    2          46 N A running  Sat Jan12 01:00:00 2019               Timer Schedule_Backup
    3         645 N A running  Wed Sep09 01:00:00 2020               Timer Schedule_Backup
    4         729 N A running  Mon Nov30 01:00:00 2020               Timer Schedule_Backup
    5         730 N A running  Tue Dec01 01:00:00 2020               Timer Schedule_Backup

If i execute event manager run Schedule_Backup i get this:

N5K01# sh event manager policy pending

Key: p - Priority        :L - Low, H - High, N - Normal, Z - Last
     s - Scheduling node :A - Active, S - Standby
 default  class -1 applet events
   no.      job id p s  status            time of event          event type name
    1         787 N A    pend  Tue Jan19 11:42:12 2021                     Schedule_Backup
        maxrun = 20

The new job is added to pending jobs and will never be executed since five jobs are hanged in "active" state and can not be stopped.

[balaji.bandi]

since this is backup some where stuck, stop the event manager and clear the task and re-run.

also consider email success or failure of task  - also make sure your track the task before run the same task again, if this was not completed.

[gpn-spb-cisco-access]

Can you guide me how to stop event manager and then clear tasks?

In privileged exec mode "event manager ?" prompt results only this:

N5K01# event manager ?
  clear      Clear event manager information
  run        Trigger/run an event manager policy manually
  scheduler  Embedded Event Manager scheduler