Showing results for 
Search instead for 
Did you mean: 

Troubleshooting OSPF Flaps with EEM Script




    Embedded Event Manager (EEM) is a powerful tool which monitor events and take informational, corrective, or any desired action when the monitored events occur. Capturing the state of the router during such situations can be invaluable in taking immediate recovery actions and gathering information to perform root-cause analysis.



    Understanding of OSPF Routing Protocol


    This document shows how to collect information from Router when there is an OSPF flap and the message seen is "Neighbor Down: dead timer expired". This means that the Router did not receive OSPF hello from the neighbor for the dead interface (120) seconds and then dropped the OSPF neighborship.


    For example,

    %OSPF-5-ADJCHG: Process 10, Nbr on GigabitEthernet0/1 from FULL to DOWN, Neighbor Down: Dead timer expired


    To troubleshoot this situation an EEM is configured to check for an OSPF-neighbor-down syslog message; when it occurs, it executes the following command and saves the output as a text file "GET_CLI_OUTPUT.txt" in the flash memory.


    This script will collect CPU, memory, interfaces and OSPF outputs and will also run a unicast ping to the neighbor as well as multicast check with neighbor. The script is configured in the config mode.


    Logging buffered 80000

    event manager scheduler applet thread class default number 1

    event manager applet OSPF-FLAP

    event syslog pattern "FULL to DOWN, Neighbor Down: Dead timer expired"

    action 2.0 cli command "enable"

    action 3.0 cli command "sh proc cpu sorted | append flash:GET_CLI_OUTPUT.txt"

    action 3.5 cli command "show interface | append flash:GET_CLI_OUTPUT.txt"

    action 3.10 cli command "show proc memory | append flash:GET_CLI_OUTPUT.txt"

    action 3.15 cli command "sh ip ospf nei | append flash:GET_CLI_OUTPUT.txt"

    action 3.20 cli command "sh ip ospf int br | append flash:GET_CLI_OUTPUT.txt"

    action 3.25 cli command "ping rep 10 time 1"

    action 3.25 cli command "ping rep 10 time 1"

    action 3.30 cli command "ping rep 10 time 1"

    action 5.0 syslog msg "$_cli_result" priority 5



    Note: To display the content of a file in a compact Flash memory card, use the more flash: command.

    Router# more flash: GET_CLI_OUTPUT.txt




    1. Append command was integrated in CISCO IOS Release 12.2T

    2. In case AAA is configured, user would require a Username/Password for CLI access.

    Verification Commands


    1. dir flash:
    2. more flash:GET_CLI_OUTPUT.txt




    Get CLI OUTPUT.bmp



    Troubleshooting OSPF

    EEM Configuration for Cisco Integrated Services Router Platforms



    I'm trying to do similar EEM applet on 6800, IOS 15.1(2)SY1. I believe append is not supported, so I've done it like this:


    event manager applet OSPF_FLAPS authorization bypass
     event syslog pattern "FULL to DOWN, Neighbor Down: Dead timer expired"
     action 1.0 file open LOGFILE bootdisk:OSPF_FLAPS_OUTPUT.txt a+
     action 1.1 cli command "enable"
     action 1.2 cli command "term length 0"
     action 1.3 cli command "show clock"
     action 2.0 cli command "sh proc cpu sorted"
     action 2.1 file puts LOGFILE "$_cli_result"
     action 3.0 cli command "ping 10.x.x.4 rep 10 time 1 source vlan 666"
     action 3.1 file puts LOGFILE "$_cli_result"
     action 4.0 cli command "ping 10.x.x.2 rep 10 time 1 source vlan 666"
     action 4.1 file puts LOGFILE "$_cli_result"
     action 5.0 cli command "ping 10.x.x.5 rep 10 time 1 source vlan 666"
     action 5.1 file puts LOGFILE "$_cli_result"
     action 6.0 cli command "sh ip ospf nei"
     action 6.1 file puts LOGFILE "$_cli_result"
     action 7.0 cli command "ping rep 10 time 1 source vlan 666"
     action 7.1 file puts LOGFILE "$_cli_result"
     action 8.0 cli command "ping rep 10 time 1 source vlan 666"
     action 8.1 file puts LOGFILE "$_cli_result"
     action 9.0 cli command "sh ip ospf int br"
     action 9.1 file puts LOGFILE "$_cli_result"
     action 9.2 file close LOGFILE

    After event is triggered by configured syslog message, there are two files created on bootdisk:



    6        92762 Jan 8 2018 14:38:28 +00:00 OSPF_FLAPS_OUTPUT.txt
    8        92762 Jan 8 2018 14:38:28 +00:00 OSPF_FLAPS_OUTPUT.txt

    The problem is I don't have complete output. Last entry in the files is made by action 7.0. There is no actions 8.0 and 9.0. Is there any limit on the file size or something else?







    May someone can help me.

    I have got a problem with OSPF Routing that the OSPF suddenly down for a while, and then back to normal to up state again.

    I was tried to investigated the root cause from show log and show tech-support, but there is no information about these issue.