Introduction:
This document describes about implementing an EEM script where a MPLS traffic engineering tunnel flaps randomly and the network administrator or the TAC engineer would like to gather some specific information while the flap happens.
Prerequisites:
This article expects to the audience to be familiar with basic MPLS and traffic engineering concepts and configuration.
This EEM script will capture required logs and debug messages when there is a random MPLS traffic engineering tunnel flap. This script can he extended according to user needs.
Requirements:
This EEM script needs version EEM version 3.0 or higher else some functionality might now work as intended.
Cisco recommends that you have knowledge of these topics:
• MPLS
• MPLS traffic-engineering concepts and configurations
• Basic EEM applet scripting
Components Used:
The information in this document is based on 7600/6500 platform. But if the EEM functionality is supported on other platforms and the show commands are supported, then this script can be used but it might be tested for accuracy on them.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Related Products {Optional}
Conventions
Refer to Cisco Technical Tips Conventions for information on document conventions.
Problem:
In a service provider environment running multiple MPLS traffic engineering tunnels sometimes the tunnels will flap and restore itself in few minutes before a network administrator can figure out what the tunnels flapped. The information gathered when the problem is happening is very critical to determine the root-cause of the issue. The EEM script gives a sample script that can be used to capture the information that will help us identify the potential cause of the problem. This script does not require any manual intervention when the problem is happening as it is completely automated and triggers only when the problem occurs based on the console log messages that are thrown during the tunnel flap.
Solution 1:
The EEM script functionality is as below.
* When EEM sees tunnel up/down log messages, it will enable set the debugs.
* Capture the show outputs of various commands for 'n' times ('n' can be set in variable 'loop') and then let the script wait for 60 seconds to collect more debug and netdr outputs. All the outputs are logged into a file in disk0: by name TAC.txt
* Undebug the debug commands enabled by scrit, shut the 4 TE tunnels in router and remove the EEM script as the script will keep running if there are multiple flaps seen.
* You have to modify the interface names, modify tunnel interface numbers, IP address of tunnel destination in show ip route command and other places as necessary.
Limitation:
This script has no support of identifying which tunnel was flapping and shutting only that specific tunnel.
If any other tunnel was flapping for instance, this script has not support to identify if it is MPLS TE tunnel and then start the script. However, the syslog pattern can be modified if any particular tunnel is flapping.
Script:
event manager applet TUNNEL_FLAP
event syslog pattern "LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel" maxrun 36000
action 1.0 cli command "enable"
action 1.1 cli command "access-list 199 permit ip host 10.1.0.2 host 10.1.0.5"
action 1.2 cli command "debug mpls traffic-eng tunnels state detail 199"
action 1.3 cli command "debug mpls traffic-eng tunnel error detail"
action 1.4 cli command "debug mpls traffic-eng tunnels fast-reroute detail 199"
action 1.5 cli command "debug mpls traffic-eng tunnels signalling detail 199"
action 1.6 cli command "debug ip rsvp signalling"
action 1.7 cli command "debug ip rsvp message"
action 1.9 set loop1 "0"
action 2.0 while $loop1 le 1
action 2.1 cli command "show mpls traffic-eng tunnel tunnel 505 | append disk0:TAC.txt"
action 2.2 cli command "show mpls traffic-eng tunnel tunnel 506 | append disk0:TAC.txt"
action 2.3 cli command "show mpls traffic-eng tunnel tunnel 507 | append disk0:TAC.txt"
action 2.4 cli command "show mpls traffic-eng tunnel tunnel 508 | append disk0:TAC.txt"
action 2.5 cli command "show ip rsvp neighbor | append disk0:TAC.txt"
action 2.6 cli command "show ip rsvp counter summary | append disk0:TAC.txt"
action 2.7 cli command "show ip rsvp counter state teardown | append disk0:TAC.txt"
action 2.8 cli command "show proc cpu sorted | append disk0:TAC.txt"
action 2.9 cli command "show proc cpu history | append disk0:TAC.txt"
action 3.0 cli command "show ip route 10.1.0.5 | append disk0:TAC.txt"
action 3.1 cli command "show ip cef 10.1.0.5 | append disk0:TAC.txt"
action 3.2 cli command "show logging | append disk0:TAC.txt"
action 3.3 increment loop1
action 3.4 end
action 3.5 wait 60
action 3.6 cli command "undebug all"
action 3.8 cli command " configure terminal"
action 3.9 cli command " no access-list 199 permit ip host 10.1.0.2 host 10.1.0.5"
action 4.0 cli command " inter range tunnel 505 - 508"
action 4.1 cli command " shutdown"
action 4.2 cli command "no event manager applet TUNNEL_FLAP"
Sample logs collected are attached.
