Here are two documents on mpls troubleshooting. The info is pasted below, in case you can't access.
http://www.cisco.com/c/en/us/support/docs/multiprotocol-label-switching-mpls/mpls/12492-mpls-tsh.html
http://www.cisco.com/c/en/us/support/docs/multiprotocol-label-switching-mpls/mpls/13734-mpls-vpn-tsh.html
Introduction
This document describes how to troubleshoot Multiprotocol Label Switching (MPLS).
Prerequisites
Requirements
Readers of this document should have knowledge of this topic:
MPLS basics
Components Used
This document is based on the Configuring Basic MPLS Using OSPF sample configuration and presumes that you have configured these elements:
IP address and a routing protocol such as Open Shortest Path First Protocol (OSPF Protocol) or Intermediate System-to-Intermediate System Protocol (IS-IS Protocol)
Cisco Express Forwarding (CEF) or distributed CEF switching on all routers
General MPLS or tag switching on all routers
MPLS or tag switching on all required interfaces
If you have doubts about which hardware or Cisco IOS® Software releases support MPLS, refer to the Software Advisor.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.
Troubleshoot Procedures
This section contains several MPLS troubleshoot procedures.
Verify That Routing Protocol Runs
Issue the show ip protocols command in order to display the parameters and current state of the active routing protocol process:
Pomerol# show ip protocols Routing Protocol is "ospf 1" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Router ID 10.10.10.3 Number of areas in this router is 1. 1 normal 0 stub 0 nssa Maximum path: 4 Routing for Networks: 10.1.1.0 0.0.0.255 area 9 10.10.10.0 0.0.0.255 area 9 Routing Information Sources: Gateway Distance Last Update 10.10.10.2 110 10:41:55 10.10.10.3 110 10:41:55 10.10.10.1 110 10:41:55 10.10.10.6 110 10:41:55 10.10.10.4 110 10:41:55 10.10.10.5 110 10:41:55 Distance: (default is 110)
Ensure that the protocol routes for the MPLS network and all neighbors are present. You can also issue the show ip route command in order to verify the routing table:
Pomerol# show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - ISIS level-1, L2 - ISIS level-2, ia - ISIS inter area
* - candidate default, U - per-user static route, o - ODR
Gateway of last resort is 10.200.28.1 to network 0.0.0.0
10.0.0.0/8 is variably subnetted, 13 subnets, 3 masks
C 10.1.1.8/30 is directly connected, Serial0/1.2
O 10.1.1.12/30 [110/390] via 10.1.1.5, 15:26:38, Serial0/1.1
O 10.10.10.2/32 [110/196] via 10.1.1.10, 15:26:38, Serial0/1.2
C 10.10.10.3/32 is directly connected, Loopback0
O 10.1.1.0/30 [110/390] via 10.1.1.5, 15:26:38, Serial0/1.1
[110/390] via 10.1.1.10, 15:26:38, Serial0/1.2
O 10.10.10.1/32 [110/196] via 10.1.1.5, 15:26:38, Serial0/1.1
O 10.10.10.6/32 [110/98] via 10.1.1.22, 15:26:38, Serial0/1.3
O 10.10.10.4/32 [110/391] via 10.1.1.5, 15:26:38, Serial0/1.1
C 10.1.1.4/30 is directly connected, Serial0/1.1
C 10.1.1.20/30 is directly connected, Serial0/1.3
If the routers or routes are not present, investigate the routing protocol process. Refer to the OSPF Support Page in order to investigate the routing protocol process.
Verify CEF Switching
Issue the show ip cef summary command in order to display specific entries in the Forwarding Information Base (FIB) with IP address information as a basis. This output shows Normal status:
Pomerol# show ip cef summary IP CEF with switching (Table Version 131), flags=0x0, bits=8 32 routes, 0 reresolve, 0 unresolved (0 old, 0 new) 32 leaves, 18 nodes, 23004 bytes, 125 inserts, 93 invalidations 1 load sharing elements, 336 bytes, 1 references universal per-destination load sharing algorithm, id B642EBCF 1 CEF resets, 6 revisions of existing leaves 6 in-place modifications refcounts: 4909 leaf, 4864 node
Issue the show ip cef and show ip cef interface commands in order to verify CEF status. If CEF has not been enabled, nothing appears:
Pomerol# show ip cef %CEF not running Prefix Next Hop Interface
Refer to the Cisco Express Forwarding Overview if you continue to have problems with the enablement of CEF.
Verify MPLS
Issue the show mpls interfaces command in order to ensure that MPLS is globally enabled. This command also verifies that a Label Distribution Protocol (LDP) runs on the requested interfaces:
Pomerol# show mpls interfaces Interface IP Tunnel Operational (...) Serial0/1.1 Yes (tdp) Yes Yes Serial0/1.2 Yes Yes No Serial0/1.3 Yes (tdp) Yes Yes (...)
| show mpls interfaces Command Output Field Descriptions | |
|---|---|
| Field | Description |
| IP | This field shows that MPLS IP is configured for an interface. The LDP appears in parentheses to the right of the IP status. The LDP is either:
|
| Tunnel | This field indicates the capacity of traffic engineering on the interface. |
| Operational | This field shows the status of the LDP. Note: In the example output, the Operational field is down on Serial0/1.2 because the interface is down. |
Ping the Neighbors
An unlabeled connection must be up between each pair of router neighbors. The routing protocol and the LDP use the unlabeled connection to build the routing table and the label forwarding information base (LFIB).
Pomerol# ping 10.10.10.6 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.10.10.6, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 56/56/60 ms
Verify Label Distribution
Issue the show tag-switching tdp discovery command in order to display the discovered neighbors:
Pomerol# show tag-switching tdp discovery
Local TDP Identifier:
10.10.10.3:0
Discovery Sources:
Interfaces:
Serial0/1.1 (tdp): xmit/recv
TDP Id: 10.10.10.1:0
Serial0/1.2 (tdp): xmit/recv
TDP Id: 10.10.10.2:0
Serial0/1.3 (tdp): xmit/recv
TDP Id: 10.10.10.6:0
In the show tag-switching tdp discovery command output, the use of TDP binds labels with routes. If any of the presumed neighbors is not present and you cannot ping the presumed neighbor, a connectivity problem exists and the LDP cannot run. If LDP runs correctly, it assigns one label per forwarding equivalent class.
Note: If the router ID for the LDP cannot be reached from the global routing table, the neighbor relationship fails to establish.
Verify Label Bindings
Issue the show tag-switching tdp bindings command in order to ensure the assignment of labels to each destination. You can use commands such as the show tag-switching forwarding-table {ip address | prefix} detail command in order to verify the different routes and the labels associated with the routes.
The output that this section shows contains label bindings for 10.10.10.x/32 networks, which are the interfaces of each label switch router (LSR):
Note: There are multiple labels for each LSR. Each label corresponds to a different path.
Pomerol# show tag-switching tdp bindings
(...)
tib entry: 10.10.10.1/32, rev 31
local binding: tag: 18
remote binding: tsr: 10.10.10.1:0, tag: imp-null
remote binding: tsr: 10.10.10.2:0, tag: 18
remote binding: tsr: 10.10.10.6:0, tag: 21
tib entry: 10.10.10.2/32, rev 22
local binding: tag: 17
remote binding: tsr: 10.10.10.2:0, tag: imp-null
remote binding: tsr: 10.10.10.1:0, tag: 19
remote binding: tsr: 10.10.10.6:0, tag: 22
tib entry: 10.10.10.3/32, rev 2
local binding: tag: imp-null
remote binding: tsr: 10.10.10.2:0, tag: 17
remote binding: tsr: 10.10.10.1:0, tag: 20
remote binding: tsr: 10.10.10.6:0, tag: 23
tib entry: 10.10.10.4/32, rev 40
local binding: tag: 20
remote binding: tsr: 10.10.10.1:0, tag: 16
remote binding: tsr: 10.10.10.2:0, tag: 20
remote binding: tsr: 10.10.10.6:0, tag: 24
tib entry: 10.10.10.5/32, rev 44
local binding: tag: 22
remote binding: tsr: 10.10.10.1:0, tag: 17
remote binding: tsr: 10.10.10.2:0, tag: 22
remote binding: tsr: 10.10.10.6:0, tag: 25
tib entry: 10.10.10.6/32, rev 48
local binding: tag: 23
remote binding: tsr: 10.10.10.6:0, tag: imp-null
remote binding: tsr: 10.10.10.1:0, tag: 22
remote binding: tsr: 10.10.10.2:0, tag: 24
(...)
Pomerol# show tag-switching forwarding-table 10.10.10.4 detail
Local Outgoing Prefix Bytes
tag Outgoing Next Hoptag tag or VC or Tunnel Id switched interface
20 16 10.10.10.4/32 0 Se0/1.1 point2point
MAC/Encaps=4/8, MTU=1500, Tag Stack{16}
48D18847 00010000
No output feature configured
Per-packet load-sharing
Verify That Labels Are Set
Use the debug mpls packet command or the MPLS-aware traceroute command functionality in order to make sure that the labels are set.
Pesaro# traceroute 10.10.10.4 Type escape sequence to abort. Tracing the route to 10.10.10.4 1 10.1.1.21 [MPLS: Label 20 Exp 0] 272 msec 268 msec 300 msec 2 10.1.1.5 [MPLS: Label 16 Exp 0] 228 msec 228 msec 228 msec 3 10.1.1.14 92 msec * 92 msec
Introduction
This document shows you how to troubleshoot the Configuring a Basic MPLS VPN document. We recommend you read this sample configuration and view the network diagram before you use this document.
Configuring a Basic MPLS VPN shows a fully functional MPLS backbone network which means provider edge (PE) routers are able to reach each other through the backbone. Refer to the MPLS Verification and Troubleshooting Support Page for information on troubleshooting an MPLS network.
Before establishing an MPLS VPN, you must be able to ping PE router A (10.10.10.4) from PE router B (10.10.10.6) and vice-versa.
Remember that VPN routing/forwarding instance (VRF) names are case sensitive, for example, Customer_A is not the same as customer_a.
Prerequisites
Requirements
Readers of this document should be familiar with:
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Conventions
For more information on document conventions, refer to the Cisco Technical Tips Conventions.
Troubleshooting VRF Configurations
show ip vrf [vrf-name]
The show ip vrf [vrf-name] command shows a summary of all VRFs present on the current router and their associated route-distinguishers and interface(s).
Pesaro# show ip vrf
Name Default RD Interfaces
Customer_A 100:101 Loopback101
Loopback111
Customer_B 100:102 Loopback102
This command allows you to verify:
The configuration of VRFs (and their names).
That each route-distinguisher (RD) is the same at each concerned PE.
show ip vrf [{detail | interfaces}] vrf-name
The show ip vrf [{detail | interfaces}] vrf-name command shows detailed configurations about the VRF.
Pesaro# show ip vrf detail Customer_A
VRF Customer_A; default RD 100:101
Interfaces:
Loopback101 Loopback111
Connected addresses are not in global routing table
Export VPN route-target communities
RT:100:1001
Import VPN route-target communities
RT:100:1001
No import route-map
No export route-map
Pesaro# show ip vrf interfaces
Interface IP-Address VRF Protocol
Loopback101 200.0.6.1 Customer_A up
Loopback111 200.1.6.1 Customer_A up
Loopback102 200.0.6.1 Customer_B up
These commands allow you to verify:
That connected addresses are not in the global routing table.
The routing attributes of each VRF. What is exported on one side should be imported somewhere else.
The interface status (and IP addresses) of interfaces.
Routing Information
Use the same commands you use to verify the global routing table with the extensions shown in this section to verify routing tables or routing protocol databases.
Routing Table
To check the routing table, Add the vrf [vrf-name] extension to the show ip route command to verify the routing table, as shown here:
Pescara# show ip route vrf Customer_A
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - ISIS level-1, L2 - ISIS level-2, ia - ISIS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is not set
B 200.0.6.0/24 [200/0] via 10.10.10.6, 00:42:14
B 200.1.6.0/24 [200/0] via 10.10.10.6, 00:42:14
C 200.0.4.0/24 is directly connected, Loopback101
You can also use the show ip route vrf Customer_A 1.2.3.4 command to verify the destination for a particular address.
BGP
Border Gateway Protocol (BGP) is used between the PE routers and is necessary for inter-site connectivity. In this example, we use internal BGP (iBGP). You can also use external BGP (eBGP) as an external routing protocol for PE-CE route propagation.
You can use these commands to troubleshoot BGP:
show ip bgp neighbors
show ip bgp vpnv4 all (or show ip bgp vpnv4 vrf [VRF name])
show ip bgp vpnv4 vrf VRF name tags (this command is VPN/MPLS specific)
show ip bgp vpnv4 vrf VRF name A.B.C.D
For example:
Pescara# show ip bgp vpnv4 vrf Customer_A BGP table version is 40, local router ID is 10.10.10.4 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 100:101 (default for vrf Customer_A) *>i200.0.6.0 10.10.10.6 0 100 0 ? *> 200.0.4.0 0.0.0.0 0 32768 ? *>i200.1.6.0 10.10.10.6 0 100 0 ?
Refer to the BGP Support Pages for more information on troubleshooting BGP issues.
PE-CE Routing Protocol
If the routing protocol used on the customer side isn't BGP, you can use traditional show commands, and apply them to the correct VRF.
Use the show ip rip database vrf [VRF name] command if you use Routing Information Protocol (RIP). For example:
Alcazaba# show ip rip database vrf vrf101
0.0.0.0/0 auto-summary
0.0.0.0/0
[2] via 150.150.0.2, 00:00:12, Ethernet1/1
6.0.0.0/8 auto-summary
6.6.6.6/32 redistributed
[1] via 223.0.0.21,
7.0.0.0/8 auto-summary
7.7.7.0/24
[1] via 150.150.0.2, 00:00:12, Ethernet1/1
10.0.0.0/8 auto-summary
10.0.0.0/8 redistributed
[1] via 125.2.2.2,
10.0.0.0/16
[1] via 150.150.0.2, 00:00:12, Ethernet1/1
10.200.8.0/22
Use the show ip ospf [process-id area-id] database command and specify the correct process number if you use OSPF. For example:
Alcazaba# show ip ospf 2 database
OSPF Router with ID (222.0.0.10) (Process ID 2)
Router Link States (Area 1)
Link ID ADV Router Age Seq# Checksum Link count
222.0.0.1 222.0.0.1 1364 0x80000013 0x7369 3
222.0.0.10 222.0.0.10 1363 0x80000002 0xFEFE 2
Net Link States (Area 1)
Link ID ADV Router Age Seq# Checksum
150.150.0.1 222.0.0.10 1363 0x80000001 0xEC6D
Summary Net Link States (Area 1)
Link ID ADV Router Age Seq# Checksum
6.6.6.6 222.0.0.10 1328 0x80000001 0x4967
69.69.0.0 222.0.0.10 1268 0x80000001 0x2427
222.0.0.3 222.0.0.10 1328 0x80000001 0xEEF7
222.0.0.30 222.0.0.10 1268 0x80000001 0x7B5A
This command allows you to verify:
If the routing table is correct (from a customer point of view), or what is missing from the routing table.
That BGP is up and working (or you can see which neighbor is missing).
Labels
MPLS VPN uses a two-level label stack. One of the labels is used to identify the VRF and is set up between the two PEs. The other label (on the top of the stack) is the "backbone" label, set up by the standard MPLS network.
You can use the traceroute VRF [vrf-name] A.B.C.B command to verify transport labels.
Note: This command only works with a MPLS-aware traceroute, if the backbone routers are configured to propagate and generate IP Time to Live (TTL) information. Refer to the documentation on the mpls ip propagate-ttl command for more information.
Pesaro# traceroute vrf Customer_B 200.0.4.1 Type escape sequence to abort. Tracing the route to 200.0.4.1 1 10.1.1.21 [MPLS: Labels 25/28 Exp 0] 464 msec 280 msec 308 msec 2 10.1.1.5 [MPLS: Labels 22/28 Exp 0] 236 msec 572 msec 228 msec 3 200.0.4.1 108 msec * 100 msec
The absence of 10.1.1.14 in this traceroute is normal due to the MPLS/VPN architecture.
You can use the show ip bgp vpnv4 all tags command to obtain more precise output, like the labels table for a particular VRF, for example:
Pescara# show ip bgp vpnv4 all tags Network Next Hop In tag/Out tag Route Distinguisher: 100:101 (Customer_A) 200.0.6.0 10.10.10.6 notag/28 200.0.4.0 0.0.0.0 16/aggregate(Customer_A) 200.1.6.0 10.10.10.6 notag/29 Route Distinguisher: 100:102 (Customer_B) 200.0.6.0 10.10.10.6 notag/30 200.0.4.0 0.0.0.0 28/aggregate(Customer_B)
You can also use the traditional show ip cef command:
Pescara# show ip cef vrf Customer_B detail
IP CEF with switching (Table Version 10), flags=0x0
8 routes, 0 reresolve, 0 unresolved (0 old, 0 new)
46 leaves, 51 nodes, 54640 bytes, 361 inserts, 315 invalidations
0 load sharing elements, 0 bytes, 0 references
universal per-destination load sharing algorithm, id F968AD29
5 CEF resets, 38 revisions of existing leaves
refcounts: 1400 leaf, 1392 node
Adjacency Table has 2 adjacencies
0.0.0.0/32, version 0, receive
200.0.6.0/24, version 9, cached adjacency to Serial0/1.1
0 packets, 0 bytes
tag information set
local tag: VPN-route-head
fast tag rewrite with Se0/1.1, point2point, tags imposed: {20 30}
via 10.10.10.6, 0 dependencies, recursive
next hop 10.1.1.13, Serial0/1.1 via 10.10.10.6/32
valid cached adjacency
tag rewrite with Se0/1.1, point2point, tags imposed: {20 30}
200.0.4.0/24, version 6, attached, connected
0 packets, 0 bytes
tag information set
local tag: 28
via Loopback102, 0 dependencies
valid discard adjacency
tag rewrite with , , tags imposed: {}
200.0.4.0/32, version 4, receive
200.0.4.1/32, version 3, receive
200.0.4.255/32, version 5, receive
224.0.0.0/24, version 2, receive
255.255.255.255/32, version 1, receive
This command allows you to verify:
That labels are effectively used.
That a stack of (at least) two labels is used for VPN destinations.
Test
You can use the ping command to verify that the VRF works, but if you are on a PE router, you must indicate the specific VRF name.
Pescara# ping vrf Customer_A 200.0.6.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 200.0.6.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 176/264/576 ms
