MPLS VPN Services are one of the most common services provided by Service Providers. Because of its wide and huge deployments, there is a need to have deep troubleshooting knowledge to fix the issues that occur in MPLS VPN deployment.
Today, we shall see how to troubleshoot Basic Layer-3 MPLS VPN Services. For this, lets consider the below topology:
Configuration
Config on PE1: ============== ip vrf ABC rd 1:1 route-target export 1:1 route-target import 1:1 ! interface Loopback0 ip address 1.1.1.1 255.255.255.255 ! interface Ethernet0/0 ip vrf forwarding ABC ip address 192.168.10.1 255.255.255.252 ! interface Ethernet1/0 ip address 12.12.12.1 255.255.255.252 ip ospf network point-to-point mpls ip ! router ospf 100 router-id 1.1.1.1 network 1.1.1.1 0.0.0.0 area 0 network 12.12.12.1 0.0.0.0 area 0 ! router bgp 100 bgp router-id 1.1.1.1 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor 3.3.3.3 remote-as 100 neighbor 3.3.3.3 update-source Loopback0 ! address-family ipv4 exit-address-family ! address-family vpnv4 neighbor 3.3.3.3 activate neighbor 3.3.3.3 send-community both exit-address-family ! address-family ipv4 vrf ABC redistribute connected neighbor 192.168.10.2 remote-as 65535 neighbor 192.168.10.2 activate exit-address-family ! Config on P-1: ============== interface Loopback0 ip address 2.2.2.2 255.255.255.255 ! interface Ethernet0/0 ip address 23.23.23.1 255.255.255.252 ip ospf network point-to-point mpls ip ! interface Ethernet1/0 ip address 12.12.12.2 255.255.255.252 ip ospf network point-to-point mpls ip ! router ospf 100 router-id 2.2.2.2 network 2.2.2.2 0.0.0.0 area 0 network 12.12.12.2 0.0.0.0 area 0 network 23.23.23.1 0.0.0.0 area 0 ! Config PE2: =========== ip vrf ABC rd 100:1 route-target export 1:1 route-target import 1:1 ! interface Loopback0 ip address 3.3.3.3 255.255.255.255 ! interface Ethernet0/0 ip address 23.23.23.2 255.255.255.252 ip ospf network point-to-point mpls ip ! interface Ethernet1/0 ip vrf forwarding ABC ip address 192.168.20.1 255.255.255.252 ! router ospf 100 router-id 3.3.3.3 network 3.3.3.3 0.0.0.0 area 0 network 23.23.23.2 0.0.0.0 area 0 ! router bgp 100 bgp router-id 3.3.3.3 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor 1.1.1.1 remote-as 100 neighbor 1.1.1.1 update-source Loopback0 ! address-family ipv4 exit-address-family ! address-family vpnv4 neighbor 1.1.1.1 activate neighbor 1.1.1.1 send-community both exit-address-family ! address-family ipv4 vrf ABC redistribute connected neighbor 192.168.20.2 remote-as 65000 neighbor 192.168.20.2 activate exit-address-family ! Config on CE1: ============= interface Loopback0 ip address 100.1.1.1 255.255.255.255 ! interface Ethernet0/0 ip address 192.168.10.2 255.255.255.252 ! router bgp 65535 bgp router-id 100.1.1.1 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor 192.168.10.1 remote-as 100 ! address-family ipv4 network 100.1.1.1 mask 255.255.255.255 neighbor 192.168.10.1 activate exit-address-family ! Config on CE2: ============== interface Loopback0 ip address 100.2.2.1 255.255.255.255 ! interface Ethernet1/0 ip address 192.168.20.2 255.255.255.252 ! router bgp 65000 bgp router-id 100.2.2.1 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor 192.168.20.1 remote-as 100 ! address-family ipv4 network 100.2.2.1 mask 255.255.255.255 neighbor 192.168.20.1 activate exit-address-family !
Verification
Output on CE1:
==============
CE1#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is not set
100.0.0.0/32 is subnetted, 2 subnets
C 100.1.1.1 is directly connected, Loopback0
B 100.2.2.1 [20/0] via 192.168.10.1, 00:42:16
192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.10.0/30 is directly connected, Ethernet0/0
L 192.168.10.2/32 is directly connected, Ethernet0/0
192.168.20.0/30 is subnetted, 1 subnets
B 192.168.20.0 [20/0] via 192.168.10.1, 00:23:50
CE1#ping 100.2.2.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.2.2.1, timeout is 2 seconds:
Packet sent with a source address of 100.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
CE1#
The above output shows that the routing table on CE1 has the routes from remote CE router CE2. Also, we can see that there is end to end reachability between CE1 and CE2 router loopbacks. Further looking at the output on PE1 (below), the VRF routing table of PE1 (VRF ABC) is populated with necessary prefixes.
PE1#show ip route vrf ABC
Routing Table: ABC
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is not set
100.0.0.0/32 is subnetted, 2 subnets
B 100.1.1.1 [20/0] via 192.168.10.2, 01:06:49
B 100.2.2.1 [200/0] via 3.3.3.3, 00:44:36
192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.10.0/30 is directly connected, Ethernet0/0
L 192.168.10.1/32 is directly connected, Ethernet0/0
192.168.20.0/30 is subnetted, 1 subnets
B 192.168.20.0 [200/0] via 3.3.3.3, 00:26:15
PE1#
Troubleshooting
Step1: The first step in troubleshooting MPLS VPN setup is to verify the LSP path between PE to PE. This can be done using two methods
1. MPLS PING
If the MPLS ping goes through from PE to PE loopback, then it would confirm that the LSP (Label Switched Path) is complete and there is no problem with it.
PE1# ping mpls ipv4 3.3.3.3 255.255.255.255
Sending 5, 72-byte MPLS Echos to 3.3.3.3/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'l' - Label switched with FEC change, 'd' - see DDMAP for return code,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Total Time Elapsed 3 ms
PE1#
2. PE to PE VRF PING
You can also perform a PE to PE to the VRF's IP Address:
PE1# ping vrf ABC 192.168.20.1 source 192.168.10.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds: Packet sent with a source address of 192.168.10.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms PE1#
This would again confirm the same thing that the LSP is complete in the Service Provider Core.
Step2: Verifying the VPN and IGP label along the path. For this, the first step is check the VPN label.
1. Verifying VPN Label:
PE1#show mpls forwarding vrf ABC Local Outgoing Prefix Bytes Label Outgoing Next Hop Label Label or Tunnel Id Switched interface 19 No Label 100.1.1.1/32[V] 2136 Et0/0 192.168.10.2 20 No Label 192.168.10.0/30[V] \ 570 aggregate/ABC PE1#
The local label value shown above i.e. 19 and 20 are nothing but the VPN labels. show mpls forwarding vrf command helps with checking the VPN label. There is another way to check the VPN labels and that is using BGP. Since BGP allocates the VPN label, you can verify the VPN labels for the prefixes (local as well as remote).
PE1# show ip bgp vpnv4 all 100.1.1.1
BGP routing table entry for 1:1:100.1.1.1/32, version 2
Paths: (1 available, best #1, table ABC)
Advertised to update-groups:
2
Refresh Epoch 1
65535
192.168.10.2 (via vrf ABC) from 192.168.10.2 (100.1.1.1)
Origin IGP, metric 0, localpref 100, valid, external, best
Extended Community: RT:1:1
mpls labels in/out 19/nolabel
rx pathid: 0, tx pathid: 0x0
PE1#
All the local as well as remove VPN labels can be verified using the above command. There is another easy way to look at the VPN labels:
PE1# show ip bgp vpnv4 all labels Network Next Hop In label/Out label Route Distinguisher: 1:1 (ABC) 100.1.1.1/32 192.168.10.2 19/nolabel 100.2.2.1/32 3.3.3.3 nolabel/19 192.168.10.0/30 0.0.0.0 20/nolabel(ABC) 192.168.20.0/30 3.3.3.3 nolabel/20 Route Distinguisher: 100:1 100.2.2.1/32 3.3.3.3 nolabel/19 192.168.20.0/30 3.3.3.3 nolabel/20 PE1#
In the above output, the label values highlighted on the left are the local labels (In Label) where as label values highlighted in the right (Out Label) are the remote prefix VPN label. You can filter the labels based on the RD value as well using "show ip bgp vpnv4 rd <rd_value> labels".
2. Verifying IGP labels in the path
For for the traffic destined to the remote prefix learnt from remote PE - PE2, we can check the vrf Routing table and then check the IGP labels for the same along the path:
PE1# show ip route vrf ABC 100.2.2.1 Routing Table: ABC Routing entry for 100.2.2.1/32 Known via "bgp 100", distance 200, metric 0 Tag 65000, type internal Last update from 3.3.3.3 01:40:38 ago Routing Descriptor Blocks: * 3.3.3.3 (default), from 3.3.3.3, 01:40:38 ago Route metric is 0, traffic share count is 1 AS Hops 1 Route tag 65000 MPLS label: 19 MPLS Flags: MPLS Required PE1# PE1# show mpls forwarding 3.3.3.3 Local Outgoing Prefix Bytes Label Outgoing Next Hop Label Label or Tunnel Id Switched interface 16 17 3.3.3.3/32 0 Et1/0 12.12.12.2 PE1#
In above output, the next-hop for prefix learnt (100.2.2.1/32) from remote CE router CE2 is 3.3.3.3 which is the loopback IP of PE2 router. The LFIB (Label Forwarding Information Base) for 3.3.3.3 shows that outgoing label as 17. This label 17 is learnt from the P-1 router where it should be the local label.
P-1# show mpls forwarding 3.3.3.3 Local Outgoing Prefix Bytes Label Outgoing Next Hop Label Label or Tunnel Id Switched interface 17 Pop Label 3.3.3.3/32 476193 Et0/0 23.23.23.2 P-1#
Since the prefix is present on PE2 router itself, it will send a Pop label (implicit-null label) to P-1 router which is seen as the outgoing label on P-1 router for the PE2 loopback.
Step 3. Verifying CEF
On Cisco platform, CEF takes care of forwarding. Thus, its important to verify that the CEF is programmed correctly.
PE1# show ip cef vrf ABC 100.2.2.1 det 100.2.2.1/32, epoch 0, flags [rib defined all labels] recursive via 3.3.3.3 label 19 nexthop 12.12.12.2 Ethernet1/0 label 17 PE1#
In the cef output for the remote prefix, we can see that label 19 is the VPN label where as 17 is the IGP label and we can also see the respective next-hop value. We can also check the Service Provider core for the IGP labels and forwarding information using the above method.
Step 4. Traceroute
Traceroute can be used for checking the labelling and figure where a possibly problem could be.
PE1#traceroute vrf ABC 100.2.2.1 numeric Type escape sequence to abort. Tracing the route to 100.2.2.1 VRF info: (vrf in name/id, vrf out name/id) 1 12.12.12.2 [MPLS: Labels 17/19 Exp 0] 1 msec 1 msec 0 msec 2 192.168.20.1 [MPLS: Label 19 Exp 0] 0 msec 0 msec 1 msec 3 192.168.20.2 0 msec * 1 msec PE1#
We can check the LSP path as well using the trace from PE to PE:
PE1# trace mpls ipv4 3.3.3.3 255.255.255.255 Tracing MPLS Label Switched Path to 3.3.3.3/32, timeout is 2 seconds Codes: '!' - success, 'Q' - request not sent, '.' - timeout, 'L' - labeled output interface, 'B' - unlabeled output interface, 'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch, 'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry, 'P' - no rx intf label prot, 'p' - premature termination of LSP, 'R' - transit router, 'I' - unknown upstream index, 'l' - Label switched with FEC change, 'd' - see DDMAP for return code, 'X' - unknown return code, 'x' - return code 0 Type escape sequence to abort. 0 12.12.12.1 MRU 1500 [Labels: 17 Exp: 0] L 1 12.12.12.2 MRU 1500 [Labels: implicit-null Exp: 0] 1 ms ! 2 23.23.23.2 1 ms PE1#
Hope the above steps helps you troubleshoot Basic MPLS VPN deployments. Even for complex MPLS VPN deployments like Inter-AS MPLS VPN or CSC (Carrier Supporting Carrier), these steps can serve useful.
1 Comment