One of the biggest challenges in troubleshooting is understanding expected behavior. If we don't know what it is supposed to look like in the working state, how can we tell if we are in a broken state?
In this document we will explore label operations hop-by-hop through three common topologies for Traffic Engineering tunnels. These examples carry customer pseudowires (l2vpn) but the same concepts would apply for L3VPN as well.
PE to PE Tunnel
For our first example we have a pseudowire between PE2 and PE4. There is also a pair of TE tunnels that run end-to-end between PE2 and PE4.
Let's look at the label operations from PE2 to PE4
First, we verify the labels on the pseudowire
PE2#show mpls l2 vc detail
Local interface: Et0/1 up, line protocol up, Ethernet up
Destination address: 14.14.14.14, VC ID: 24, VC status: up
Output interface: Tu0, imposed label stack {41 31}
Preferred path: Tunnel0, active
Default path: ready
Next hop: point2point
We can see that two labels get imposed: The LDP label, 41 and the Pseudowire Label, 31
Since we are using a TE Tunnel (as you can see above, "Output interface: Tu0") the LDP label will be called the "TE Label" or "Tunnel Label".
We can dive a little deeper into the tunnel and see the Tunnel Instance, Outgoing Tunnel Label and exact path
PE2#show mpls traffic-eng tunnels tunnel 0
Name: PE2_t0 (Tunnel0) Destination: 14.14.14.14
Status:
Admin: up Oper: up Path: valid Signalling: connected
path option 1, type dynamic (Basis for Setup, path weight 50)
InLabel : -
OutLabel : Ethernet0/0, 41
Next Hop : 2.0.0.2
RSVP Signalling Info:
Src 22.22.22.22, Dst 14.14.14.14, Tun_Id 0, Tun_Instance 203
RSVP Path Info:
My Address: 2.0.0.12
Explicit Route: 2.0.0.2 4.0.0.2 4.0.0.4 6.0.0.4
6.0.0.6 8.0.0.6 8.0.0.8 84.0.0.8
84.0.0.4 14.14.14.14
Knowing that label 41 will be on the top of the label stack (remember the label output is read {top->bottom}) we can look for this label on P2 (the next hop according to the Explicit Route).
P2#show mpls forwarding label 41
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
41 28 22.22.22.22 0 [203] \
5610 Et0/0 4.0.0.4
In this output we can see the local label (the label we assigned and sent upstream to the PE), the outgoing label (the label we need to use to go downstream to the next hop) and the Tunnel ID, which matches the Tunnel Instance we saw on the head-end.
If we needed to find out more information about this specific tunnel from the P2 router, we can do that too. By asking P2 about the tunnel sourced from PE2 (22.22.22.22) and destined for PE4 (14.14.14.14)
P2#show mpls traffic-eng tunnel source-id 22.22.22.22 destination 14.14.14.14
P2P TUNNELS/LSPs:
LSP Tunnel PE2_t0 is signalled, connection is up
InLabel : Ethernet0/1, 41
Prev Hop : 2.0.0.12
OutLabel : Ethernet0/0, 28
Next Hop : 4.0.0.4
RSVP Signalling Info:
Src 22.22.22.22, Dst 14.14.14.14, Tun_Id 0, Tun_Instance 203
RSVP Path Info:
My Address: 4.0.0.2
Explicit Route: 4.0.0.4 6.0.0.4 6.0.0.6 8.0.0.6
8.0.0.8 84.0.0.8 84.0.0.4 14.14.14.14
Now we continue down the path and look at the next P routers and verify label information
P4#show mpls forwarding label 28
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
28 40 22.22.22.22 0 [203] \
46841 Et0/0 6.0.0.6
P6#show mpls forwarding label 41
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
40 144 22.22.22.22 0 [203] \
83918 Et0/1 6.0.0.4
P8#show mpls forwarding label 40
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
144 Pop Label 22.22.22.22 0 [203] \
47095 Et0/0 84.0.0.4
Now on P8, the Penultimate hop, we see that our outgoing label is pop. With Traffic Engineering, just like L2 or L3VPN, we perform PHP and will pop the top label on the second to last (Penultimate) router. Now, to know what label we expect to see on PE4, we have to think about the label stack. We started with two labels (TE and Pseudowire labels). As we moved through the core we never added any additional labels. This means that the label P8 is removing is only the TE label, leaving the original Pseudowire label. So let's look on PE4 for the Pseudowire label
PE4#show mpls forwarding label 31
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
31 No Label l2ckt(4096) 40483 Et0/0 point2point
Here we see label 31 points to our l2ckt (pseudowire) out Ethernet 0/0.
PE to P Tunnel
Now let's look a scenario where the tunnel terminates on a P router. Again, we'll use a Pseudowire between the CE devices.
When we have a TE tunnel that is not end to end and can't forward unlabeled traffic (BGP Free Core, L2/L3VPN) we must run LDP over the TE tunnel. We'll go into the "why" as soon as we start looking at the label forwarding.
First, on PE1, let's look at the Pseudowire information
PE1#show mpls l2 vc 13 detail
Output interface: Tu0, imposed label stack {23 19 17}
Unlike the previous example, we see three labels being placed on the frame. These labels (again, {Top->Bottom}) are the TE Label (Between PE1 and P1), LDP Label (between PE1 and P7), and Pseudowire Label (between PE1 and PE3).
As the labeled packet arrives on P1, it looks as the top (TE) label and swaps.
P1#show mpls forwarding label 23
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
23 20 11.11.11.11 0 [10] \
549604 Et0/0 3.0.0.3
P3#show mpls forwarding label 20
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
20 22 11.11.11.11 0 [10] \
727533 Et0/0 5.0.0.5
Now on P5, things get interesting. P5 is the penultimate router, so just like in our previous example P5 needs to pop the label.
P5#show mpls forwarding label 22
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
22 Pop Label 11.11.11.11 0 [10] \
755760 Et0/0 7.0.0.7
Again, we think about what our label stack looked like entering P5: {TE Label, LDP Label, Pseudowire Label}. When we pop a single label P5 we expose the LDP Label for P7 to process. Remember, since we turned LDP (mpls ip command) on the Tunnel interface we exchange an LDP label with P7.
P7#show mpls forwarding label 19
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
19 Pop Label 13.13.13.13/32 686357 Et0/0 73.0.0.3
What would happen without this? P5 would still be the Penultimate router and would still pop the top (TE) Label. This would expose the Pseudowire and P7 wouldn't know what to do with the packet. Anytime a TE tunnel terminates on a P router you need to enable "mpls ip" on that Tunnel interface.
We should also note that just like P5, P7 is also a Penultimate router, but for a different LSP. Whereas P5 was the Penultimate router for the TE Tunnel, P7 is the Penultimate router for the Pseudowire. So, like any good Penultimate router does, P7 pops the top (LDP) Label.
Finally, what arrives on PE3 is a single label representing the Pseudowire.
PE3#show mpls forwarding label 17
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
17 No Label l2ckt(4096) 530983 Et0/0 point2point
So the only twist in this scenario is that the TE tunnel ends on a P router, which requires us to add an additional label by enabling "mpls ip" on that tunnel interface.
Out of a Tunnel and Back Into a Tunnel
Now let's take a slightly different approach to the previous scenairo. Now we have the first tunnel from PE1 to P3 and a second tunnel from P3 to P7. So our packet will leave the first TE tunnel and enter the second tunnel on the same router.
Last the last two times, we look at the L2VPN information on PE1 to see our starting label stack
PE1#show mpls l2 vc det
Local interface: Et0/1 up, line protocol up, Ethernet up
Destination address: 13.13.13.13, VC ID: 13, VC status: up
Output interface: Tu3, imposed label stack {42 22 17}
Since this tunnel ends on a P router we are running LDP over it and see three labels {Tunnel LDP Pseudowire}. Since there are only three routers in the tunnel, the next hop is the Penultimate router.
P1#show mpls forwarding label 42
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
42 Pop Label 11.11.11.11 3 [1] \
10209 Et0/0 3.0.0.3
P1 will pop the top (Tunnel) label and send the exposed LDP label (22) to P3. When we look at this label on P3 we now see the [T] flag set
P3#show mpls forwarding label 22
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
22 [T] 19 13.13.13.13/32 1587549 Tu7 point2point
[T] Forwarding through a LSP tunnel.
View additional labelling info with the 'detail' option
This [T] flag is set because we are entering another TE Tunnel. More importantly this LSP is inside the tunnel. This means the outgoing label we see here is not at the top of the stack. Our TE label will be added to the stack after this. We can use the detail option to see both the TE label and LDP label.
P3#show mpls forwarding label 22 detail
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
22 19 13.13.13.13/32 1597350 Tu7 point2point
MAC/Encaps=14/22, MRU=1496, Label Stack{22 19}, via Et0/0
AABBCC025C10AABBCC025B008847 0001600000013000
No output feature configured
P3#show mpls traffic-eng tunnels tunnel 7
Name: P3_t7 (Tunnel7) Destination: 7.7.7.7
Status:
Admin: up Oper: up Path: valid Signalling: connected
path option 1, type dynamic (Basis for Setup, path weight 20)
InLabel : -
OutLabel : Ethernet0/0, 22
Next Hop : 5.0.0.5
Again, the label seen in the "show mpls forwarding" output is for the end to end LSP. This LSP gets carried inside a Traffic Engineering tunnel, so we can see this TE label with the detail option or by looking for the TE label the old-fashioned way with "show mpls traffic-eng tunnels tunnel 7"
Now P5 only looks at the top (TE) label and acts.
P5#show mpls forwarding label 22
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
22 Pop Label 3.3.3.3 7 [2] 2242162 Et0/0 7.0.0.7
Just as we'd expect the Penultimate router to do, P5 pops the TE label sending on the original LDP label on top to P7. Before we look at P7, let's think about the label stack on the packet leaving P5.
P5 just removed the TE label leaving the P3 <---> P7 LDP label. Under this we should still have the PE1 <---> PE3 Pseudowire label.
P7 sees the P3 LDP label (the one with the [T] flag earlier)
P7#show mpls forwarding label 19
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
19 Pop Label 13.13.13.13/32 29952955 Et0/0 73.0.0.3
This exposes the Pseudowire label to be processed by PE3
PE3#show mpls forwarding label 17
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
17 No Label l2ckt(4096) 22508357 Et0/0 point2point
Hopefully this paints a picture of how label operations work in various TE tunnel topologies. Fast Reroute scenarios will be covered in the next chapter, coming soon!
