ASR9k xconnect output interface / first hop with ECMP

[Jason M.]

Hello,

When using the "show mpls l2transport vc detail" command on an IOS router the output shows an "output interface" field. What is the best way to verify the same thing on an XR when ECMP are involved?

When the xconnect is bound to a TE Tunnel it's easy to verify using the "show l2vpn forwarding interface" but when it is IGP routed to a destination with ECMP how can I be sure which interface will be used?

Two of the options I used was "show cef exact-route *XC local loopback* *XC peer loopback*" which gives me one output interface. But "show mpls forwarding exact-route label *local label for XC peer prefix* label *peer vc label*" gives me a different output interface interface.

Since the traffic is sourced from this ASR9k do figure using the local label in the "show mpls forwarding exact-route" command is probably not correct so I would lean more towards the CEF exact-route path but it doesn't factor in the VC label which is usually used for ECMP path selection.

EXAMPLES:

==========================================================================

IOS Example "show mpls l2transport vc det" :

Local interface: Gi0/1 up, line protocol up, Eth VLAN 768 up
  Interworking type is Ethernet
  Destination address: 172.31.255.133, VC ID: 2500010178, VC status: up
    Output interface: Te0/1, imposed label stack {17844 18425}

==========================================================================

==========================================================================

XR Example of "show l2vpn forwarding interface" with bound TE:

Segment 1                            Segment 2                            State
------------------------------------ ------------------------------------ ------
Te0/0/0/3.2143                       mpls    tunnel-te60640                UP    

==========================================================================

==========================================================================

XR Example of "show l2vpn forwarding interface" IGP routed:

Segment 1                            Segment 2                            State
------------------------------------ ------------------------------------ ------
Gi0/0/1/0(Eth port)                  mpls    172.21.255.153               UP

show mpls forwarding exact-route label 16267 bottom-label 1149
Thu Oct  8 14:31:33.646 EDT
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes       
Label  Label       or ID              Interface                    Switched    
------ ----------- ------------------ ------------ --------------- ------------
16267  16146       172.21.255.153/32  Te0/0/2/0    remote          N/A 

show cef exact-route 172.31.255.133 172.21.255.153
Thu Oct  8 14:32:10.886 EDT
172.21.255.153/32, version 3550305, internal 0x4004001 (ptr 0x9dc7499c) [1], 0x0 (0x9d595c10), 0x450 (0x9eaf4870)
 Updated Oct  5 11:42:40.234
 remote adjacency to TenGigE0/0/2/1

==========================================================================

[xthuijs]

hi jason,

for ecmp you can use

show cef exact route or

show mpls forwarding exact (<- this for l2vpn since it is mpls encapped)

and for bundle member selection you can use

the command bundle-hash which derivces the member based on some inputs provided.

cheers!

xander

[Jason M.]

I have the output from both of those commands in the original post which show that they both provide different results. When tracing through the network I use the "mpls forwarding exact" but first hop there isn't really a local label involved.

[xthuijs]

hi jason,

on the first hop, PE router you have a next hop label and a pw label, you'd select based on pw label here.

on the next P router(s) you'd be selecting based on inner label (PW label or FAT if configured)

xander

[Jason M.]

I understand how the router forwards the traffic but I want to know what command specifically shows which interface will be selected.

"show mpls forwarding exact" - This command requires a local label to be used but because the EVC is created on this router, traffic does not enter the router with the local label. The router selects the label for the next hop and forwards the traffic with the PW label at the bottom. With no local label being used, how can this be the correct answer?

"show cef exact" - This command requires the source IP address. I can make an assumption the the EVC will take the same next hop that cef has chosen for the loopback of the EVC source router but I have not read any documentation that confirms that. Because this is an MPLS only service, the EVC source loopback will not be in the packet. How could this be the correct answer?

I don't particularly have confidence in either one of those commands providing the definitive answer to the question.

[Jason M.]

Tested it out with live customer traffic and confirmed that the outputs from "show cef exact" and "show mpls forwarding exact" are not accurate for the first hop with ECMP. Opened a TAC case for it as well.

[Jason M.]

After several months we were not able to find a command that accurately shows the first hop interface. The TAC agent reports that he will be submitting a feature request for this functionality.

[Jason M.]

The TAC case is complete and Cisco has reported that due to the architecture there is no way to determine the first hop in a ECMP scenario using show commands and it will not be possible to create the command.

[xthuijs]

hi jason, it is possible when you know the label that is used for the pw or fat.

I am adding a tool here compiled for mac taht you can use to derive what fat label will be created for a given flow.

also it will give the bucket distribution.

with this and the show mpls exact you can find out what path will be taken for that label.

we're working on a portal along side with this tool to interact with the router to find out the paths or members and spell out which path or member will be chosen for a given set. more to follow.

xander

[Jason M.]

We aren't using FAT on our pseudowires at this time.

If I understand you correctly, you're referring to using the "show mpls forwarding exact-route label **peer loopback label** bottom-label **remote vc label** by taking the remote VC label from the "show l2vpn xconnect detail" command and the peer loopback label from the local label field of the "show mpls forwarding prefix **peer loopback prefix**" command.

If so, we have already proven that the command provides a result but it is not accurate on our router. To test we set up a pseudowire between the ASR9k and an ME3800X with two ECMPs available. Then we ran the commands above and put test traffic on the pseudowire to watch the uplink interface counters to verify the path/first hop.

In this scenario no matter that paths the command returned there is a 50/50 chance it will be correct just by chance (since only 2 paths are available). So watching the traffic we used the "clear xconnect" command at the ME3800X which will change the VC label so we can see if it consistently follows the path that "show mpls forwarding exact" shows it will take. We confirmed regardless of the remote VC label the traffic takes the same path which suggests that the VC label is not the determining factor for the traffic path for the first hop.

We demonstrated this to multiple TAC engineers over the 10 months the case was opened and they had no answers for us.

If there are another set of labels that should be used with the "show mpls forwarding exact" command please let me know because none of the TAC agents were able to provide any working solution.

If needed I can forward you the information and set up a webex.

[xthuijs]

on a PE, the device has access to teh IP level info and will produce a hash based on that. so the PE doing the imposition will still do a loadbalancing based on whether the loadbalancing flow mac or ip is configured and select a member/path based on that in the later releases.

the flow label or vc label will be used by P routers and the egress PE.

cheers

xander

[Jason M.]

Thanks for the information. I have included an example below to verify I understand what you're saying.

Example:

I config the ASR9K port Te0/0/0/1.100 to match dot1q 100.

I configure a test set using vlan 100 and send traffic using the following

source mac1111.1111.1111
destination mac 2222.2222.2222
source ip 1.1.1.1
destination ip 2.2.2.2
tcp source: 1111
tcp destination 2222

When the first packet hits ASR9k matching the dot1q statement (and without FAT configured) it will use the standard load balancing method to create the hash (in the case of the ASR9001 without a bundle and using IGP this would be L3/L4 information according to your load balancing write-up). The hash will be used to select the outgoing interface and all subsequent pseudowire traffic will take that same interface regardless of the header information (unless FAT is configured).

Is that correct?

If that is correct what condition would cause it to rehash?

[xthuijs]

your scenario: ack, but just to make sure: the default loadbalancing in l2vpn is source-dest mac. you'd need to configure the l2vpn load-bal flow src-dest-ip to make it l3/l4.

there is (fortunately) no such thing as rehashing. a packet comes in and gets it hash computed, pretty much before anything else the rules as to what feeds into the hash are understood right?

then depending on the scenario, P, PE, fat or not, ecmp bundle a path or member is chosen.

and a knob setting (ya ya, lots of things :) that allows you to somewhat control how the hash is derived in certain circumstances.

To get a better visual on that, take CL preso id 2904 from san diego 2015 that has a few nice details there.

[Jason M.]

Final question:

Can you give me a list of steps for the scenario below?

1. I'm on router 8.8.8.8 and I create the pseudowire below using no flow label.

source 8.8.8.8
destination 9.9.9.9
pw-id 100
pw local label: 888
pw remote label 999

2. The packet below enters the pseudowire on Te0/0/0/1.100

source mac1111.1111.1111
destination mac 2222.2222.2222
VLAN: 100
source ip 1.1.1.1
destination ip 2.2.2.2
tcp source: 1111
tcp destination 2222

3. The 8.8.8.8 router has two core facing interfaces that are equal cost to 9.9.9.9

Te0/0/0/2

Te0/0/0/3

Is there a command sequence I can use on router 8.8.8.8 to get an output showing which of those two equal cost core facing interfaces will be used?

**btw I'm still viewing the CL presentation right now**