MPLS Double lable swap on router - looping

[yualme]

Hello Team,
I am ran in to wierd problem here , below is my simple topologie.

R1 (Lo0 - 192.168.254.204)<=> R2 (Lo0-192.168.254.27)<GRE> R3 <=> R4 <=> R5 <=> R6 <=> R7 (Lo0-192.168.254.160)

We are running is-is as igp & mpls is enbled on routes.

Goal - ping Lo0 from all routes but it's not pingable from R4,R5,R6 & R7.

Below is the traceroute from R1 to R7 working fine

R1#traceroute 192.168.254.160 source 192.168.254.204

Type escape sequence to abort.
Tracing the route to 192.168.254.160
 1 172.16.13.197 [MPLS: Label 163 Exp 0] 80 msec 80 msec 80 msec => R2
 2 172.16.13.9 [MPLS: Label 62 Exp 0] 80 msec 80 msec 84 msec => R3
 3 172.16.13.33 [MPLS: Label 662 Exp 0] 76 msec 80 msec 80 msec => R4
 4 172.16.3.33 [MPLS: Label 120 Exp 0] 80 msec 80 msec 80 msec => R5
 5 172.16.0.10 [MPLS: Label 741 Exp 0] 80 msec 80 msec 80 msec => R6
 6 172.16.20.165 76 msec * 76 msec => R7
R1#

Below is the traceroute from R7 to R1 looping

R7#traceroute 192.168.254.204 source 192.168.254.160
Type escape sequence to abort.
Tracing the route to
VRF info: (vrf in name/id, vrf out name/id)
 1 172.16.20.166 [MPLS: Label 85 Exp 0] 236 msec 236 msec 232 msec => R6
 2 172.16.0.9 [MPLS: Label 143 Exp 0] 232 msec 236 msec 232 msec => R5
3 172.16.3.34 [MPLS: Label 579 Exp 0] 236 msec 236 msec 232 msec => R4
 4 172.16.13.34 [MPLS: Label 346 Exp 0] 80 msec 80 msec 80 msec => R3
 5 172.16.13.9 [MPLS: Label 453 Exp 0] 236 msec 232 msec 232 msec => R3
 6 172.16.13.33 [MPLS: Label 389 Exp 0] 236 msec 232 msec 236 msec => R4
 7 172.16.3.33 [MPLS: Label 317 Exp 0] 236 msec 232 msec 236 msec => R5
 8 * * *
 9 * * *

I can see MPLS lable swapping @ R3 due to GRE I guess.

Lable path changes from 0.214.254.204 to 192.168.254.95

R3#show ver | i Version
Cisco IOS Software, s3223_rp Software (s3223_rp-ADVIPSERVICESK9_WAN-VM), Version 12.2(33)SXI8a, RELEASE SOFTWARE (fc1)
ROM: System Bootstrap, Version 12.2(17r)S6, RELEASE SOFTWARE (fc1)

R3#show ip route 192.168.254.204
Routing entry for 192.168.254.204/32
 Known via "isis", distance 115, metric 40, type level-2
 Redistributing via isis
 Last update from 172.16.13.10 on Tunnel0, 08:33:28 ago
 Routing Descriptor Blocks:
 * 172.16.13.10, from 192.168.254.204, via Tunnel0
 Route metric is 40, traffic share count is 1

R3#show mpls forwarding-table 192.168.254.204
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
346 453 192.168.254.204/32 \
 25666369 Tu0 point2point
R3#show mpls forwarding-table labels 346
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
346 453 192.168.254.204/32 \
 25682375 Tu0 point2point
R3#show mpls forwarding-table labels 453
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
453 389 192.168.254.95/32 6187289317 Gi1/9 172.16.13.33

R4#show mpls forwarding-table labels 389
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
389 317 192.168.254.95/32 341674091 Gi1/4 172.16.3.33

[Giuseppe Larosa]

Hello Yagnesh,

it is quite strange behaviuor because the following output says:

R3#show mpls forwarding-table labels 346
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
346 453 10.214.254.204/32 \
25682375 Tu0 point2point

traffic should be sent into the GRE tunnel instead of being sent elsewhere.

What is the hardware equipment of R3 I understand is a C6500, I wonder if you have any service module that can interfer. Can you post a

show module

for R3?

Hope to help

Giuseppe

[yualme]

Hello Giuseppe,

Thank you for prompt revert. Below is the asked output , PLEASE let me know if anything else is needed from my end.

One question here router does the lable swap 2 times , isn't this strange or this is normal behavior with GRE tunnel

R3#show ver | i WS-C6504-E
cisco WS-C6504-E (R7000) processor (revision 2.0) with 516096K/8192K bytes of memory.

R3#show module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1    9  Supervisor Engine 32 8GE (Active)      WS-SUP32-GE-3B     SALXXXXXXXX
  3   48  48-port 10/100/1000 RJ45 EtherModule   WS-X6148A-GE-TX   SALXXXXXXXX

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  0xxx.xxxx.xxxe to 0xxx.xxxx.xxx9   4.4   12.2(18r)SX5 12.2(33)SXI8 Ok
  3  6xxx.xxxx.xxx0 to 6xxx.xxxx.xxxf   4.1   8.4(1)       12.2(33)SXI8 Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  1  Policy Feature Card 3       WS-F6K-PFC3B       SALXXXXXXXX 2.3    Ok
  1  Cat6k MSFC 2A daughterboard WS-F6K-MSFC2A     SALXXXXXXXX 3.1    Ok

Mod  Online Diag Status
---- -------------------
  1  Pass
  3  Pass

Traceroute from R3 to R1 - Success

R3#traceroute 192.168.254.204

Type escape sequence to abort.
Tracing the route to 192.168.254.204

  1 172.16.13.10 [MPLS: Label 453 Exp 0] 8 msec 12 msec 8 msec => R2
  2 172.16.13.198 8 msec *  8 msec                             => R1

Traceroute from R4 to R1 in same loop

R4#traceroute 192.168.254.204

Type escape sequence to abort.
Tracing the route to 192.168.254.204

  1 172.16.13.34 [MPLS: Label 346 Exp 0] 4 msec 4 msec 4 msec  => R4
  2 172.16.13.9 [MPLS: Label 453 Exp 0] 236 msec 232 msec 236 msec => R3
  3 172.16.13.33 [MPLS: Label 389 Exp 0] 236 msec 232 msec 236 msec => R3
  4 172.16.3.33 [MPLS: Label 317 Exp 0] 236 msec 232 msec 236 msec => R4
  5 172.16.0.5 [MPLS: Label 412 Exp 0] 236 msec 232 msec 236 msec
*
*
*

But Trace from R4(or R5 , R6 & R7) to R2 - Able to ping but not traceroute

R4#traceroute 192.168.254.27

Type escape sequence to abort.
Tracing the route to 192.168.254.27

  1 172.16.13.34 [MPLS: Label 542 Exp 0] 4 msec 4 msec 4 msec => R4
  2 172.16.13.9 0 msec 4 msec 0 msec       => R3
  3  *  *  *
  4  *  *  *
  5  *  *  *
  6  *

R4#ping 192.168.254.27

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.254.27, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms

[Giuseppe Larosa]

Hello Yagnesh,

from the show module we see that there is only the sup32 supervisor and the 48 ports ethernet linecard, so the GRE tunnel is implemented on the supervisor for sure.

About your question:

>>One question here router does the lable swap 2 times , isn't this strange or this is normal behavior with GRE tunnel 

I don't know if it is correct when a GRE tunnel is involved, but I remember to have seen tracerouote over MPLS where a specific PE node appeared two times. (but it was usually the last one or the penultimate hop)

It can be helpful to review how traceroute works over an MPLS network.

I have found the following document that describes the process:

http://www.cisco.com/c/en/us/support/docs/multiprotocol-label-switching-mpls/mpls/26585-mpls-traceroute.html

According to the document, the originating router starts to send packets with IP TTL=1, but the next router in the path instead of answering back to the original sender forwards the ICMP unreachable message along the path to the final destination of the LSP. It is the final destination of the LSP that then sends back over a different LSP with destination the originator router the ICMP unreachable message(s).

And this should happen at each hop on the path of the LSP to the final destination.

I wonder if the problem is only limited to traceroute you show that the ping works to R2.

>>>R4#ping 10.214.254.27

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.214.254.27, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms

Hope to help

Giuseppe

[yualme]

Hello Giuseppe,

From ping command I just want to show you that we can ping R2 here as in that case we are removing label due to paneltimate hope , so its not going to routing loops like R1.

And no this not trace route issue , we can't ping the R1 .

This can be iOS bus?

What should be my next step ?

Thanks for your help so far 

[Giuseppe Larosa]

Hello Yagnesh,

>> And no this not trace route issue , we can't ping the R1 .

You have a global connectivity issue for devices downstream the GRE tunnel that requires labeled frames to be sent over the tunnel.

I would open a service request with Cisco TAC to have them investigate the issue.

Of course, you cannot remove the GRE tunnel that allows you to go through a non MPLS IP only network.

If you are using service provider services like an MPLS L3 VPN to interconnect the two parts of your network you can ask the provider a quotation for Carrier supporting Carrier service. This is  to avoid the GRE tunnel.

Hope to help

Giuseppe

[yualme]

Hi Giuseppe ,Thanks for your revert .

We have GRE tunnel over internet & advertising router loopback over it.

Goal here is to connect R1 to route reflecter (R7) for connectivity to vpnv4 cloud.

Will ask Cisco guys & update the forum in case we get resolution.

[yualme]

We got below update from Cisco TAC

MPLS over GRE is not supported officially on PFC3B that equipped the SUP32.

This Feature is limited to SIP-400 with GE SPA as core facing interface and running SRB1 or greater using Sup720 and RSP720.