Odd traceroute problem inside provider MPLS cloud/vrf

[IRMAN SETIAWAN]

Just recently we have received a few complaint from one of our site connected via Verizon MPLS.

Upon further troubleshooting we've identified that routing is off when tracerouting to a device on that remote site.

For some reason a traceroute to one of the ip local to site shows that it hit he PER ip address of the WAN router, then it bounce off to a CER ip address of another location within the region, then to the destination IP.

I've double check advertisement and made sure it was done correctly, and when asking verizon, their aswer is " its a false possitive caused by their CRS equipment".

We did some traffic analysis using netflow and verified that traffic is flowing to the right destination, and not being bounce off like the traceroute shows.

Moving along, we recently add another circuit at one other remote location, and for some reason it is indicating the same issue as the previous site, but this time with ATT.

The only similarities between the 2 site that shows this behavior is that both of them are fairly close in distance. I have a feeling that they might be connected to the same PE router.

Have anyone come aross this problem ?

Thanks

[Bilal Nawaz]

Hello, If they are different ISPs I don't believe they'd be sharing a PE router or CE - unless there is some sort of agreement between them. And personally I don't think there is anything wrong seeing the trace route hitting a routed hop, probably a hop in the CE router - a vrf maybe where it will have its customer routes.
Another question, how do you know it's hit the PE router? Have they told you which IPs they are using within MPLS environment?
They're just providing you an MPLS service, and it works right?...
So.
What is the average latency to the other sites when you ping over both MPLS networks for Verizon and AT&T.

Hope this helps

Sent from Cisco Technical Support iPhone App

[IRMAN SETIAWAN]

Bilal, we have MPLS services from different provider and we see this weird behavior in both ATT and Verizon MPLS service.

The only similarities between them are that the 2 location in questions are close to each other which have the potential that the 2 site is connected to the same PE router.

How do i tell if I am hitting the PE router is by the PER address.

[Bilal Nawaz]

I have created an MPLS VPN network, and my traceroute goes through one hop in the Service Provider cloud as expected. This is normal since you are hitting their VRF. But I still find it hard to believe this is the PE router that is shared between service providers.

Please show the traceroute output. My one from the Customer router shows this:

CUST1# traceroute 200.200.200.1

Type escape sequence to abort.

Tracing the route to 200.200.200.1

1      10.1.1.2      80 msec 56 msec 36 msec

2      1.1.1.3      [MPLS: Label 19 Exp 0] 56 msec 60 msec 88 msec - THIS IS IN THE SERVICE PROVIDER

3      1.1.1.1      124 msec * 104 msec

Please show the traceroute for both routes over the ISPs.

Hope this helps

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

[milan.kulik]

Hi,

this is how tracroute behaves in MPLS networks:

http://www.cisco.com/en/US/tech/tk436/tk428/technologies_tech_note09186a008020a42a.shtml

Possibly it helps you?

Best regards,

Milan