VRF traceroute/ping problem

[cwalek]

Hello guys,

I have encountered and issue when I try to traceroute/ping from one router to another. They are both in VRFs but I have configured VRF Leaking between the routers. When I try to ping/traceroute from one of the routers trough ( for example "VRF 1") it goes trough but when I try the same from ( for example "VRF 2") it doesn't go trough, altough the routers both have the routes to the destination int their BGP and Routing tables. In the topology I have suspended R4-R7 link and R1 also so in that case route has to go from R5 trough R2-R4-R11-R3-R7. R5 is in VRF "LP", when I try traceroute from R2/R4 to R7 trough VRF-DM it goes but it doesn't trough VRF-LP it stops at R11. What should I do?

R2 Config: ( Don't mind the descriptions on the interfaces they are leftover from other stage of the topology) 

hostname R2
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
no ip icmp rate-limit unreachable
!
!
ip cef
no ip domain lookup
!
!
ip vrf CUST-DM
rd 6501:2
route-target export 6501:2
route-target import 6501:2
route-target import 6501:1
!
ip vrf CUST-LP
rd 6501:1
route-target export 6501:1
route-target import 6501:1
route-target import 6501:2
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip tcp synwait-time 5
!
!
!
!
!
interface Loopback0
ip address 2.2.2.2 255.255.255.255
!
interface FastEthernet0/0
no ip address
shutdown
duplex half
!
interface Ethernet1/0
no ip address
duplex half
!
interface Ethernet1/0.10
encapsulation dot1Q 10
ip vrf forwarding CUST-DM
ip address 10.2.2.1 255.255.255.252
!
interface Ethernet1/0.20
encapsulation dot1Q 20
ip vrf forwarding CUST-LP
ip address 10.2.2.1 255.255.255.252
!
interface Ethernet1/1
description ### To SW-1 ###
ip vrf forwarding CUST-LP
ip address 10.10.8.1 255.255.255.252
duplex full
!
interface Ethernet1/2
no ip address
shutdown
duplex half
!
interface Ethernet1/3
no ip address
shutdown
duplex half
!
interface Ethernet1/4
no ip address
shutdown
duplex half
!
interface Ethernet1/5
no ip address
shutdown
duplex half
!
interface Ethernet1/6
no ip address
shutdown
duplex half
!
interface Ethernet1/7
no ip address
shutdown
duplex half
!
interface Ethernet2/0
no ip address
shutdown
duplex half
!
interface Ethernet2/1
ip vrf forwarding CUST-DM
ip address 10.10.6.1 255.255.255.252
duplex half
!
interface Ethernet2/2
no ip address
shutdown
duplex half
!
interface Ethernet2/3
no ip address
shutdown
duplex half
!
interface FastEthernet3/0
no ip address
shutdown
duplex half
!
interface FastEthernet4/0
no ip address
shutdown
duplex auto
speed auto
!
interface FastEthernet4/1
no ip address
shutdown
duplex auto
speed auto
!
!
router bgp 6500
no synchronization
bgp log-neighbor-changes
redistribute connected
no auto-summary
!
address-family ipv4 vrf CUST-LP
redistribute connected
neighbor 10.2.2.2 remote-as 6500
neighbor 10.2.2.2 timers 5 20
neighbor 10.2.2.2 activate
neighbor 10.2.2.2 next-hop-self
neighbor 10.2.2.2 advertisement-interval 2
neighbor 10.10.8.2 remote-as 500
neighbor 10.10.8.2 timers 5 20
neighbor 10.10.8.2 activate
neighbor 10.10.8.2 as-override
neighbor 10.10.8.2 advertisement-interval 2
neighbor 10.10.8.2 route-map MED_PREF out
no synchronization
exit-address-family
!
address-family ipv4 vrf CUST-DM
redistribute connected route-map COMMUNITY_2
neighbor 10.2.2.2 remote-as 6500
neighbor 10.2.2.2 timers 5 20
neighbor 10.2.2.2 activate
neighbor 10.2.2.2 send-community both
neighbor 10.2.2.2 next-hop-self
neighbor 10.2.2.2 advertisement-interval 2
neighbor 10.10.6.2 remote-as 500
neighbor 10.10.6.2 timers 5 20
neighbor 10.10.6.2 activate
neighbor 10.10.6.2 send-community both
neighbor 10.10.6.2 as-override
neighbor 10.10.6.2 advertisement-interval 2
neighbor 10.10.6.2 route-map COMMUNITY_2 out
no synchronization
exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip bgp-community new-format
!
no cdp log mismatch duplex
!
route-map MED_PREF permit 10
set metric 100
!
route-map COMMUNITY_2 permit 10
set community 6500:600
!

R4 Config: 

ip cef
no ip domain lookup
!
!
ip vrf CUST-DM
rd 6503:2
route-target export 6503:2
route-target import 6503:2
route-target import 6503:1
!
ip vrf CUST-LP
rd 6503:1
route-target export 6503:1
route-target import 6503:1
route-target import 6503:2
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip tcp synwait-time 5
!
!
!
!
!
interface Loopback0
ip address 4.4.4.4 255.255.255.255
!
interface FastEthernet0/0
no ip address
shutdown
duplex half
!
interface Ethernet1/0
description ### To R2 ###
ip address 10.2.2.2 255.255.255.252
duplex half
!
interface Ethernet1/0.10
encapsulation dot1Q 10
ip vrf forwarding CUST-DM
ip address 10.2.2.2 255.255.255.252
!
interface Ethernet1/0.20
encapsulation dot1Q 20
ip vrf forwarding CUST-LP
ip address 10.2.2.2 255.255.255.252
!
interface Ethernet1/1
ip vrf forwarding CUST-LP
ip address 10.10.8.1 255.255.255.252
duplex half
!
interface Ethernet1/1.10
shutdown
!
interface Ethernet1/2
no ip address
shutdown
duplex half
!
interface Ethernet1/3
no ip address
shutdown
duplex half
!
interface Ethernet1/4
no ip address
shutdown
duplex half
!
interface Ethernet1/5
no ip address
shutdown
duplex half
!
interface Ethernet1/6
no ip address
shutdown
duplex half
!
interface Ethernet1/7
no ip address
shutdown
duplex half
!
interface Ethernet2/0
no ip address
shutdown
duplex half
!
interface Ethernet2/1
ip vrf forwarding CUST-DM
ip address 10.10.6.1 255.255.255.252
duplex half
!
interface Ethernet2/2
no ip address
shutdown
duplex half
!
interface Ethernet2/3
no ip address
shutdown
duplex half
!
interface FastEthernet3/0
no ip address
shutdown
duplex half
!
interface FastEthernet4/0
no ip address
shutdown
duplex auto
speed auto
!
interface FastEthernet4/1
no ip address
shutdown
duplex auto
speed auto
!
!
router bgp 6500
no synchronization
bgp log-neighbor-changes
no auto-summary
!
address-family ipv4 vrf CUST-LP
redistribute connected
neighbor 10.2.2.1 remote-as 6500
neighbor 10.2.2.1 timers 5 20
neighbor 10.2.2.1 activate
neighbor 10.2.2.1 next-hop-self
neighbor 10.2.2.1 advertisement-interval 2
neighbor 10.10.8.2 remote-as 500
neighbor 10.10.8.2 timers 5 20
neighbor 10.10.8.2 activate
neighbor 10.10.8.2 as-override
neighbor 10.10.8.2 advertisement-interval 2
neighbor 10.10.8.2 route-map MED_PREF out
no synchronization
exit-address-family
!
address-family ipv4 vrf CUST-DM
redistribute connected
neighbor 10.2.2.1 remote-as 6500
neighbor 10.2.2.1 timers 5 20
neighbor 10.2.2.1 activate
neighbor 10.2.2.1 next-hop-self
neighbor 10.2.2.1 advertisement-interval 2
neighbor 10.10.6.2 remote-as 500
neighbor 10.10.6.2 timers 5 20
neighbor 10.10.6.2 activate
neighbor 10.10.6.2 send-community both
neighbor 10.10.6.2 next-hop-self
neighbor 10.10.6.2 as-override
neighbor 10.10.6.2 advertisement-interval 2
neighbor 10.10.6.2 allowas-in
no synchronization
exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip bgp-community new-format
!
no cdp log mismatch duplex
!
route-map MED_PREF permit 10
set metric 100
!

R11 Config: 

hostname R11
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
no ip icmp rate-limit unreachable
!
!
no ip cef
no ip domain lookup
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip tcp synwait-time 5
!
!
!
!
!
interface Loopback0
ip address 11.11.11.11 255.255.255.255
!
interface FastEthernet0/0
ip address 172.32.0.1 255.255.255.252
duplex half
!
interface Ethernet1/0
ip address 10.10.9.2 255.255.255.252
duplex half
!
interface Ethernet1/1
ip address 10.10.6.2 255.255.255.252
duplex half
!
interface Ethernet1/2
no ip address
shutdown
duplex half
!
interface Ethernet1/3
no ip address
shutdown
duplex half
!
interface Ethernet1/4
no ip address
shutdown
duplex half
!
interface Ethernet1/5
no ip address
shutdown
duplex half
!
interface Ethernet1/6
no ip address
shutdown
duplex half
!
interface Ethernet1/7
no ip address
shutdown
duplex half
!
interface Ethernet2/0
no ip address
shutdown
duplex half
!
interface Ethernet2/1
no ip address
shutdown
duplex half
!
interface Ethernet2/2
no ip address
shutdown
duplex half
!
interface Ethernet2/3
no ip address
shutdown
duplex half
!
interface FastEthernet3/0
no ip address
shutdown
duplex half
!
interface FastEthernet4/0
no ip address
shutdown
duplex auto
speed auto
!
interface FastEthernet4/1
no ip address
shutdown
duplex auto
speed auto
!
!
router bgp 500
bgp log-neighbor-changes
neighbor 10.10.6.1 remote-as 6500
neighbor 10.10.6.1 timers 5 20
neighbor 10.10.9.1 remote-as 6500
neighbor 10.10.9.1 timers 5 20
neighbor 172.32.0.2 remote-as 500
neighbor 172.32.0.2 timers 5 20
!
address-family ipv4
redistribute connected
neighbor 10.10.6.1 activate
neighbor 10.10.6.1 advertisement-interval 2
neighbor 10.10.6.1 route-map LP_C in
neighbor 10.10.9.1 activate
neighbor 10.10.9.1 advertisement-interval 2
neighbor 172.32.0.2 activate
neighbor 172.32.0.2 send-community both
neighbor 172.32.0.2 advertisement-interval 2
default-information originate
no auto-summary
no synchronization
exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip bgp-community new-format
ip community-list 10 permit 6500:600
!
no cdp log mismatch duplex
!
route-map LP_C permit 10
match community 10
set local-preference 350
!
route-map LP_C permit 20
set local-preference 100
!
route-map LOCAL_PREF permit 10
set local-preference 900
!
route-map LOCAL_PREF_2 permit 10
set local-preference 100
!

R3 Config : hostname R3
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
no ip icmp rate-limit unreachable
!
!
ip cef
no ip domain lookup
!
!
ip vrf CUST-DM
rd 6502:2
route-target export 6502:2
route-target import 6502:2
route-target import 6502:1
!
ip vrf CUST-LP
rd 6502:1
route-target export 6502:1
route-target import 6502:1
route-target import 6502:2
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip tcp synwait-time 5
!
!
!
!
!
interface Loopback0
ip address 3.3.3.3 255.255.255.255
!
interface FastEthernet0/0
no ip address
shutdown
duplex half
!
interface Ethernet1/0
no ip address
duplex half
!
interface Ethernet1/0.10
encapsulation dot1Q 10
ip vrf forwarding CUST-LP
ip address 10.1.1.2 255.255.255.252
!
interface Ethernet1/0.20
encapsulation dot1Q 20
ip vrf forwarding CUST-DM
ip address 10.1.1.2 255.255.255.252
!
interface Ethernet1/1
ip vrf forwarding CUST-LP
ip address 10.10.5.1 255.255.255.252
duplex half
!
interface Ethernet1/1.10
!
interface Ethernet1/2
no ip address
shutdown
duplex half
!
interface Ethernet1/3
no ip address
shutdown
duplex half
!
interface Ethernet1/4
no ip address
shutdown
duplex half
!
interface Ethernet1/5
no ip address
shutdown
duplex half
!
interface Ethernet1/6
no ip address
shutdown
duplex half
!
interface Ethernet1/7
no ip address
shutdown
duplex half
!
interface Ethernet2/0
no ip address
shutdown
duplex half
!
interface Ethernet2/1
ip vrf forwarding CUST-DM
ip address 10.10.9.1 255.255.255.252
duplex half
!
interface Ethernet2/2
no ip address
shutdown
duplex half
!
interface Ethernet2/3
no ip address
shutdown
duplex half
!
interface FastEthernet3/0
no ip address
shutdown
duplex half
!
interface FastEthernet4/0
no ip address
shutdown
duplex auto
speed auto
!
interface FastEthernet4/1
no ip address
shutdown
duplex auto
speed auto
!
!
router bgp 6500
no synchronization
bgp log-neighbor-changes
no auto-summary
!
address-family ipv4 vrf CUST-LP
redistribute connected
neighbor 10.1.1.1 remote-as 6500
neighbor 10.1.1.1 timers 5 20
neighbor 10.1.1.1 activate
neighbor 10.1.1.1 next-hop-self
neighbor 10.1.1.1 advertisement-interval 2
neighbor 10.10.5.2 remote-as 500
neighbor 10.10.5.2 timers 5 20
neighbor 10.10.5.2 activate
neighbor 10.10.5.2 send-community both
neighbor 10.10.5.2 as-override
neighbor 10.10.5.2 advertisement-interval 2
no synchronization
exit-address-family
!
address-family ipv4 vrf CUST-DM
redistribute connected
neighbor 10.1.1.1 remote-as 6500
neighbor 10.1.1.1 timers 5 20
neighbor 10.1.1.1 activate
neighbor 10.1.1.1 next-hop-self
neighbor 10.1.1.1 advertisement-interval 2
neighbor 10.10.9.2 remote-as 500
neighbor 10.10.9.2 timers 5 20
neighbor 10.10.9.2 activate
neighbor 10.10.9.2 as-override
neighbor 10.10.9.2 advertisement-interval 2
neighbor 10.10.9.2 route-map MED_PREF out
no synchronization
exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip bgp-community new-format
!
no cdp log mismatch duplex
!
route-map MED_PREF permit 10
set metric 100
!

R7 Config: 

no ip cef
no ip domain lookup
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
ip tcp synwait-time 5
!
!
!
!
!
interface Loopback0
ip address 7.7.7.7 255.255.255.255
!
interface FastEthernet0/0
ip address 172.32.0.1 255.255.255.252
duplex half
!
interface Ethernet1/0
ip address 10.10.5.2 255.255.255.252
duplex half
!
interface Ethernet1/1
ip address 10.10.8.2 255.255.255.252
duplex half
!
interface Ethernet1/2
no ip address
shutdown
duplex half
!
interface Ethernet1/3
no ip address
shutdown
duplex half
!
interface Ethernet1/4
no ip address
shutdown
duplex half
!
interface Ethernet1/5
no ip address
shutdown
duplex half
!
interface Ethernet1/6
no ip address
shutdown
duplex half
!
interface Ethernet1/7
no ip address
shutdown
duplex half
!
interface Ethernet2/0
no ip address
shutdown
duplex half
!
interface Ethernet2/1
no ip address
shutdown
duplex half
!
interface Ethernet2/2
no ip address
shutdown
duplex half
!
interface Ethernet2/3
no ip address
shutdown
duplex half
!
interface FastEthernet3/0
no ip address
shutdown
duplex half
!
interface FastEthernet4/0
no ip address
shutdown
duplex auto
speed auto
!
interface FastEthernet4/1
no ip address
shutdown
duplex auto
speed auto
!
!
router bgp 500
bgp log-neighbor-changes
neighbor 10.10.5.1 remote-as 6500
neighbor 10.10.5.1 timers 5 20
neighbor 10.10.8.1 remote-as 6500
neighbor 10.10.8.1 timers 5 20
neighbor 172.32.0.2 remote-as 500
neighbor 172.32.0.2 timers 5 20
!
address-family ipv4
redistribute connected
neighbor 10.10.5.1 activate
neighbor 10.10.5.1 advertisement-interval 2
neighbor 10.10.5.1 route-map LP_C in
neighbor 10.10.8.1 activate
neighbor 10.10.8.1 advertisement-interval 2
neighbor 10.10.8.1 route-map LOCAL_PREF_2 in
neighbor 172.32.0.2 activate
neighbor 172.32.0.2 send-community both
neighbor 172.32.0.2 advertisement-interval 2
no auto-summary
no synchronization
exit-address-family
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip bgp-community new-format
ip community-list 10 permit 6500:500
!
no cdp log mismatch duplex
!
route-map LP_C permit 10
match community 10
set local-preference 350
!
route-map LP_C permit 20
set local-preference 100
!
route-map LOCAL_PREF permit 10
set local-preference 900
!
route-map LOCAL_PREF_2 permit 10
set local-preference 100
!

NOTE: There might be route-maps that are present but not applied anywhere.