VRF route leak

[networkadmin AQ]

I have the following network; 

A ring backbone consisting of Nokia Routers running multiprotocol BGP, MPLS, LDP, SDP, VPRN service. The Sites would run a Cisco Switch that communicates with the Noka Router through BGP.
The Cisco switches can communicate with the Nokia routers fine.
I am running VRF on the cisco switches and I am able to ping vrf to the other side and vice versa. However I am not able to run a successful ping from the clients at each side of the Cisco switches.
I have read a lot about VRF route leaking however I am unable to get it to work.
I need some help with this.

[MHM Cisco World]

 above debug from the IS_03_1090301 ?

[networkadmin AQ]

I enabled 'terminal monitor' on the BN site, .. 
That is the output I am getting.

Jun 27 10:08:02.059: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 13312ms (35000ms max, 60% jitter)
Jun 27 10:08:15.373: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 13312ms (35000ms max, 60% jitter)
Jun 27 10:08:28.689: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 6144ms (35000ms max, 60% jitter)
Jun 27 10:08:34.833: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 14336ms (35000ms max, 60% jitter)
Jun 27 10:08:37.497: BGP: topo global:IPv4 Unicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo global:IPv6 Unicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo global:VPNv4 Unicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo VRF1000:VPNv4 Unicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo global:IPv4 Multicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo global:L2VPN E-VPN:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo global:VPNv4 Multicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo VRF1000:VPNv4 Multicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo global:MVPNv4 Unicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo VRF1000:MVPNv4 Unicast:base Scanning routing tables
Jun 27 10:08:37.497: BGP: topo global:MVPNv6 Unicast:base Scanning routing tables
Jun 27 10:08:49.172: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 11264ms (35000ms max, 60% jitter)
Jun 27 10:09:00.437: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 12288ms (35000ms max, 60% jitter)
Jun 27 10:09:12.726: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 11264ms (35000ms max, 60% jitter)
Jun 27 10:09:23.991: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 8192ms (35000ms max, 60% jitter)


The pings start to work for a bit, but then they stop.

[networkadmin AQ]

These are the debug logs I am getting on the IS_03_1090301 switch;

Jun 27 10:41:37.790: BGP: topo global:IPv4 Unicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo global:IPv6 Unicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo global:VPNv4 Unicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo VRF1000:VPNv4 Unicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo global:IPv4 Multicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo global:L2VPN E-VPN:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo global:VPNv4 Multicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo VRF1000:VPNv4 Multicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo global:MVPNv4 Unicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo VRF1000:MVPNv4 Unicast:base Scanning routing tables
Jun 27 10:41:37.790: BGP: topo global:MVPNv6 Unicast:base Scanning routing tables
Jun 27 10:41:44.678: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 12288ms (35000ms max, 60% jitter)
Jun 27 10:41:56.966: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 12288ms (35000ms max, 60% jitter)
Jun 27 10:42:09.254: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 6144ms (35000ms max, 60% jitter)
Jun 27 10:42:15.401: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 12288ms (35000ms max, 60% jitter)
Jun 27 10:42:27.689: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 7168ms (35000ms max, 60% jitter)
Jun 27 10:42:34.858: BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 7168ms (35000ms max, 60% jitter)

[MHM Cisco World]

check Mr. @Harold Ritter reply about active same Peer under global and VRF 
it can explain the log you get. 
MHM

[networkadmin AQ]

I removed 'no neighbor 10.255.255.X' from each Routers.
However I the hosts are still not able to ping each other.

Did I mention that somehow, when I leave the ping running, it starts to ping for a while, then it droppes dead again.



I am seeing this in the debugging:

Jun 27 10:52:37.887: BGP: topo global:IPv4 Unicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo global:IPv6 Unicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo global:VPNv4 Unicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo VRF1000:VPNv4 Unicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo global:IPv4 Multicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo global:L2VPN E-VPN:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo global:VPNv4 Multicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo VRF1000:VPNv4 Multicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo global:MVPNv4 Unicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo VRF1000:MVPNv4 Unicast:base Scanning routing tables
Jun 27 10:52:37.887: BGP: topo global:MVPNv6 Unicast:base Scanning routing tables

[MHM Cisco World]

can you point in which router you see this debug 
also I ask you before you run L3VPN or L2VPN 
I see VPNv4 and I see L2VPN
it can the client use L2 and that explain the next-hop appear in traceroute 
MHM

[networkadmin AQ]

How/Where can I see this ? 
How do I determine what the client is using ? 

[Harold Ritter]

Hi @networkadmin AQ ,

Removing the neighbor from the global will not fix the PC ping issue, but will get rid of the error messages in the log.

As for the ping, I would still recommend disabling the FW on the PC during testing.

Regards,

[networkadmin AQ]

VRF Routing Tables:

BN_03_1030301#show ip route vrf VRF1000

Routing Table: VRF1000
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 8 subnets, 3 masks
B 10.1.3.0/24 [20/0] via 10.255.255.205, 00:16:26
C 10.6.3.0/24 is directly connected, Vlan1603
L 10.6.3.1/32 is directly connected, Vlan1603
B 10.9.3.0/24 [20/0] via 10.255.255.205, 00:15:58
C 10.255.255.204/30 is directly connected, Vlan800
L 10.255.255.206/32 is directly connected, Vlan800
B 10.255.255.208/30 [20/0] via 10.255.255.205, 00:16:26
B 10.255.255.224/30 [20/0] via 10.255.255.205, 00:16:26
172.16.0.0/32 is subnetted, 1 subnets
C 172.16.0.6 is directly connected, Loopback0

IS_03_1090301#show ip route vrf VRF1000

Routing Table: VRF1000
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 8 subnets, 3 masks
B 10.1.3.0/24 [20/0] via 10.255.255.209, 00:16:09
B 10.6.3.0/24 [20/0] via 10.255.255.209, 00:16:09
C 10.9.3.0/24 is directly connected, Vlan1903
L 10.9.3.1/32 is directly connected, Vlan1903
B 10.255.255.204/30 [20/0] via 10.255.255.209, 00:16:09
C 10.255.255.208/30 is directly connected, Vlan800
L 10.255.255.210/32 is directly connected, Vlan800
B 10.255.255.224/30 [20/0] via 10.255.255.209, 00:16:09
172.16.0.0/32 is subnetted, 1 subnets
C 172.16.0.9 is directly connected, Loopback0

[networkadmin AQ]

Somehowe along the way of testing, .. the pings where working.
And then they were NOT.

After checking and rechecking .. I have no clue, why it worked, and stopped working.
However, I have a theory, based on some logs I saw on the Nokia Routers, and the Cisco router, that keeps telling me,that there is no connection with the bgp peer. ' BGP: 10.255.255.205 Active open failed - no route to peer, open active delayed 12288ms (35000ms max, 60% jitter'

When I run 'show ip bgp summary' the status is still 'idle'.
Some time along the way, I played with the mtu settings between the ports between the Cisco and the Nokia, and I was able to see the status on 'active'.
But, this was not long.
Although many of you don't have the full picture yet, ..this is my question:

The Nokia routers, connect to each other forming a ring-topology, and the ports they are connected to each other are configured with mtu 9212.
Let's say .. every Nokia router has ports 1/1/5 and 1/2/5 connected to another Nokia router, forming the Multiprotocol BGP. And I am testing between 2 sites, which are connected to Nokia 1 at port 1/3/1, and Nokia 2 at port 1/3/1. 
these ports are connected to a port Gi1/0/48 on my Cisco switches ( 1 & 2 ) .
Ports 1/3/1 are configured with mtu 1504, and ports Gi1/0/48 are configured with mtu 1504.
So, .. would changing ALL the mtu to be the same solve this problem ?