Problem with ospf sham link and end to end ping failure in MPLS VPN

princesai · ‎12-16-2010

Hello friends,

can u please help me with my configuration i configured gre tunnelling which is working fine but the problem is my customers cant ping each other but i can see the routes. i dont really understand where the problem is...

my topology is CE1S1 (ospf configured)on one router and on the other CE2S1 (eigrp configured) -------->PE1-------->P-------->P--------->PE2-------->CE1S2(ospf cinfigured)on one router and on the other customer router CE2S2(eigrp configured).

please check my running configuration which i attached and help me to solve my radle.....

thank you in advance....

Regards,

Chinni

Mahesh Gohil · ‎12-16-2010

Hello Sai,

Can you post output of

> sh ip route vrf from PE1 and PE2 (only for source/ destination ip for which customer is pinging)

> sh ip route from CE end (for source and destination ip only)

> I assume that mpls cloud has correct config..additionally you can post sh mpls ldp forwarding from all P/PE routers

this will help to refine the problem

Regards

Mahesh

princesai · ‎12-16-2010

Hi mahesh,

i am giving outputs for the network

in PE1

PE1#sh ip route vrf C1

Routing Table: C1
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     5.0.0.0/24 is subnetted, 1 subnets
D       5.5.5.0 [90/156160] via 192.168.4.2, 00:02:41, FastEthernet1/0
     6.0.0.0/24 is subnetted, 1 subnets
B       6.6.6.0 [200/156160] via 2.2.2.2, 00:01:33
C    192.168.4.0/24 is directly connected, FastEthernet1/0
B    192.168.5.0/24 [200/0] via 2.2.2.2, 00:06:03
PE1#sh ip route vrf C2

Routing Table: C2
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     3.0.0.0/24 is subnetted, 1 subnets
C       3.3.3.0 is directly connected, Loopback1
     4.0.0.0/24 is subnetted, 1 subnets
B       4.4.4.0 [200/0] via 2.2.2.2, 00:06:58
     7.0.0.0/32 is subnetted, 1 subnets
O       7.7.7.7 [110/2] via 192.168.6.2, 00:06:49, FastEthernet0/1
     8.0.0.0/32 is subnetted, 1 subnets
B       8.8.8.8 [200/2] via 2.2.2.2, 00:05:27
C    192.168.6.0/24 is directly connected, FastEthernet0/1
B    192.168.7.0/24 [200/0] via 2.2.2.2, 00:06:58

in PE2

PE2#sh ip route vrf C1

Routing Table: C1
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     5.0.0.0/24 is subnetted, 1 subnets
B       5.5.5.0 [200/156160] via 1.1.1.1, 00:07:30
     6.0.0.0/24 is subnetted, 1 subnets
D       6.6.6.0 [90/156160] via 192.168.5.2, 00:06:44, FastEthernet1/0
B    192.168.4.0/24 [200/0] via 1.1.1.1, 00:11:00
C    192.168.5.0/24 is directly connected, FastEthernet1/0
PE2#sh ip route vrf C2

Routing Table: C2
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     3.0.0.0/24 is subnetted, 1 subnets
B       3.3.3.0 [200/0] via 1.1.1.1, 00:11:05
     4.0.0.0/24 is subnetted, 1 subnets
C       4.4.4.0 is directly connected, Loopback1
     7.0.0.0/32 is subnetted, 1 subnets
B       7.7.7.7 [200/2] via 1.1.1.1, 00:10:51
     8.0.0.0/32 is subnetted, 1 subnets
O       8.8.8.8 [110/2] via 192.168.7.2, 00:09:41, FastEthernet0/1
B    192.168.6.0/24 [200/0] via 1.1.1.1, 00:11:05
C    192.168.7.0/24 is directly connected, FastEthernet0/1

in PE1

PE1#sh ip ospf sham-links
Sham Link OSPF_SL0 to address 4.4.4.4 is down
Area 0 source address 3.3.3.3
Run as demand circuit
DoNotAge LSA allowed. Cost of using 1 State DOWN,
Timer intervals configured, Hello 10, Dead 40, Wait 40,

in PE2

PE2#sh ip ospf sham-links
Sham Link OSPF_SL0 to address 3.3.3.3 is down
Area 0 source address 4.4.4.4
Run as demand circuit
DoNotAge LSA allowed. Cost of using 1 State DOWN,
Timer intervals configured, Hello 10, Dead 40, Wait 40,

in CE1S1

CE1S1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     3.0.0.0/24 is subnetted, 1 subnets
O E2    3.3.3.0 [110/1] via 192.168.6.1, 00:13:50, FastEthernet0/0
     4.0.0.0/24 is subnetted, 1 subnets
O E2    4.4.4.0 [110/1] via 192.168.6.1, 00:13:50, FastEthernet0/0
     7.0.0.0/24 is subnetted, 1 subnets
C       7.7.7.0 is directly connected, Loopback0
     8.0.0.0/32 is subnetted, 1 subnets
O IA    8.8.8.8 [110/502] via 192.168.6.1, 00:13:55, FastEthernet0/0
C    192.168.6.0/24 is directly connected, FastEthernet0/0
O IA 192.168.7.0/24 [110/501] via 192.168.6.1, 00:15:13, FastEthernet0/0

in CE1S2

CE1S2#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     3.0.0.0/24 is subnetted, 1 subnets
O E2    3.3.3.0 [110/1] via 192.168.7.1, 00:14:46, FastEthernet0/0
     4.0.0.0/24 is subnetted, 1 subnets
O E2    4.4.4.0 [110/1] via 192.168.7.1, 00:14:46, FastEthernet0/0
     7.0.0.0/32 is subnetted, 1 subnets
O IA    7.7.7.7 [110/502] via 192.168.7.1, 00:14:46, FastEthernet0/0
     8.0.0.0/24 is subnetted, 1 subnets
C       8.8.8.0 is directly connected, Loopback0
O IA 192.168.6.0/24 [110/501] via 192.168.7.1, 00:14:46, FastEthernet0/0
C    192.168.7.0/24 is directly connected, FastEthernet0/0

in CE2S1

CE2S1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     5.0.0.0/24 is subnetted, 1 subnets
C       5.5.5.0 is directly connected, Loopback0
     6.0.0.0/24 is subnetted, 1 subnets
D       6.6.6.0 [90/158720] via 192.168.4.1, 00:12:20, FastEthernet0/0
C    192.168.4.0/24 is directly connected, FastEthernet0/0
D    192.168.5.0/24 [90/30720] via 192.168.4.1, 00:13:28, FastEthernet0/0

in CE2S2

CE2S2#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     5.0.0.0/24 is subnetted, 1 subnets
D       5.5.5.0 [90/158720] via 192.168.5.1, 00:13:36, FastEthernet0/0
     6.0.0.0/24 is subnetted, 1 subnets
C       6.6.6.0 is directly connected, Loopback0
D    192.168.4.0/24 [90/30720] via 192.168.5.1, 00:13:36, FastEthernet0/0
C    192.168.5.0/24 is directly connected, FastEthernet0/0

and their is no command called " sh mpls ldp forwarding " in routers

these are the outputs as u can c i have the routing information but when i tried to ping the ping failed

please help me out....

Mahesh Gohil · ‎12-16-2010

Hi sai,

I have replicated the scenario in lab and it seems that it is problem with ipsec.

I frequently see below message while debug

*Mar 1 00:46:07.931: %CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed
connection id=1, sequence number=344

Since I have very little exposure to IPSEC i request you to go through below notes

"

Replay Check Failed
This output shows an example of the "Replay Check Failed" error:

%CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed connection id=#.This error is a result of reordering in transmission medium (especially if parallel paths exist), or unequal paths of packet processing inside Cisco IOS for large versus small packets plus under load. Change the transform-set to reflect this. The reply check is only seen when transform-set esp-md5-hmac is enabled. In order to surpress this error message, disable esp-md5-hmac and do encryption only. "

also go through below doc to fine tune some parameters

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t14/feature/guide/gt_iarwe.html#wp1027188

Regards

Mahesh

princesai · ‎12-16-2010

hi mahesh can you please check with out ipsec... even after removing ipsec i am unable to ping....

one more thing is that possible if ipsec fails but u can see the routes??

sudjacob · ‎12-17-2010

To deal with the sham link issues.

1. sham link wont work with /24 network

2. change the sham link *(network 3.3.3.0 mask 255.255.255.0) to /32 network

Try this it will work 100 percent

check using the command show ip ospf sham-links

Sudhin Jacob CCIE R&S,SP#28680

Mahesh Gohil · ‎12-17-2010

Hello Sai,

Good suggestion to bring sham-link up . But even if your sham link is downyou should be able to ping

resason sham-link is used just to convert inter-area routes to intra-area . So in summary if sham-link is down

routes will travel through mp-bgp

well I found the issue. You need to change mask of loopback0 (used for ldp peering) to /32

like

PE1#interface Loopback0
ip address 1.1.1.1 255.255.255.255
end

PE2#interface Loopback0
ip address 2.2.2.2 255.255.255.255
end

Reason: for /24 mask it creates aggregate label on P routers: short description below why aggregate label is not desirabble and packet is dropped

"Removes the top label in the MPLS label stack and does a Layer 3 lookup on the underlying IP packet. The removed label is the bottom label in the MPLS label stack; otherwise, the datagram is discarded. "

and good news is it is working with ipsec config also. Sorry dear for mis-guiding during first post

Regards

Mahesh

sudjacob · ‎12-17-2010

The Sham link must be created with /32 network and one more issue here when the backdoor link you have to manipulate the cost other wise all traffic goes through the backdoor. Some times when you configure sham link every thing traffic wont go it will be stuck in that case rare chance you have to remove the ospf process from VRF and cretes it. I have done many configuration with sham links. please feel free to get back to me for queries.

Sudhin Jacob CCIE R&S,SP#28680

ceracaza08 · ‎03-12-2012

HI Sudhin,

I'm having here a practice lab, shamlink is up but still 1 Router which should prefer the backdoor still prefers the MPLS link

I didn't do remove the the ospf process from the vrf. I just clear the ospf process of both PE's...

Any inputs on this?