topic Re: MPLS LDP Session Establishment failed in Routing and SD-WAN

MPLS LDP Session Establishment failed

Steve — Fri, 07 Jul 2023 06:48:36 GMT

Hello,

i have tried to connect an

ASR1004(RP2,ESP20,SIP10) and a C6504E(VS-S720-10G)

over 10G direct links.

10G port on C6504E

is located on a

WS-X6708-10GE ( with WS-F6700-DFC3C)

linecard. Both interfaces are

MPLS enabled via mpls ip

IP connectivity between the

mpls loopbacks

of the routers is working fine, can ping with max

MTU 9216 and df-bit set

The tcpdump (via monitor session) from the interface on the

C6504E

shows "LDP Hello Messages" from both routers.

ASR1004

which has the higher

LSR-ID

initiates the LDP TCP Connection to

C6504E

but connection is refused on

C6504E

by sending TCP (RST,ACK).

What could be the reason for dropping the connection ? i have attached a zip with the pcap trace from interface traffic

ASR1004	loopback1	172.16.217.254/32
C6504E	loopback1	172.16.211.254/32
ASR1004	te1/0/0	172.16.220.14/30
C6504E	te2/1	172.16.220.13/30

Global setting on both routers:

mpls ldp router-id Loopback1 force

C6504E#ping 172.16.220.14 size 9216 df-bit source lo1 repeat 10
Type escape sequence to abort.
Sending 10, 9216-byte ICMP Echos to 172.16.220.14, timeout is 2 seconds:
Packet sent with a source address of 172.16.211.254
Packet sent with the DF bit set
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 1/2/4 ms

ASR1004#ping 172.16.220.13 source lo1 size 9216 df-bit repeat 10
Type escape sequence to abort.
Sending 10, 9216-byte ICMP Echos to 172.16.220.13, timeout is 2 seconds:
Packet sent with a source address of 172.16.217.254
Packet sent with the DF bit set
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 2/2/3 ms

Re: MPLS LDP Session Establishment failed

MHM Cisco World — Fri, 07 Jul 2023 06:50:50 GMT

show mpls ldp neighbor

I need to see this

Re: MPLS LDP Session Establishment failed

Steve — Fri, 07 Jul 2023 06:52:26 GMT

C6504E

there are other LDP Sessions active which i won't post here - so i have selected the affected interface

C6504E#sh mpls ldp neighbor te2/1
C6504E#

ASR1004#sh mpls ldp neighbor
ASR1004#

As there is no LDP Session established we won't see here anything (i think)

Some more infos

ASR1004#sh mpls ldp discovery detail
Local LDP Identifier:
172.16.217.254:0
Discovery Sources:
Interfaces:
TenGigabitEthernet1/0/0 (ldp): xmit
Enabled: Interface config
Hello interval: 5000 ms; Transport IP addr: 172.16.217.254
Clients: IPv4, mLDP

C6504E#sh mpls ldp discovery detail
Local LDP Identifier:
172.16.211.254:0
Discovery Sources:
Interfaces:
TenGigabitEthernet2/1 (ldp): xmit
Enabled: Interface config
Hello interval: 5000 ms; Transport IP addr: 172.16.211.254
Clients: IPv4, mLDP

Re: MPLS LDP Session Establishment failed

MHM Cisco World — Wed, 07 Jun 2023 16:20:33 GMT

Udp port 646 is open or not ?

Re: MPLS LDP Session Establishment failed

Steve — Fri, 07 Jul 2023 06:53:44 GMT

Yes

 no access-lists

on interfaces

ASR1004#telnet 172.16.211.254 646 /source-interface lo1
Trying 172.16.211.254, 646 ...
% Connection refused by remote host

As you can see TCP connection is refused - so port should be open

In the meanwhile i have done some more connectivity tests and it seems there is packetloss on the link. 10 pings was not enough to loose packets

ASR1004#ping 172.16.211.254 size 1500 df-bit repeat 100 source lo1
Type escape sequence to abort.
Sending 100, 1500-byte ICMP Echos to 172.16.211.254, timeout is 2 seconds:
Packet sent with a source address of 172.16.217.254
Packet sent with the DF bit set
!!!!!!!!!!!.!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!!.
!!!!!!!!!!!.!!!!!!!!!!!.!!!!!!
Success rate is 92 percent (92/100), round-trip min/avg/max = 1/1/1 ms

i will have to fix this before we can follow up - will take some time (until next week) as this is a remote site - thanks for your help

Re: MPLS LDP Session Establishment failed

MHM Cisco World — Wed, 07 Jun 2023 16:56:37 GMT

I think I found issue here

Re: MPLS LDP Session Establishment failed

Harold Ritter — Fri, 07 Jul 2023 06:56:46 GMT

Hi @Steve ,

For the output you provided, it looks like neither router receive the LDP hellos from the neighbour, as both of them show as

xmit

when they should as

xmit/recv

Regards,

Re: MPLS LDP Session Establishment failed

MHM Cisco World — Wed, 07 Jun 2023 22:24:56 GMT

I see in ldp discovery thst you use multiPoint Ldp (mLDP) not LDP

This change source destination of ldp packet.

Disbale mldp under interface.

Re: MPLS LDP Session Establishment failed

Steve — Fri, 07 Jul 2023 07:01:35 GMT

Hello Harold,

i think until LDP session isn't established interface will always be

xmit

If you check the the pcap file you can see hellos from both routers. i have enabled debug on

ASR1004

and there you can see hello is working and

ASR1004

(Router with the higher LSR-ID) tries to establish a TCP connection (LDP Session) to

C6504E

ASR1004

MPLS ldp:
LDP transport events debugging is on for interface

TenGigabitEthernet1/0/0

LDP transport connection events debugging is on for interface

TenGigabitEthernet1/0/0

*Jun 7 22:11:31.010: ldp: Send ldp hello; TenGigabitEthernet1/0/0, src/dst 172.16.220.14/224.0.0.2, inst_id 0
*Jun 7 22:11:34.993: ldp: Rcvd ldp hello; TenGigabitEthernet1/0/0, from 172.16.220.13 (172.16.211.254:0), intf_id 0, opt 0xC
*Jun 7 22:11:34.993: ldp: ldp Hello from 172.16.220.13 (172.16.211.254:0) to 224.0.0.2, opt 0xC
*Jun 7 22:11:34.993: ldp: New adj 0x7F19D75FA358 for 172.16.211.254:0, TenGigabitEthernet1/0/0
*Jun 7 22:11:34.993: ldp: adj_addr/xport_addr 172.16.220.13/172.16.211.254
*Jun 7 22:11:34.993: ldp: Request adj send hello back on TenGigabitEthernet1/0/0 to (xport addr 172.16.211.254) in 50 msec
*Jun 7 22:11:34.993: ldp: local interface = TenGigabitEthernet1/0/0, holdtime = 15000, peer 172.16.220.13 holdtime = 15000
*Jun 7 22:11:34.993: ldp: Link intvl min cnt 2, intvl 5000, interface TenGigabitEthernet1/0/0
*Jun 7 22:11:34.993: ldp: Opening ldp conn; adj 0x7F19D75FA358, 172.16.217.254 <-> 172.16.211.254; with normal priority
*Jun 7 22:11:34.993: ldp: :{ldp conn 172.16.217.254:64033=>172.16.211.254:646}: Registered tcb 0x7F19CA59D528 [key 48199] with LDP TCB database, total 1
*Jun 7 22:11:34.994: ldp: Conn failed (TCP connect notify)!; adj 0x7F19D75FA358, 172.16.220.13
*Jun 7 22:11:34.994: ldp: : rx_notify tcp_rc 5, adj_state 3
*Jun 7 22:11:34.994: ldp: {ldp conn 172.16.217.254:64033=>172.16.211.254:646} (Te1/0/0) (adj 0x7F19D75FA358): processing transport close request
*Jun 7 22:11:34.994: ldp: Unregistered from LDP TCB database tcb 0x7F19CA59D528 [key 48199], total 0
*Jun 7 22:11:34.994: ldp: Close tcp connection to 172.16.211.254
*Jun 7 22:11:34.994: ldpx_tcp: {ldp conn 172.16.217.254:64033=>172.16.211.254:646}: client requesting close of tcb
*Jun 7 22:11:34.994: ldp: Adj 0x7F19D75FA358; state set to closed
*Jun 7 22:11:35.043: ldp: Send ldp hello; TenGigabitEthernet1/0/0, src/dst 172.16.220.14/224.0.0.2, inst_id 0
*Jun 7 22:11:35.553: ldp: Scan listening TCBs

Re: MPLS LDP Session Establishment failed

Steve — Fri, 07 Jul 2023 06:54:37 GMT

This doesn't change anything

interface TenGigabitEthernet2/1
mtu 9216
ip address 172.16.220.13 255.255.255.252
no ip redirects
no ip proxy-arp
no mpls mldp
mpls ip

C6504E#sh mpl ldp discovery detail | begin 2/1
TenGigabitEthernet2/1 (ldp): xmit
Enabled: Interface config
Hello interval: 5000 ms; Transport IP addr: 172.16.211.254
Clients: IPv4, mLDP

Maybe this indicates only interface is capable of mLDP or it refers to this

Directly Connected MPLS LDP Sessions

If a label switch router (LSR) is one hop from its neighbor, it is directly connected to its neighbor. The LSR sends out Label Distribution Protocol (LDP) link Hello messages as User Datagram Protocol (UDP) packets to all the devices on the subnet (multicast). A neighboring LSR may respond to the link Hello message, allowing the two devices to establish an LDP session. This is called basic discovery.

Re: MPLS LDP Session Establishment failed

MHM Cisco World — Wed, 07 Jun 2023 22:54:42 GMT

you disable it in C6000 side, disable it in ASR1K side

Re: MPLS LDP Session Establishment failed

Harold Ritter — Fri, 07 Jul 2023 07:07:22 GMT

Hi @Steve ,

> i think until LDP session isn't established interface will always be

xmit

This

xmit/rcvd

state only pertains to LDP hellos. So the

show mpls ldp dis det

output you provided clearly state that the routers did not consider to have two way reachability.

Regards,

Re: MPLS LDP Session Establishment failed

Harold Ritter — Thu, 08 Jun 2023 00:44:55 GMT

Hi @Steve ,

You probably want to run the same debug command on the 6500 side to see what it says.

Regards,

Re: MPLS LDP Session Establishment failed

Steve — Fri, 07 Jul 2023 06:55:14 GMT

Sorry my answer was too inaccurate - it was changed on both sides

ASR1004#sh run int te1/0/0
Building configuration...

Current configuration : 302 bytes
!
interface TenGigabitEthernet1/0/0
mtu 9216
ip address 172.16.220.14 255.255.255.252
no ip redirects
no ip proxy-arp
load-interval 30
mpls ip
no mpls mldp
!
ASR1004#sh mpls ldp discovery detail
Local LDP Identifier:
172.16.217.254:0
Discovery Sources:
Interfaces:
TenGigabitEthernet1/0/0 (ldp): xmit
Enabled: Interface config
Hello interval: 5000 ms; Transport IP addr: 172.16.217.254
Clients: IPv4, mLDP

Re: MPLS LDP Session Establishment failed

Steve — Fri, 07 Jul 2023 07:17:39 GMT

Hi @Harold Ritter

this is a little bit strange now - the debug on

C6504E

shows no receiving hello messages but if i check the pcap trace i can see it. The trace is taken from

C6504E

with a monitor session (source is te2/1 in both directions).

As @MHM Cisco World noted earlier 'is port udp 646 open' - is there a way to check this ? ( i only have verified tcp)

Below the captured hello packet on

C6504E

which was sent from ASR1004 and the debug log from

C6504E

DEBUG LOG from C6504E (no received ldp hello packets)

031445: Jun 8 11:14:47.821 CEST: ldp: Send ldp hello; TenGigabitEthernet2/1, src/dst 172.16.220.13/224.0.0.2, inst_id 0
031446: Jun 8 11:14:51.617 CEST: ldp: bytes_written = 34, at offset = 160
031447: Jun 8 11:14:51.617 CEST: ldp: Send ldp hello; TenGigabitEthernet2/1, src/dst 172.16.220.13/224.0.0.2, inst_id 0
031448: Jun 8 11:14:55.474 CEST: ldp: bytes_written = 34, at offset = 160
031449: Jun 8 11:14:55.474 CEST: ldp: Send ldp hello; TenGigabitEthernet2/1, src/dst 172.16.220.13/224.0.0.2, inst_id 0
031450: Jun 8 11:14:59.430 CEST: ldp: bytes_written = 34, at offset = 160
031451: Jun 8 11:14:59.430 CEST: ldp: Send ldp hello; TenGigabitEthernet2/1, src/dst 172.16.220.13/224.0.0.2, inst_id 0

i have cross checked the debug command on a different interface (on the same Linecard) which has a working LDP Session and there i can see received and send hello packets.

So it seems C6504E is receiving the hello packets from ASR1004 but it drops it

Re: MPLS LDP Session Establishment failed

MHM Cisco World — Thu, 08 Jun 2023 09:51:04 GMT

I run lab and test some thing here
the mLDP is not effect the LDP session
the LDP use two protocol UPD for hello and TCP for LDP binding
so I need some time to more check

Re: MPLS LDP Session Establishment failed

MHM Cisco World — Fri, 07 Jul 2023 07:20:09 GMT

that first thing and I mention before,
C6504E have CoPP can drop any packet toward the CPU (LO for MPLS ID)
here in lab I add ACL deny udp 646 and result as you have both side have Xmit ONLY
share below for C6500


show class-map
show policy-map
show policy-map control-plane

Re: MPLS LDP Session Establishment failed

Steve — Fri, 07 Jul 2023 07:21:05 GMT

Hi - no CoPP active

C6504E# show class-map
Class Map match-any class-default (id 0)
Match any

C6504E#show policy-map
Policy Map PE-SET-MPLS-EXP-IMP-VIDEO
Description: ### Mark MPLS experimental Bits to reflect QoS / VIDEO / CoS-ToS: 4 ###
Class class-default
set mpls experimental imposition 4

Policy Map PE-SET-MPLS-EXP-IMP-RESERVED
Description: ### Mark MPLS experimental Bits to reflect QoS / RESERVED / CoS-ToS: 1 ###
Class class-default
set mpls experimental imposition 1

Policy Map PE-SET-MPLS-EXP-IMP-BUSINESS
Description: ### Mark MPLS experimental Bits to reflect QoS / BUSINESS / CoS-ToS: 3 ###
Class class-default
set mpls experimental imposition 3

Policy Map PE-SET-MPLS-EXP-IMP-BESTEFFORT
Description: ### Mark MPLS experimental Bits to reflect QoS / BESTEFFORT / CoS-ToS: 0 ###
Class class-default
set mpls experimental imposition 0

Policy Map PE-SET-MPLS-EXP-IMP-REALTIME
Description: ### Mark MPLS experimental Bits to reflect QoS / REALTIME / CoS-ToS: 5 ###
Class class-default
set mpls experimental imposition 5

Policy Map PE-SET-MPLS-EXP-IMP-STANDARD
Description: ### Mark MPLS experimental Bits to reflect QoS / STANDARD / CoS-ToS: 2 ###
Class class-default
set mpls experimental imposition 2


C6504E#show policy-map control-plane
C6504E#

Re: MPLS LDP Session Establishment failed

MHM Cisco World — Fri, 07 Jul 2023 07:22:47 GMT

SW(config)#mls qos protocol mpls ldp  precedence change ip-precedence(used to map the dscp to cos value)

there is no CoPP run I check some bug and find one that can cause this issue, the QoS of LDP is low-priority and this make CPU drop it, check the workaround
CSCsd31503 : Bug Search Tool (cisco.com)

Re: MPLS LDP Session Establishment failed

Steve — Thu, 08 Jun 2023 10:25:13 GMT

Nice catch - i have to check this later and will update you asap.