MPLS LDP Session Establishment failed

[Steve]

Hello,

i have tried to connect an

ASR1004(RP2,ESP20,SIP10) and a C6504E(VS-S720-10G)

over 10G direct links.

10G port on C6504E

is located on a

WS-X6708-10GE ( with WS-F6700-DFC3C)

linecard. Both interfaces are

MPLS enabled via mpls ip

IP connectivity between the

mpls loopbacks

of the routers is working fine, can ping with max

MTU 9216 and df-bit set

The tcpdump (via monitor session) from the interface on the

C6504E

shows "LDP Hello Messages" from both routers.

ASR1004

which has the higher

LSR-ID

initiates the LDP TCP Connection to

C6504E

but connection is refused on

C6504E

by sending  TCP (RST,ACK).

What could be the reason for dropping the connection ? i have attached a zip with the pcap trace from interface traffic

ASR1004	loopback1	172.16.217.254/32
C6504E	loopback1	172.16.211.254/32
ASR1004	te1/0/0	172.16.220.14/30
C6504E	te2/1	172.16.220.13/30

Global setting on both routers:

mpls ldp router-id Loopback1 force

C6504E#ping 172.16.220.14 size 9216 df-bit source lo1 repeat 10
Type escape sequence to abort.
Sending 10, 9216-byte ICMP Echos to 172.16.220.14, timeout is 2 seconds:
Packet sent with a source address of 172.16.211.254
Packet sent with the DF bit set
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 1/2/4 ms

ASR1004#ping 172.16.220.13 source lo1 size 9216 df-bit repeat 10
Type escape sequence to abort.
Sending 10, 9216-byte ICMP Echos to 172.16.220.13, timeout is 2 seconds:
Packet sent with a source address of 172.16.217.254
Packet sent with the DF bit set
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 2/2/3 ms

[Denis_from_BE]

Hi, 

it looks to me that as long as the C6504E can't intercept the LDP Hello packets properly, it won't go further, including the process of the SYN sent out by ASR1004. (ASR1004 received the Hello from C6504E so it seems to continues establishment of the transport session with its facing peer)

Would makes sense that C6504E refuses the TCP establishement if he does not know more about its facing peer (Router identifier and label space to use, present in the Hello messages)

I know the packet captures was taken from the C6504E perspective but where have you done it exactly ?

[Steve]

Hi Denis,

maybe this can help to get a better view of the setup

[MHM Cisco World]

one more think I see in your wiresharke capture is the ethernet header, there is 4 bytes add to header, this 4 bytes can be

vlan-id

or as I see it FCS add to frame. 
the hello not contain this header the TCP have. 
do you run FCS in one side ?

[Denis_from_BE]

Hi, 

thanks for the diagram, the capture is clearer. 

@MHM Cisco World the pcap shows the use of

vlan-id 1033

@Steve it appears that other people experienced similar issues because of wrong linecard programming. 
Can you list us the installed modules of your nodes? Is the port used on the

C6504E

(facing to ASR) located in the same LC as the other working peers (toward the other working PE's) ? Or you use another LC for this new connection ? 

How are the connections made from the ASR standpoint ? 

[MHM Cisco World]

I Know that, but check the header one side using padding & trailer  and other no.

[Denis_from_BE]

Yes indeed, good catch. 
@Steve can you investigate why such use of header overhead from the ASR sent frames ? 

[Steve]

I think the "original frame" from the ASR is smaller than the required minimum frame size of 64 byte so the ASR has to do padding. All other frames (LDP hello) are large enough and no padding is done. The (RST,ACK) packets from

C6504E

has a padding because they are smaller than 64 byte too.

[Steve]

Hi,

C6504E#sh module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 5 Supervisor Engine 720 10GE (Active) VS-S720-10G SAL11477LS8
2 8 CEF720 8 port 10GE with DFC WS-X6708-10GE SAL14049XKJ
3 24 CEF720 24 port 1000mb SFP WS-X6724-SFP SAL160639CZ

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 001d.45e1.ac00 to 001d.45e1.ac07 2.0 8.5(4) 15.1(2)SY10 Ok
2 001f.6ca4.3c88 to 001f.6ca4.3c8f 2.5 12.2(18r)S1 15.1(2)SY10 Ok
3 c464.1304.8ce8 to c464.1304.8cff 5.1 12.2(18r)S1 15.1(2)SY10 Ok

Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
1 Policy Feature Card 3 VS-F6K-PFC3C SAL1135YYUZ 1.0 Ok
1 MSFC3 Daughterboard VS-F6K-MSFC3 SAD114304UX 1.0 Ok
2 Distributed Forwarding Card WS-F6700-DFC3C SAL14028B86 1.6 Ok
3 Distributed Forwarding Card WS-F6700-DFC3C SAL1535P0SM 1.5 Ok

Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass

ASR1004#sh platform
Chassis type: ASR1004

Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ASR1000-SIP10 ok 6d05h
0/0 SPA-1X10GE-L-V2 ok 6d05h
0/1 SPA-1X10GE-L-V2 ok 6d05h
0/3 SPA-1X10GE-L-V2 ok 6d05h
1 ASR1000-SIP10 ok 6d05h
1/0 SPA-1X10GE-L-V2 ok 6d05h
1/1 SPA-1X10GE-L-V2 ok 6d05h
1/3 SPA-1X10GE-L-V2 ok 6d05h
R0 ASR1000-RP2 ok, active 6d05h
F0 ASR1000-ESP20 ok, active 6d05h
P0 ASR1004-PWR-AC ok 6d05h
P1 ASR1004-PWR-AC ok 6d05h

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 09111601 16.3(2r)
1 07091401 16.3(2r)
R0 13092401 16.3(2r)
F0 08041102 16.3(2r)

The connections from

C6504E

to the other working peers are originated on the same Linecard. There are other connections too which are using the 10G ports located on the supervisor. I think there must be something wrong with the new

ASR1004

I will be on site at thursday and will do some tests with other hardware.

[Steve]

i was on site today and did some extensive testing - i have nailed down the problem to the ports on the WS-6708-10GE Linecard in C6504E. 

STEPS i have done

• replaced all affected tranceiver and cables between

  ASR1004 and C6504E 

  => failure still exist
• move config from

  C6504E te2/1 to te2/2

  => failure still exist
• replaced

  ESP,SIP  and SPA on ASR1004

  => failure still exist
• moved uplink on

  ASR1004 te1/0/0

  to a cisco 1921 (over a nexus3k as converter between 1G and 10G) => everything OK
• moved uplink from

  C6504E te2/1

  to a cisco 1921 (over a nexus3k as converter between 1G and 10G) => failure still exist

So it seems ports

te2/1 and te2/2 on C6504E

are not working correctly. All other ports on the linecard are working as expceted, These both ports were in use some months ago - i have checked confiig history and the last setup was

interface TenGigabitEthernet2/1
 mtu 9000
 no ip address 
 no ip redirects 
 no ip proxy-arp
 no keepalive 
 xconnect 172.16.214.254 901020007 encapsulation mpls pw-class TE-via-9255

 interface TenGigabitEthernet2/2
  mtu 9000
  no ip address 
  no ip redirects 
  no ip proxy-arp
  no keepalive 
  xconnect 172.16.214.254 901020008 encapsulation mpls pw-class TE-via-9255

Maybe config wasn't clean removed - i will look for an maintenance window to reload the linecard and see if it will clear the problem. If this doesn't help too i will replace the linecard on C6504E. Post will be updated after reload was done.

[Steve · Accepted solution]

hi all,

i restarted the module in slot 2 on

C6504E  with hw-module module 4 reset

but nothing changed. so i decided to reboot the entire

C6504E

and now all is working.

C6504E#sh mpls ldp neighbor te2/1
Peer LDP Ident: 172.16.217.254:0; Local LDP Ident 172.16.211.254:0
TCP connection: 172.16.220.14.52729 - 172.16.220.13.646
State: Oper; Msgs sent/rcvd: 67/12; Downstream
Up time: 00:03:59
LDP discovery sources:

Thanks all to you for input 

[MHM Cisco World]

Hi Steve 

Just what to ask' are

c6500 is config as vss

? Are you use port channel to both

vss peer

?

Thanks 

MHM

[Steve]

Hello,

Sorry for the late response - no there is no

VSS

configuration - all connections are

Point-2-Point links with /30 Subnet and IS-IS enabled

( no L2 aggregation)