mpls ping returns QQQQQ, though ldp neighborship is established

[aswetha]

Hi Team,

ASR9k is connected to a Nokia router. IGP neighborship is Up, ldp is up. Regular ping also works but Mpls ping doesn't work.

RP/0/RP1/CPU0:KV-ASR9912-AGG2#ping 10.100.235.67 so lo0
Mon Apr 27 13:54:49.735 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.100.235.67, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
RP/0/RP1/CPU0:KV-ASR9912-AGG2#ping mpls ipv4 10.100.235.67/32 fec-type ldp source 10.71.241.223
Mon Apr 27 13:54:58.907 UTC

Sending 5, 100-byte MPLS Echos to 10.100.235.67/32,
timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.

QQQQQ
Success rate is 0 percent (0/5)

RP/0/RP1/CPU0:KV-ASR9912-AGG2#sh route ipv4 10.100.235.67/32
Mon Apr 27 13:56:58.613 UTC

Routing entry for 10.100.235.67/32
Known via "ospf CORE", distance 110, metric 100, labeled SR, type intra area
Installed Apr 27 12:01:46.087 for 01:55:13
Routing Descriptor Blocks
10.198.97.89, from 10.100.235.67, via GigabitEthernet0/2/0/36, Protected
Route metric is 100
10.198.97.67, from 10.100.235.67, via TenGigE0/2/0/1.11, Backup (TI-LFA)
Repair Node(s): 10.100.235.66
Route metric is 120
No advertising protos.

RP/0/RP1/CPU0:KV-ASR9912-AGG2#sh mpls forwarding | i 10.100.235.67/32
Mon Apr 27 13:57:39.708 UTC
32784 524287 10.100.235.67/32 Gi0/2/0/36 10.198.97.89 17851919
32774 10.100.235.67/32 Te0/2/0/1.11 10.198.97.67 0 (!)

RP/0/RP1/CPU0:KV-ASR9912-AGG2#sh cef 10.100.235.67/32
Mon Apr 27 13:58:49.621 UTC
10.100.235.67/32, version 1433563, labeled SR, internal 0x1000001 0x85 (ptr 0x787c9378) [1], 0x0 (0x7877e758), 0xa28 (0x797f86f0)
Updated Apr 27 12:01:46.105
remote adjacency to GigabitEthernet0/2/0/36
Prefix Len 32, traffic index 0, precedence n/a, priority 15
Extensions: context-label:18902
via 10.198.97.89/32, GigabitEthernet0/2/0/36, 12 dependencies, weight 0, class 0, protected [flags 0x400]
path-idx 0 bkup-idx 1 NHID 0x0 [0x79627bd0 0x0]
next hop 10.198.97.89/32
local label 32784 labels imposed {524287}
via 10.198.97.67/32, TenGigE0/2/0/1.11, 29 dependencies, weight 0, class 0, backup [flags 0x300]
path-idx 1 NHID 0x0 [0x79517410 0x0]
next hop 10.198.97.67/32
remote adjacency
local label 32784 labels imposed {32774 18902}

RP/0/RP1/CPU0:KV-ASR9912-AGG2#sh mpls ldp bindings 10.100.235.67/32
Mon Apr 27 14:00:38.467 UTC
10.100.235.67/32, rev 250
Local binding: label: 32784
Remote bindings: (4 peers)
Peer Label
----------------- ---------
10.198.85.3:0 69
10.101.92.4:0 524155
10.100.235.67:0 524287
10.71.241.222:0 32775

[Harold Ritter]

Hi @bsn1980in ,

Can you please give us more information on the issue you are seeing. What platform you use and so on.

Regards,

[bsn1980in]

Hi Harold

The scenario is quite similar AGG01<-> Pre-AGG01 <->Pre-AGG02 <-> Pre-AGG03 <-> AGG02.  All are running ISIS, SR and iBGP between AGG and Pre-AGG routers (No BGP between Pre-AGGs). AGG routers are acting as Route Reflectors for Pre-AGG routers.

AGG routers are Huawei and Pre-AGG routers are Cisco NCS560-7E.

Problem: To start with we were not able to establish iBGP between Pre-AGG01<->AGG02, Pre-AGG03<->AGG01 and Pre-AGG02 <-> AGG01, AGG02. iGBP worked between Pre-AGG01<->AGG01 and Pre-AGG03<->AGG03. Then we found we were not able to ping loopback0 for all the iBGP sessions which were not establishing.

Troubleshooting:
- Control plane worked fine: could see respective routes, labels and CEF entries on all routers.
- We disabled SR on Pre-AGG routers and we were able to ping and establish iBGP. So we concluded that ISIS is working well with IP only but not with SR.
- tried ping sr-mpls but it failed
- tried ping mpls along with "debug mpls oam errror" and got following output

QRP/0/RP0/CPU0:2022 Oct 19 10:09:40.215 BST: isis[1014]: SR-OAM: Responded to OAM query - pfx 1 adj 0 dpm 0 unk 0
QQQRP/0/RP0/CPU0:2022 Oct 19 10:09:40.216 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IGP (routes in FIB 1, routes in CP 0)
Q
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.216 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IP Services
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.216 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Failed to get label stack, phy out intf and nexthop for IP FEC
RP/0/RP0/CPU0:CALTRP-PAG01#RP/0/RP0/CPU0:2022 Oct 19 10:09:40.219 BST: isis[1014]: SR-OAM: Responded to OAM query - pfx 1 adj 0 dpm 0 unk 0
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.219 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IGP (routes in FIB 1, routes in CP 0)
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.219 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IP Services
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.219 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Failed to get label stack, phy out intf and nexthop for IP FEC
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.221 BST: isis[1014]: SR-OAM: Responded to OAM query - pfx 1 adj 0 dpm 0 unk 0
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.221 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IGP (routes in FIB 1, routes in CP 0)
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.221 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IP Services
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.222 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Failed to get label stack, phy out intf and nexthop for IP FEC
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.223 BST: isis[1014]: SR-OAM: Responded to OAM query - pfx 1 adj 0 dpm 0 unk 0
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.223 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IGP (routes in FIB 1, routes in CP 0)
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.223 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IP Services
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.223 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Failed to get label stack, phy out intf and nexthop for IP FEC
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.226 BST: isis[1014]: SR-OAM: Responded to OAM query - pfx 1 adj 0 dpm 0 unk 0
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.227 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IGP (routes in FIB 1, routes in CP 0)
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.227 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Consistency check failed for IP Services
RP/0/RP0/CPU0:2022 Oct 19 10:09:40.227 BST: lspv_server[1149]: [798123840] ERR-Pkt: : Failed to get label stack, phy out intf and nexthop for IP FEC
RP/0/RP1/CPU0:2022 Oct 19 10:10:16.643 BST: isis[1015]: SR-NODE-LBL: Success released lbl=106377 pfx 10.192.16.121/32 tbl 0xe0000000 So far 6283 success 0 fail

We have already raised a TAC with Cisco but waiting for their feedback.  We have deployed more than 100 NCS560 and haven't seen this problem.  It seems there is new hardware stock of devices we recently got and got hit by a bug.

Regards/Bharat

[bsn1980in]

Hi All

Cisco TAC finally released a SMU to fix the problem.  It seems MPLS package isn't enabled by default (IOSXR - 7.2.2). To identify the problem, try to run following commands:

show ETHer-ea interface bundle-ether 1 location 0/rp0/cpu0 | i MPLS   >>> use correct BE number as per your setup

expected output: MPLS : 1   >>> if MPLS: 0 or blank then there is a problem

show controllers fia diagshell 0 "diag pp rif core=1" location 0/RP0/CPU0  >>> if no output, try different core=value or location like RP1

expected output: ^M routing_enablers: IPV4UC IPV4MC MPLS >>>> if MPLS is missing from the list then there is a problem

Hope it helps and save time for you!

Regards

Bharat

[Harold Ritter]

Thanks for the feedback @bsn1980in . Very much appreciated.