Dear Xander,

Arun Seetha · ‎06-05-2014

Hi All,

I am facing lot of challenges related to CEF issue in IOS-XR. Router shows Glean adjacency for connected IP subnets.

Sometime issue gets resolved by rebooting routers, but many a times it doesn't get resolved even after doing so.

To troubleshoot the same, we have done many steps like Clearing Arp, CEF table, adding static ARP entry, putting static routes etc etc; but no luck.

Can someone please help me to know that what is the root cause of the issue and the remedy of same.

I am preparing for CCIE SP Lab, and the challenge is that reboot of router is not allowed during exam. EVen if it will be allowed, then not sure that whether it will get resolved.

Please help.

Below is the details of IOS-XR image for reference.

System image file is "disk0:c12k-os-mbi-3.9.1/mbiprp-rp.vm"

cisco 12410/PRP (7457) processor with 2097152K bytes of memory.
7457 processor at 1266Mhz, Revision 1.2

1 Cisco 12000 Series Performance Route Processor

RP/0/8/CPU0:R4#sh cef vrf ABC 172.9.114.11 detail
Thu Jun 5 17:31:11.132 UTC
0.0.0.0/0, version 0, proxy default, default route handler, drop adjacency, internal 0x40020201 (ptr 0x9cd5c4f8) [1], 0x0 (0x9cd2a248), 0x0 (0x0)
Updated Jun 5 16:08:24.368
Prefix Len 0, traffic index 0, precedence routine (0)
gateway array (0x9cbfe560) reference count 1, flags 0x4000, source internal (4),
[2 type 3 flags 0x109001 (0x9cc942a8) ext 0x0 (0x0)]
LW-LDI[type=3, refc=1, ptr=0x9cd2a248, sh-ldi=0x9cc942a8]
via point2point, 7 dependencies, weight 0, class 0 [flags 0x0]
next hop point2point
drop adjacency

Load distribution: 0 (refcount 2)

Hash OK Interface Address
0 Y Unknown drop

Regards,

Arun Seetha

xthuijs · ‎06-07-2014

Arun:

in your example the route doesnt exist and follows the default which doesnt exist either hence it is a drop adj.

But a glean adj is perfectly normal and should be there for multi-access networks.

It basically means that there are multiple endpoints in this subnet and we need to complete an L2 ADJ for those in order to resolve them.

Example:

gig 0/0/0/0 54.1.1.1/24

This results in a cef ADJ of GLEAN on the subnet:

RP/0/RSP0/CPU0:A9K-BNG#show cef 54.1.1.0/24 det loc 0/0/cpu0
Sat Jun 7 16:14:06.298 EDT
54.1.1.0/24, version 32, attached, connected, glean adjacency, internal 0xc0000c1 0x0 (ptr 0x8b2bd5b0) [1], 0x0 (0x8abe94e0), 0x0 (0x0)
Updated Jun 7 06:35:06.649
Prefix Len 24, traffic index 0, precedence n/a, priority 0
gateway array (0x8a95fbe8) reference count 1, flags 0x0, source rib (6), 0 backups
                [2 type 3 flags 0x10101 (0x8aa5a2c8) ext 0x0 (0x0)]
LW-LDI[type=3, refc=1, ptr=0x8abe94e0, sh-ldi=0x8aa5a2c8]
   via GigabitEthernet0/0/0/0, 2 dependencies, weight 0, class 0 [flags 0x8]
    path-idx 0 NHID 0x0 [0x8a6c57c4 0x0]
     glean adjacency

Load distribution: 0 (refcount 2)

Hash OK Interface Address
0 Y GigabitEthernet0/0/0/0 glean

But we will have resolved entries also: This is my local address:

RP/0/RSP0/CPU0:A9K-BNG#show cef 54.1.1.1/32 det loc 0/0/cpu0
Sat Jun 7 16:16:10.035 EDT
54.1.1.1/32, version 31, attached, receive
Updated Jun 7 06:35:06.650
Prefix Len 32
internal 0xc000091 (ptr 0x8b2bd534) [3], 0x0 (0x8abe7258), 0x0 (0x0)

For a peer address that I resolved with an arp:

RP/0/RSP0/CPU0:A9K-BNG#sh arp | i 54.1
Sat Jun 7 16:17:15.949 EDT
54.1.1.1 - 6666.4444.2222 Interface ARPA GigabitEthernet0/0/0/0
54.1.1.2 02:18:17 0006.2aaa.24a8 Dynamic ARPA GigabitEthernet0/0/0/0

RP/0/RSP0/CPU0:A9K-BNG#show cef 54.1.1.2/32 det loc 0/0/cpu0
Sat Jun 7 16:16:47.405 EDT
54.1.1.2/32, version 0, internal 0x4080001 0x0 (ptr 0x8b2bdb04) [1], 0x0 (0x8abe9b60), 0x0 (0x0)
Updated Jun 7 06:35:07.492
local adjacency 54.1.1.2
Prefix Len 32, traffic index 0, Adjacency-prefix, precedence n/a, priority 0
gateway array (0x8a960368) reference count 1, flags 0x0, source aib-hi (2), 0 backups
                [2 type 3 flags 0x10101 (0x8aa5ac48) ext 0x0 (0x0)]
LW-LDI[type=3, refc=1, ptr=0x8abe9b60, sh-ldi=0x8aa5ac48]
   via 54.1.1.2, GigabitEthernet0/0/0/0, 4 dependencies, weight 0, class 0 [flags 0x0]
    path-idx 0 NHID 0x4 [0x8ddff058 0x0]
    next hop 54.1.1.2
    local adjacency

Load distribution: 0 (refcount 2)

Hash OK Interface Address
0 Y GigabitEthernet0/0/0/0 54.1.1.2

And when you look at an unresolved adj it will point to the parent glean one:

RP/0/RSP0/CPU0:A9K-BNG#show cef 54.1.1.3 det loc 0/0/cpu0
Sat Jun 7 16:18:07.604 EDT
54.1.1.0/24, version 32, attached, connected, glean adjacency, internal 0xc0000c1 0x0 (ptr 0x8b2bd5b0) [1], 0x0 (0x8abe94e0), 0x0 (0x0)
Updated Jun 7 06:35:06.654
Prefix Len 24, traffic index 0, precedence n/a, priority 0
gateway array (0x8a95fbe8) reference count 1, flags 0x0, source rib (6), 0 backups
                [2 type 3 flags 0x10101 (0x8aa5a2c8) ext 0x0 (0x0)]
LW-LDI[type=3, refc=1, ptr=0x8abe94e0, sh-ldi=0x8aa5a2c8]
   via GigabitEthernet0/0/0/0, 2 dependencies, weight 0, class 0 [flags 0x8]
    path-idx 0 NHID 0x0 [0x8a6c57c4 0x0]
     glean adjacency

Load distribution: 0 (refcount 2)

Hash OK Interface Address
0 Y GigabitEthernet0/0/0/0 glean

Note that it is important to look at the location, because omitting that looks at the RP. Since the ADJ's are held on the LC, they generally look "REMOTE" from the RP or a different linecard then where the egress interface is.

2 more suggestions:

1) look at cisco live 2904 from last year and this year (2014) for some more detailed cef troubleshooting and understanding this output.

2) I think that 39x is not the right release to study on. you will want to use 43 for instance. 39x 40x are now end of life and I think even 41x is also (or soon).

regards

xander

Arun Seetha · ‎06-08-2014

Hi Xander ,

Thanks for your time and efforts to explain it.

In your example, you referred outputs for three IP Addresses 54.1.1.1/32, 54.1.1.2/32 and 54.1.1.3/32.

In the output of 54.1.1.3/32, Adjacency shows as Glean, which I could not understand. You mentioned that it is "Unresolved Adjacency"; does that mean that this IP address host doesn't exist in LAN ??

In my case, exactly the same Glean Adjacency is showing in output; however the host exists and even the BGP neighborship comes up with that host and routes gets exchanged. But surprisingly both router's directly connected interface does not ping to each other. It means Control Plane works, but Data Plane fails. I am unable to understand this behavior.

Please suggest.

Regards,

Arun

xthuijs · ‎06-08-2014

Hi arun,

a glean is adj forms when there are multiple endpoints and we need an extra step to resolve the L2 adj. For ipv4 that means an arp entry, for v6 a neighbor for instance. Because the L2 adj for 54.1.1.3 is unresolved, we keep it glean so that when there is a packet for that destination we will punt to assist in resolution (eg arp request). It will resolve when an L2 adj is completed:

eg:

RP/0/RSP0/CPU0:A9K-BNG(config)#arp 54.1.1.3 0000.2222.2222 arpa

RP/0/RSP0/CPU0:A9K-BNG#show cef 54.1.1.3 loc 0/0/cPU0
Sun Jun 8 06:53:14.526 EDT
54.1.1.3/32, version 0, internal 0x4080001 0x0 (ptr 0x93028464) [1], 0x0 (0x8abebcb0), 0x0 (0x0)
Updated Jun 8 06:53:09.010
local adjacency 54.1.1.3
Prefix Len 32, traffic index 0, Adjacency-prefix, precedence n/a, priority 0
   via 54.1.1.3, GigabitEthernet0/0/0/0, 3 dependencies, weight 0, class 0 [flags 0x0]
    path-idx 0 NHID 0x0 [0x8ddff8f0 0x0]
    next hop 54.1.1.3
    local adjacency

In your case there could be a variety of issues:

since the bgp peering comes up, the arp entry is resolved.
if you have a /24 on the connecting interface, the /24 will always be glean. the /32's for the 2 endpoints should show resolved as shown with some sample outputs above.
unresolved /32's in the /24 will remain glean and result in punts.
in older releases ping isnt selecting the optimum source address like IOS does by nature. IOS selects the egress interface to that ping address as source. You may be able to fix this problem by doing say a ping 54.1.1.2 source g 0/0/0/0 (where g0/0/0/0 is my source interface with 54.1.1.1).
You may get more joy out of a more mature release as referenced above.

xander

memphis003 · ‎08-02-2017

Dear Xander,

You wrote that "unresolved /32's in the /24 will remain glean and result in punts".

Can we somehow rate-limit those punts n XR, like we do it in IOS on 7600 with the command "mls rate-limit unicast cef glean 500 20".

Thanks in advance.

Tom

xthuijs · ‎08-02-2017

hi tom,

yup that is done by default already actually by lpts.

it follows the punt adj exception and is set for already, but you can tune that via:

RP/0/RP0/CPU0:ASR9922(config)#lpts punt police location 0/5/CPU0 exception adjacency ?

rate Rate in PPS

and verify via:

RP/0/RP0/CPU0:ASR9922#show lpts pifib hardware static-police location 0/5/CPU0

Wed Aug 2 07:29:32.846 EDT

-------------------------------------------------------------

Node 0/5/CPU0:

-------------------------------------------------------------

Burst = 100ms for all flow types

-------------------------------------------------------------

Punt Reason SID Flow Rate Burst Rate Accepted Dropped Destination

----------------------- ----------------------- --------- --------- -------------------- -------------------- -----------

PUNT_ALL NETIO_HI 1000 200 0 0 Local

cheers!

xander

CEF Glean or Drop Adjacency issue