NCS CEF resources limitation

[alirezaroshanaiezade]

We have purchased 4 NCS 5011 products and we are running MPLS, OSPF, and BGP protocols(MPLS VPN Scenario) on this platform.
After establishing a couple bgp , ospf and mpls neighborship on this device and receiving more than 5K routes via vpnv4 address family,
we started to encounter unusual behavior regarding new additional routes and connected interfaces. it basically dropped new routes and didn't allow to enter its cef table. based on cef resources of the device and show commands cef table was dropping new routes, here is the related output :
RP/0/RP0/CPU0:#show cef resource hardware egress

Fri Mar 4 02:19:18.443 UTC
CEF resource availability summary state: YELLOW
CEF will drop route updates
No. of times HW caused oor: 2
CEF entered oor at : Mar 4 00:54:17.868
CEF came out of oor at : Mar 4 00:32:37.369
ipv4 shared memory resource: GREEN
ipv6 shared memory resource: GREEN
mpls shared memory resource: GREEN
common shared memory resource: GREEN
DATA_TYPE_TABLE_SET hardware resource: YELLOW
DATA_TYPE_TABLE hardware resource: YELLOW
DATA_TYPE_IDB hardware resource: YELLOW
DATA_TYPE_IDB_EXT hardware resource: YELLOW
DATA_TYPE_LEAF hardware resource: YELLOW
DATA_TYPE_LOADINFO hardware resource: YELLOW
DATA_TYPE_PATH_LIST hardware resource: YELLOW
DATA_TYPE_NHINFO hardware resource: YELLOW
DATA_TYPE_LABEL_INFO hardware resource: YELLOW
DATA_TYPE_FRR_NHINFO hardware resource: YELLOW
DATA_TYPE_ECD hardware resource: YELLOW
DATA_TYPE_RECURSIVE_NH hardware resource: YELLOW
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: YELLOW
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: YELLOW
DATA_TYPE_ECD_TRACKER hardware resource: YELLOW
DATA_TYPE_ATTRIBUTE hardware resource: YELLOW
DATA_TYPE_LSPA hardware resource: YELLOW
DATA_TYPE_LDI_LW hardware resource: YELLOW
DATA_TYPE_LDSH_ARRAY hardware resource: YELLOW
DATA_TYPE_TE_TUN_INFO hardware resource: YELLOW
DATA_TYPE_DUMMY hardware resource: YELLOW
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: YELLOW
DATA_TYPE_PROTO_GBL hardware resource: YELLOW
DATA_TYPE_MOL hardware resource: YELLOW
DATA_TYPE_MPI hardware resource: YELLOW
DATA_TYPE_SUBS_INFO hardware resource: YELLOW
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: YELLOW
DATA_TYPE_LISP_IPENCAP hardware resource: YELLOW
DATA_TYPE_LSM_ID hardware resource: YELLOW
DATA_TYPE_INTF_LIST hardware resource: YELLOW
DATA_TYPE_TUNNEL_ENCAP_STR hardware resource: YELLOW
DATA_TYPE_LABEL_RPF hardware resource: YELLOW
DATA_TYPE_L2_SUBS_INFO hardware resource: YELLOW
DATA_TYPE_LISP_IID_MAPPING hardware resource: YELLOW
DATA_TYPE_LISP_RLOC_TBL hardware resource: YELLOW
DATA_TYPE_NHID hardware resource: YELLOW
DATA_TYPE_LOOKUP hardware resource: YELLOW
DATA_TYPE_PREFIX_FILTER hardware resource: YELLOW
DATA_TYPE_PREFIX_FILTER_TBL hardware resource: YELLOW
DATA_TYPE_VXLAN_RTEP hardware resource: YELLOW
DATA_TYPE_LLC_TBL hardware resource: YELLOW
DATA_TYPE_LLC hardware resource: YELLOW
DATA_TYPE_TI_PL_TBL hardware resource: YELLOW
DATA_TYPE_RETRY_TBL hardware resource: YELLOW
DATA_TYPE_RETRY hardware resource: YELLOW
DATA_TYPE_OBJECT_QUEUE_HEAD hardware resource: YELLOW
DATA_TYPE_OBJECT_MARKER hardware resource: YELLOW
DATA_TYPE_PL_TRKR_ENTRY hardware resource: YELLOW
DATA_TYPE_PL_TRKR_SHARE_NH hardware resource: YELLOW
DATA_TYPE_NH_TRKR_SHARE_NH hardware resource: YELLOW
DATA_TYPE_LEAF_TRKR_SHARE_NH hardware resource: YELLOW
DATA_TYPE_FRR_NH_TRKR_SHARE_NH hardware resource: YELLOW
DATA_TYPE_NH_REPL hardware resource: YELLOW
DATA_TYPE_LEAF_EXT hardware resource: YELLOW
DATA_TYPE_QUEUE_EXT hardware resource: YELLOW
DATA_TYPE_COFO_TBL hardware resource: YELLOW
DATA_TYPE_COFO_TBL_ENTRY hardware resource: YELLOW
DATA_TYPE_COFO_IDB_TBL hardware resource: YELLOW
DATA_TYPE_COFO_IDB_ENTRY hardware resource: YELLOW
DATA_TYPE_DELETED_OBJECT_TBL hardware resource: YELLOW
DATA_TYPE_DELETED_OBJECT hardware resource: YELLOW
DATA_TYPE_SR6_GBL hardware resource: YELLOW
DATA_TYPE_SR6A hardware resource: YELLOW
DATA_TYPE_SR6I hardware resource: YELLOW
DATA_TYPE_TEP hardware resource: YELLOW

it worth mentioning that we were running version "6.5.3" of IOS XR when this problem occurred. we even tried upgrading the IOS X R version to "7.3.1" but the problem got even more exacerbated and the output for the hardware resource cef turned from "YELLOW" to "RED" .

After researching this problem, we found a command related to our network configuration and cef problem, which resolved our initial cef resources problems and the state turned from "RED" to "Green" and the output of mentioned command showed there is no problem anymore :
"CEF will work normally"

However, this command erupted another problem that is we can't connect any device to this NCS devices or even loopback addresses under any vrf . although is supports the command but there is no reachability from one hub further. there is no problem under "Global" routing.

We would appreciate if you check the provided information and help us in this regard.

[Flavio Miranda]

Hi

 First did you guys look at the device datasheet to make sure this is the right device for your deployment? 

Second, it is necessary to receive  all this route? Where is device posicioned? Isso it a core? Is it talking to ISP directly?

Lastly and more important. Here is not cisco TAC, here is a community to share knowledge so please share the commad  you ran and fix your problem so that you can help others with similar problem. 

[alirezaroshanaiezade]

Hi Flavio 
Based on our initial search, this device was perfect to play the radio access network role. supported 16K labels and  up to 32T traffic. we absolutely need our device to support these numbers of routes since we are an ISP. 
the command that we issued that solved our initial problem was: " cef encap-sharing disable" . 

[Rich R]

1. This is wireless - your question actually has nothing to do with wireless!  Your question would be more appropriate in https://community.cisco.com/t5/xr-os-and-platforms/bd-p/5996-discussions-xr-os-and-platforms or https://community.cisco.com/t5/routing/bd-p/5991-discussions-wan-routing-switching

2. You say "appreciate if you check the provided information" but apart from a high level description you've literally not provided any detailed info apart from the command that solved your first problem and that was only after Flavio asked for it!

Suggest you post in the correct forum and provide full detail of your design, config and the exact nature of your problem to have any chance of community help.

But as Flavio said you should probably be talking to Cisco TAC about this type of problem.  It sounds like you should probably have engaged Cisco's professional services (or a 3rd party consultant with relevant expertise) to help you design this network and tested it thoroughly to make sure it worked as expected.

[alirezaroshanaiezade]

I have created this question with additional info in the related category .

https://community.cisco.com/t5/routing/ncs-cef-resources-limitation/td-p/4602991

We thought there was a misconfiguration on our part and we were looking for a workaround or a command to resolve this issue on IOS XR. TAC is not an option for us since we bought this device from a third-party retailer. nevertheless, this is a cisco product and they need to clarify this problem.