MPLS label_stack_id count keeps incrementing on 9606

[cynthia.raymond]

We replaced our Cisco 6500 devices with Cisco 9606 devices last fall. Since that time, we have seen increasing MPLS label_stack_id counts to the point where it reached the maximum value of ~65000 and it was impacting service to users. 

The message received in the log file is "FED_L3_ERRMSG-3-mpls_out_of_resource" followed by and MPLS_LABEL_PAUSE & then an MPLS_LABEl_RESUME. This pattern repeats until the device is reloaded and the counter is reset. Has anyone else seen this issue?

[marce1000]

 - Ref : https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/217703-understand-hardware-resources-on-catalys.html#anc37

 >...

%FED_L3_ERRMSG-3-mpls_out_of_resource: Switch 1 R0/0: fed: Out of resource for MPLS LABEL ENTRY. Failed to program local label:8205 (8192/8192) in hardware

Local label Allocation: Hardware reserved for MPLS local labels has run out of space (EM or TCAM)

Take one of these actions to reduce number of lables used on local PE: 1. Summarize prefixes at local CE or local PE 2. Change label allocation mode from per-prefix to per-vrf on the local PE