MPLS out of resources

[cynthia.raymond]

We had Cisco 6500 devices running MPLS. 

We replaced the 6500s with 9600s last fall.

We never had issues with MPLS on the 6500 platform.

We are running out of MPLS resources on the 9600s. 

The MPLS label_stack_id is hitting its max value of ~65000.

Once that occurs, we lose network connectivity for MPLS.

We restored service by rebooting the devices.

The devices were just rebooted about 3 weeks ago and the label count is already approximately ~27,000.

We have ~1000 MPLS labels on the network. 

[MHM Cisco World]

One think in my mind 
"Change label allocation mode from
per-prefix
to
per-vrf "

this can reduce the allocation for label 

[cynthia.raymond]

Thank you for your reply. We can do that. However, I don't understand why we weren't experiencing this issue before on the old gear which was using the  per prefix allocation mode.

[MHM Cisco World]

show platform hardware fed active fwd-asic resource tcam utilization

can you share output here 

[Giuseppe Larosa]

Hello @cynthia.raymond ,

>>

The devices were just rebooted about 3 weeks ago and the label count is already approximately ~27,000.

We have ~1000 MPLS labels on the network.

if the number of allocated labels increase over time without increase in the number of prefixes in Global Routing Table and in MPLS L£ VPN services or other services this can be a software bug.

For the prefixes in the global routing table GRT the number of consumed and stored labels is greater then one per prefix because the LFIB stores also the labels learned over non best path , for example the labels learned by downstream devices

You should be able to check this using

show mpls ldp discovery

show mpls ldp binding  <prefix>

Depending on your network and  the services that are implemented you may want to implement LDP filtering for example only loopbacks are strictly needed to be associated to labels in SP environment as all services use them.

see the following document that should apply to Cat9600 too

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/217704-configure-and-verify-mpls-on-catalyst-90.html#anc20

This document explains how the Catalyst 9000 family MPLS implementation works.

The suggestion by  @MHM Cisco World  of moving to per VRF label is a wise suggestion also reported here.

Hope to help

Giuseppe