Ask the Expert: Configuring and Troubleshooting MPLS VPN

ciscomoderator · ‎09-11-2015

Welcome to this Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and any ask questions about how to configure and troubleshoot MPLS VPN to Vinit Jain.

Ask questions from Tuesday September 15, to Friday September 25, 2015

Multiprotocol Label Switching (MPLS) virtual private network (VPN) is a cost-effective solution that provides backbone connectivity and other related services to end customers without compromising customer privacy. Because MPLS provides protocol-independent forwarding, MPLS VPN can be implemented to utilize the existing MPLS infrastructure to provide the service. Ever since RFC4364, many service providers now offer VPN services to their customers, using a technique in which customer edge routers are routing peers of provider edge routers. The Multiprotocol Border Gateway Protocol (MP-BGP) is used to distribute customers’ routes across the providers’ IP backbone network, and MPLS is used to tunnel customer packets across the providers’ backbone.

Vinit will be helping you with all your queries on all of the above.

Vinit Jain presented at Cisco Live in June 2015 on Troubleshooting BGP
Click here for More Information

Vinit Jain is a technical lead with the High-Touch Technical Support (HTTS) team supporting customers in areas of routing, MPLS, TE, IPv6, and multicast. He also supports a wide variety of platform issues such as high CPU; memory leaks; Cisco IOS, IOS XE, and IOS XR Software; and NxOS code base. He has delivered training within Cisco on various technologies as well as platform troubleshooting topics. He has also written a workbook about Cisco IOS XR Software fundamentals on the Cisco Support Community. Vinit holds CCIE certification (no. 22854) in R&S, Service Provider, Data Center and Security, as well as multiple certifications on programming and databases.

Find other https://supportforums.cisco.com/expert-corner/events.

**Ratings Encourage Participation! **
Please be sure to rate the Answers to Questions

Dieogeninholland · ‎09-17-2015

Hello Vinit,

Thank you first for this opportunity.

I need your advise/help how to proceed with the following issue:

Since 2 days, I'm seeing the following log entry in my CISCO3945 with IOS c3900e-universalk9-mz.SPA.153-3.M1

%LSD-4-LABEL_RESOURCE: label range 16-100000 exhausted
%LSD-4-LABEL_RESOURCE: label range 16-100000 exhausted

Can I go on and increase the MPLS label range?

What number do you suggest?

Is there any downtime expected? if yes, i will go on and upgrade the IOS too.

Thank you in advance.

Said

Vinit Jain · ‎09-17-2015

Hello Said

Thanks for posting your question. before we move on the suggestion on increasing the labels, I would like to know is it expected in your environment to have so many prefixes in the routing table and to have so many labels generated?
Could you please share the below output:

- show ip route summary

- show mpls memory

- show mpls label range

i think it would be a better idea to limit the MPLS labels using a ACL.

Though, if you want to increase the labels, then I dont think you can increase to any value above 1048575 which is the maximum value.

Upgrading the IOS will not help unless its a software defect.

Please share the above so that we can decide on the best action for the router.

Regards

Vinit

Thanks
--Vinit

Dieogeninholland · ‎09-17-2015

Hi Vinit,

Thank you for the feedback.

Yes we do expect all these routes.

The current situation is that we are loosing randomly internet routes, which are affecting customers in a bad way.

I was thinking about increase the mpls label range to twice the current which is 100000.

Please find below the requested infos:

Route Source    Networks    Subnets     Replicates Overhead    Memory (bytes)
connected       1           147         0           8880        26640
static          1           74          0           4500        13500
application     0           0           0           0           0
nhrp            0           0           0           0           0
isis            0           105         0           6300        18900
Level 1: 0 Level 2: 105 Inter-area: 0
eigrp 1         0           0           0           0           0
bgp 65010       171806      376974      0           32926800    98780400
External: 281968 Internal: 266804 Local: 8
internal        6253                                            22230500
Total           178061      377300      0           32946480    121069940
###

Allocator-Name                  In-use/Allocated            Count
----------------------------------------------------------------------------
LDP: Cap Block            :        560/3220       ( 17%) [     20] Chunk
LDP: Cap CCB Block        :         48/2284       ( 2%) [      2] Chunk
LDP: Cap LCB Block        :         28/3220       ( 0%) [      1] Chunk
LDP: LDP LSR addrinf      :       5940/7368       ( 80%) [    165] Chunk
LDP: LDP Message TLV Indi :          0/1912       ( 0%) [      0] Chunk
LDP: LDP Session DB Entry :        220/1416       ( 15%) [      5] Chunk
LDP: LDP TCB info         :        200/1416       ( 14%) [      5] Chunk
LDP: LDP peer addrinf     :       6600/10140      ( 65%) [    165] Chunk
LDP: LDP swidb            :        660/920        ( 71%) [      5]
LDP: TAGCON peer          :       1940/2200       ( 88%) [      5]
LDP: TDP adjacency        :        972/1232       ( 78%) [      5]
LDP: TDP context block    :        304/356        ( 85%) [      1]
LDP: TDP interface        :        420/680        ( 61%) [      5]
LDP: TDP large PIE chunk :          0/168        ( 0%) [      0] Chunk
LDP: TDP max PIE chunk    :          0/168        ( 0%) [      0] Chunk
LDP: TDP ptcl adj         :       1144/1404       ( 81%) [      5]
LDP: TDP small PIE chunk :       6576/65756      ( 10%) [     12] Chunk
LDP: TIB RB binding array :   59033380/86429216   ( 68%) [ 526843]
LDP: TIB RB binding info :   12683936/40083776   ( 31%) [ 526920]
LDP: TIB adv bmap         :   12683288/40083128   ( 31%) [ 526920]
LDP: TIB entry            : 107491680/114999544 ( 93%) [ 526920] Chunk
LDP: Tagcon peer uid tabl :        104/156        ( 66%) [      1]
LDP: local addr table     :       1248/1300       ( 96%) [      1]
LFD: AToM pwid            :          0/168        ( 0%) [      0] Chunk
LFD: FIB feature          :    3201856/3650212    ( 87%) [ 100058] Chunk
LFD: LTE                  :    6399168/6867224    ( 93%) [ 99987] Chunk
LFD: LT_WALK              :          0/168        ( 0%) [      0] Chunk
LFD: Label Block          :          0/168        ( 0%) [      0] Chunk
LFD: non-IP info          :        820/67520      ( 1%) [      5] Chunk
LFD: oce_set chunk        :          0/168        ( 0%) [      0] Chunk
LFD: punt to process data :          0/19604      ( 0%) [      0] Chunk
LSD: intf                 :       1300/66932      ( 1%) [      5] Chunk
LSD: label block          :          0/168        ( 0%) [      0] Chunk
LSD: label block          :    5199480/5645940    ( 92%) [ 99990] Chunk
LSD: label tbl            :        212/5152       ( 4%) [      1] Chunk
MFI: Clnt CMsg            :          0/65756      ( 0%) [      0] Chunk
MFI: Clnt SMsg            :      90336/131344     ( 68%) [      4] Chunk
MFI: Frr intf Q           :          0/1248       ( 0%) [      0] Chunk
MFI: InfoReq              :          0/1416       ( 0%) [      0] Chunk
MFI: InfoRply             :          0/65756      ( 0%) [      0] Chunk
MFI: OutInfoList          :          0/10340      ( 0%) [      0] Chunk
MFI: vrf table info       :          0/1180       ( 0%) [      0] Chunk

Total allocated: 284.482 Mb, 291310 Kb, 298301444 bytes

##

Downstream Generic label region: Min/Max label: 16/100000

Thank you again

Vinit Jain · ‎09-17-2015

Are you running MPLS VPN on services on this router.

i dont think increasing the number to 200000 should have any impact.

Thanks
--Vinit

Dieogeninholland · ‎09-17-2015

Yes I do.

Why increasing the label range to 200000 won't work any impact?

The main issue here is limited mpls labels.

Vinit Jain · ‎09-17-2015

Presently you have the label range of 100000 and you want to increase it to 200000. The max value supported on the router is over 1M. The only challenge you will have is the memory utilization but again, i dont think that should be something severe. The memory utilization will increase from present value of 285 megs to around 550 megs. So you need to check if you have sufficient memory on the router.

- show mem stat

The above command can help us understand that.

Thanks
--Vinit

Vinit Jain · ‎09-17-2015

I did some testing on the virtual environment:

MPLS_VPN_R8(config)#mpls label range ?
  <16-1048575>  Minimum label value for dynamic label range

MPLS_VPN_R8(config)#mpls label range 16 200000
MPLS_VPN_R8(config)#end
MPLS_VPN_R8#sh m
*Sep 10 19:27:04.021: %SYS-5-CONFIG_I: Configured from console by console
MPLS_VPN_R8#sh mpls for
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
17         Pop Label  7.7.7.7/32       0             Et0/0      10.1.78.7   
MPLS_VPN_R8#debug mpls la
MPLS_VPN_R8#debug mpls la?
% Unrecognized command
MPLS_VPN_R8#sh mpls lab  
MPLS_VPN_R8#sh mpls label ra
MPLS_VPN_R8#sh mpls label range 
Downstream Generic label region: Min/Max label: 16/200000

MPLS_VPN_R8#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
MPLS_VPN_R8(config)#mpls lab
MPLS_VPN_R8(config)#mpls label ran
MPLS_VPN_R8(config)#mpls label range 200 200000
MPLS_VPN_R8(config)#end
MPLS_VPN_R8#sh mpls lab
MPLS_VPN_R8#sh mpls label 
*Sep 10 20:02:39.817: %SYS-5-CONFIG_I: Configured from console by console
MPLS_VPN_R8#sh mpls label ran
MPLS_VPN_R8#sh mpls label range 
Downstream Generic label region: Min/Max label: 17/200000
[Configured range for next reload: Min/Max label: 200/200000]

MPLS_VPN_R8#

So, based on the above if you are just increasing the range, then that should take affect immediately but if you are changing the min label value then i believe reload is required and you might want to schedule a maintenance window for that.

http://www.cisco.com/c/en/us/td/docs/ios/mpls/configuration/guide/15_0s/mp_15_0s_book/mp_static_labels.html

Hope this helps

Vinit

Thanks
--Vinit

Dieogeninholland · ‎09-17-2015

Thank you Vinit.

I will go on and increase the labels range to 1 milion and upgrade the IOS to the Cisco suggested release.

Dieogeninholland · ‎09-17-2015

I'm afraid i didn't copy them. let me check the syslog server tomorrow, hopefully it s saved there.

Basically the router had not enough memory to allocate to the extra labels nor to save the running config.

Vinit Jain · ‎09-17-2015

I dont think the memory should have been a concern but will have to look at what log message was generated.

Thanks
--Vinit

Dieogeninholland · ‎09-18-2015

Upgrading the IOS did resolve the issue, the router doesn't really need 1000000 labels.

For some reason the router started assigning labels to all Internet routes.

I checked in our syslog and the only entry I could find that may have triggered the label issue I reported in the first place is the following:

888228: Sep 15 14:18:56.651 UTC: %LDP-5-GR: GR session 91.103.22.14:0 (inst. 6): completed graceful recovery

Ip address 91.103.22.14 belong to our bgp peer in Honk Kong.

What does this mean? would it be the reason behind my problem?

I really appreciate your support Vinit. thank you again.

Vinit Jain · ‎09-18-2015

That is weird. If I am not mistaken, the Internet routes should be BGP prefixes. LDP assigns labels for bgp prefixes. Having said that, even the router reload would have resolved the problem that you were facing.

Coming to the point of GR Recovery log, I am wondering if you received a log for GR recovery being initiated as well. from what i understand, when the router re-establishes the LDP neighborship with remote peer, it keeps the stale entry while the session is being estb. so that mpls forwarding can keep happening. When the session is estbd. again, both routers readvertise their bindings again.

Wondering if the stale entry was removed or not that could possibly lead to this problem. But again, its just a thought not confirmation. You mentioned you received a memory error. Was it some kind of MALLOC error or some other error related to memory. Please let me know if you were able to find that error message.

I would again request you to check with network configuration and limit the advertisement of labels. If there are prefixes which doesnt require label advertisement, then avoid them in your network.

Hope this helps.

Regards

Vinit

Thanks
--Vinit

Ahmad-Maan · ‎09-18-2015

Hi,

i have upgrade the IOS of ASA5515 and restore configuration, now everything is working but i am not able to connecting VPN and my RADIUS server is not testing the user id however RADIUS server is pinging.

Vinit Jain · ‎09-18-2015

Hello Amhad

I believe you have posted the question related to ipsec VPN but this event is for MPLS. I would request if you can post the question in the security group.

Regards

Vinit

Thanks
--Vinit