Re: 6PE label allocation question

dean holroyd · ‎10-28-2010

Hello

Can someone with experience of 6PE please clarify a question for me?

The following doc says that 6PE allocates aggregate labels for BGP prefixes from a pool of 16, which are used in a round robin fashion

http://www.Cisco.com/en/US/products/sw/iosswrel/ps1835/products_data_sheet09186a008052edd3.html

3. MP-BGP assigns a local label to prefix P2. This label is an aggregate label4 . It is not specific to a given prefix, it is chosen out of a sixteen-label pool in a round robin fashion. Consequently, egress 6PE is not able to take a forwarding decision solely based on this label. The advantage of using more than one label value is that it results in better load sharing in the core of the network, since the P routers hash the bottom label value because the encapsulated packet is not IPv4.

Testing this theory in the lab yields a different answer though. the following prefixes were received from an eBGP neighbor, and it appears the 6PE device assigns one label per prefix from it's gloabl pool.

So my question is why is it not assigning just 16 labels? and what is the implication for this on an internet facing PE device?

Thanks

PE1#sh bgp ipv6 label

Network Next Hop In label/Out label

2001:3333:1::33/128

2001:5555::1 67/nolabel

2001:3333:2::3/128

2001:5555::1 66/nolabel

2001:3333:2::33/128

2001:5555::1 65/nolabel

2001:3333:3::3/128

2001:5555::1 64/nolabel

2001:3333:3::33/128

2001:5555::1 63/nolabel

2001:3333:4::3/128

2001:5555::1 62/nolabel

2001:3333:4::33/128

2001:5555::1 61/nolabel

2001:3333:5::3/128

2001:5555::1 60/nolabel

2001:3333:5::33/128

2001:5555::1 59/nolabel

2001:3333:6::3/128

2001:5555::1 58/nolabel

2001:3333:6::33/128

2001:5555::1 57/nolabel

2001:3333:7::3/128

2001:5555::1 56/nolabel

2001:3333:7::33/128

2001:5555::1 55/nolabel

2001:3333:8::3/128

2001:5555::1 54/nolabel

2001:3333:8::33/128

2001:5555::1 53/nolabel

2001:3333:9::3/128

2001:5555::1 52/nolabel

2001:3333:9::33/128

2001:5555::1 51/nolabel

2001:3333:10::3/128

2001:5555::1 50/nolabel

2001:3333:10::33/128

2001:5555::1 49/nolabel

2001:3333:11::3/128

2001:5555::1 48/nolabel

2001:3333:11::33/128

2001:5555::1 47/nolabel

2001:3333:12::3/128

2001:5555::1 46/nolabel

2001:3333:12::33/128

2001:5555::1 45/nolabel

2001:3333:13::3/128

2001:5555::1 44/nolabel

2001:3333:13::33/128

2001:5555::1 43/nolabel

2001:3333:14::3/128

2001:5555::1 42/nolabel

2001:3333:14::33/128

2001:5555::1 41/nolabel

2001:3333:15::3/128

2001:5555::1 40/nolabel

2001:3333:15::33/128

2001:5555::1 39/nolabel

2001:3333:16::3/128

2001:5555::1 38/nolabel

2001:3333:16::33/128

2001:5555::1 37/nolabel

2001:3333:17::3/128

2001:5555::1 36/nolabel

Giuseppe Larosa · ‎10-28-2010

Hello Dean,

the document you are referring to is about 12.2 mainline and refers about first 6PE implementation that we tested in year 2004-2005.

The limitation of 16 labels was visible in performance results as IPv6 performance was half of IPv4.

I'm not sure if the labels were part of the global pool or not (likely they were).

The limitation caused the need for a second lookup, it was not enough to examine the label to know how the IPv6 packet carried within had to be sent out the PE-CE link.

That limitation was removed later. And it shouldn-t be mentioned for example in 12.4T IPv6 configuration chapter of IPv6 over MPLS

see

http://www.cisco.com/en/US/docs/ios/ipv6/configuration/guide/ip6-over_mpls_ps6441_TSD_Products_Configuration_Guide_Chapter.html#wp1040663

>> So my question is why is it not assigning just 16 labels?

for better performances so that from label packet rewrite functions can be done without a second route lookup

>>and what is the implication for this on an internet facing PE device?

if you mean what happens to a router connected to public IP6 internet and receiving an IPv6 full table.

An IPv6 full table can be represented by only 8192 prefixes (in theory) and with almost a 1,000,000 label space it shouldn't be a problem to associate a label to each of them.

the so called global aggretable unicast addresses

this is by IPv6 design that is strictly multi level hierarchical

see

http://bgpmon.net/weathermap.php?inet=6

Hope to help

Giuseppe

dean holroyd · ‎10-29-2010

Thanks for that Giuseppe, that's very useful

I have been in the lab again today. I advertised 40000 prefixes from two eBGP neighbors to my 6PE device. I could see the device assign 40000 labels, and reserve 6.5Mb of memory for the MPLS labels.

However, when i shut down the neighbors at the end of the lab, the amount of memory allocated for the labels does not return to its base value (900Kb), and instead stays at around 2.5Mb. Do you think this is an issue? memory leak? I see the label count also stays at 40000. The device is a 7200 running SRD4 code.

Any ideas?

Thanks for your help

R2#sh mpls forwarding-table
Local Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label Label or VC   or Tunnel Id      Switched      interface
16     Pop Label     1.1.1.2/32        0             Fa1/1      1.0.0.2
21     No Label      2001:33::/64      0             aggregate
22     No Label      2001:22::/64      0             aggregate
23     No Label      2001:12::/64      0             aggregate
24     No Label      2001:11::/64      0             aggregate
R2#
R2#
R2#sh mpls mem
Allocator-Name                  In-use/Allocated            Count
----------------------------------------------------------------------------
LDP: LDP LSR addrinf      :         72/3052       ( 2%) [      2] Chunk
LDP: LDP Message TLV Indi :          0/2132       ( 0%) [      0] Chunk
LDP: LDP TCB info         :         40/1052       ( 3%) [      1] Chunk
LDP: LDP peer addrinf     :         80/3716       ( 2%) [      2] Chunk
LDP: MPLS Event log       :     144000/144052     ( 99%) [      1]
LDP: MPLS MIB sw subblock :        112/164        ( 68%) [      1]
LDP: TAGCON peer          :        344/396        ( 86%) [      1]
LDP: TDP adjacency        :        160/212        ( 75%) [      1]
LDP: TDP context block    :        296/348        ( 85%) [      1]
LDP: TDP interface        :         56/108        ( 51%) [      1]
LDP: TDP large PIE chunk :          0/65588      ( 0%) [      0] Chunk
LDP: TDP max PIE chunk    :          0/65588      ( 0%) [      0] Chunk
LDP: TDP ptcl adj         :        208/260        ( 80%) [      1]
LDP: TDP small PIE chunk :          0/65588      ( 0%) [      0] Chunk
LDP: TIB RB binding array :        400/556        ( 71%) [      3]
LDP: TIB RB binding info :         72/228        ( 31%) [      3]
LDP: TIB adv bmap         :         80/236        ( 33%) [      3]
LDP: TIB entry            :        624/32820      ( 1%) [      3] Chunk
LDP: Tagcon peer uid tabl :        136/188        ( 72%) [      1]
LDP: local addr table     :        136/188        ( 72%) [      1]
LDP: tagsw swidb          :        216/268        ( 80%) [      1]
LFD: AToM pwid            :          0/67220      ( 0%) [      0] Chunk
LFD: FIB feature          :        192/73924      ( 0%) [      6] Chunk
LFD: LTE                  :        504/35304      ( 1%) [      9] Chunk
LFD: LT_WALK              :          0/588        ( 0%) [      0] Chunk
LFD: non-IP info          :        432/68156      ( 0%) [      4] Chunk
LSD: intf                 :        224/33548      ( 0%) [      1] Chunk
LSD: label tbl            :    1760396/1940760    ( 90%) [ 40009] Chunk
LSD: label tbl            :         84/1764       ( 4%) [      1] Chunk
MFI: Clnt CMsg            :          0/65588      ( 0%) [      0] Chunk
MFI: Clnt SMsg            :      87616/131176     ( 66%) [      4] Chunk
MFI: Frr intf Q           :          0/1052       ( 0%) [      0] Chunk
MFI: InfoReq              :          0/808        ( 0%) [      0] Chunk
MFI: InfoRply             :          0/65588      ( 0%) [      0] Chunk
MFI: vrf table info       :          0/1304       ( 0%) [      0] Chunk

Total allocated: 2.740 Mb, 2806 Kb, 2873520 bytes
R2#
R2#sh bgp ipv6 unicast summ
BGP router identifier 4.4.4.4, local AS number 4
BGP table version is 88334, main routing table version 88334
4 network entries using 580 bytes of memory
4 path entries using 304 bytes of memory
2/1 BGP path/bestpath attribute entries using 152 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1036 total bytes of memory
BGP activity 155008/155004 prefixes, 240010/240006 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.2         4     4      82    1061    88334    0    0 01:12:10        0
2001:11::1      4     1       0       0        0    0    0 00:29:56 Active
2001:12::1      4    44       0       0        0    0    0 00:30:08 Idle

antasson · ‎11-01-2010

Hi Dean,

the labels are kept for 5 minutes, if I'm not wrong, after the neighborship goes down, and only after that they're freed, and that timer is not tunable.

You may want to try taking the outputs after 10 minutes or so.

Best regards,

Antonio

dean holroyd · ‎11-01-2010

Thanks for the reply Antonio

I left it for over an hour today, but the memory does not get freed up once the session is down.

Router#sh bgp ipv6 un summ
Load for five secs: 6%/99%; one minute: 30%; five minutes: 11%
Time source is hardware calendar, *11:55:31.471 UTC Mon Nov 1 2010
BGP router identifier 4.4.4.4, local AS number 4
BGP table version is 15004, main routing table version 15004
5003 network entries using 725435 bytes of memory
5003 path entries using 380228 bytes of memory
3/2 BGP path/bestpath attribute entries using 228 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1105915 total bytes of memory
BGP activity 10003/5000 prefixes, 10004/5001 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.2         4     4      36     172    15004    0    0 00:31:13        0
2001:11::1      4     1      43       4    15004    0    0 00:00:46     5000

Router#sh mpls mem | i lab|All|To
Allocator-Name                  In-use/Allocated            Count
LFD: AToM pwid            :          0/67220      ( 0%) [      0] Chunk
LSD: label tbl            :     220352/251580     ( 87%) [   5008] Chunk
LSD: label tbl            :         84/1764       ( 4%) [      1] Chunk
Total allocated: 1.538 Mb, 1575 Kb, 1613772 bytes

*Nov 1 11:55:47.983: %BGP-5-ADJCHANGE: neighbor 2001:11::1 Down Peer closed the session
*Nov 1 11:55:47.983: %BGP_SESSION-5-ADJCHANGE: neighbor 2001:11::1 IPv6 Unicast topology base removed from session Peer closed the session
*Nov 1 11:55:47.987: %BGP_SESSION-5-ADJCHANGE: neighbor 2001:11::1 IPv4 Unicast topology base removed from session Peer closed the session

Router#sh clock
Load for five secs: 55%/38%; one minute: 27%; five minutes: 12%
Time source is hardware calendar, *11:55:54.535 UTC Mon Nov 1 2010

*11:55:54.539 UTC Mon Nov 1 2010

Router#sh mpls mem | i lab|All|To
Allocator-Name                  In-use/Allocated            Count
LFD: AToM pwid            :          0/67220      ( 0%) [      0] Chunk
LSD: label tbl            :     220352/251580     ( 87%) [   5008] Chunk
LSD: label tbl            :         84/1764       ( 4%) [      1] Chunk
Total allocated: 1.127 Mb, 1155 Kb, 1183492 bytes

Router#sh clock
Load for five secs: 3%/0%; one minute: 4%; five minutes: 4%
Time source is hardware calendar, *13:03:18.643 UTC Mon Nov 1 2010

*13:03:18.647 UTC Mon Nov 1 2010

Last week, I graphed the memory usage (processor and mpls) for increasing prefixes on the 6PE device. The mpls mem at the start of my experiment was at 962Kb for my first run of tests, but when the session is cleared, it stayed at 2.2Mb. Once the prefixes were increased again, the mpls mem usage converged to a point that was the same as the first test. These are the dotted lines at the bottom

Giuseppe Larosa · ‎11-01-2010

Hello Dean,

I wonder if this behaviour can be caused by some BGP feature like soft-reconfiguration inbound that makes the router to store prefixes and associated labels even if the neighbors are down.

If you are in a lab environment, what happens if you remove the whole BGP process, are then the labels removed from LFIB?

of if you remove the neighbor statements from BGP?

Hope to help

Giuseppe

dean holroyd · ‎11-01-2010

Hello Giuseppe

That doesn't seem to make any difference. Soft reconfig was not configured for any neighbour. Just a very basic config

router bgp 4
no synchronization
bgp log-neighbor-changes
neighbor 1.1.1.2 remote-as 4
neighbor 1.1.1.2 update-source Loopback0
neighbor 2001:11::1 remote-as 1

address-family ipv6
no synchronization
neighbor 1.1.1.2 activate
neighbor 1.1.1.2 send-label
neighbor 2001:11::1 activate
exit-address-family

Router#sh mpls mem | i lab|All|To
Allocator-Name                  In-use/Allocated            Count
LFD: AToM pwid            :          0/67220      ( 0%) [      0] Chunk
LSD: label tbl            :     220352/251580     ( 87%) [   5008] Chunk
LSD: label tbl            :         84/1764       ( 4%) [      1] Chunk
Total allocated: 1.127 Mb, 1155 Kb, 1183492 bytes

Router(config)#no router bgp 4

*Nov 1 15:09:32.959: %BGP-5-ADJCHANGE: neighbor 1.1.1.2 Down Unknown path error
*Nov 1 15:09:32.963: %BGP_SESSION-5-ADJCHANGE: neighbor 1.1.1.2 IPv6 Unicast topology base removed from session Unknown path error

Router#sh ip bgp summ
Load for five secs: 41%/99%; one minute: 8%; five minutes: 5%
Time source is hardware calendar, *15:10:00.447 UTC Mon Nov 1 2010
% BGP not active

Router#sh mpls mem | i lab|All|To
Allocator-Name                  In-use/Allocated            Count
LFD: AToM pwid            :          0/67220      ( 0%) [      0] Chunk
LSD: label tbl            :     220352/251580     ( 87%) [   5008] Chunk
LSD: label tbl            :         84/1764       ( 4%) [      1] Chunk
Total allocated: 1.127 Mb, 1155 Kb, 1183492 bytes

antasson · ‎11-04-2010

Hi Dean,

I'm afraid there's no easy answer to this, so if that's really a concern or if you found any impact on the performances, please open a TAC case and we'll investigate it possibly involving the developers.

Best regards,

Antonio

Giuseppe Larosa · ‎11-04-2010

Hello Dean,

if even after deleting the BGP neighbors the labels and the memory are not released after a few minutes, this is clearly a software bug and I agree with Antonio: you should open a service request or you could attempt a bug search for this IOS image version with key IPv6

Hope to help

Giuseppe

dean holroyd · ‎11-05-2010

Thanks Giuseppe and Antonio

The device in question is a 7200 running 12.2(33)SRD4 code. I left it for a couple of days, but the mem was still in use. The device is not under support, however, I will try it on a spare 7600 that we have running SRD4 to see if the bug exists there. I will be able to raise a TAC case then.

Thanks for your help