Load sharing issue

[Marks Maslovs]

Hi

Have anyone encountered such behavior?

We are importing routes from vrf SGI1 to vrf C3. Both vrf's are situated on one device (ASR-3). Traffic is going in direction from the Internet to the vrf C3, then jumps to vrf SGI1 and goes to another equipment (GGSN), which routes packets to the end user. GGSN is connected with ASR with 4 equal L3 tengigs. As we think, all traffic traversing from C3 to SGI1 to ggsn should be equally shared between all tengig links. But unfortunatelly it does not. All egress traffic is passing only TenGigE0/0/0/14.2101. Question is - why? Is this some sort of bug or we haven't taken into consideration any kind of forwarding limitations?

p.s. There is ebgp running between asr-3 and ggsn

According to cef, in my understanding, load sharing has to occur. For example:

RP/0/RSP0/CPU0:ASR-3#sh cef vrf C3 10.174.0.0/16 detail
Mon Feb 23 00:16:14.731 EET
10.174.0.0/16, version 870, internal 0x4000001 (ptr 0x714d9a24) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Feb 20 15:57:49.770
 Prefix Len 16, traffic index 0, precedence n/a, priority 3
  gateway array (0x711c6b50) reference count 22, flags 0x8020, source rib (5), 0 backups
                [1 type 3 flags 0x80151 (0x7128e754) ext 0x0 (0x0)]
  LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]

  Level 1 - Load distribution: 0
  [0] via 10.20.240.49, recursive

   via 10.20.240.49, 4 dependencies, recursive, bgp-ext [flags 0x6030]
    path-idx 0 [0x7146ef74 0x0]
    next hop VRF - 'SGI1', table - 0xe0000025
    next hop 10.20.240.49 via 10.20.240.49/32

    Load distribution: _ _ _ _ (refcount 1)

    Hash  OK  Interface                 Address
    -     Y   TenGigE0/0/0/14.2101      remote
    -     Y   TenGigE0/2/0/14.2102      remote
    -     Y   TenGigE0/0/0/20.2103      remote
    -     Y   TenGigE0/2/0/20.2104      remote

RP/0/RSP0/CPU0:ASR-3#sh cef vrf C3 exact-route  95.140.229.166 10.174.219.227
Mon Feb 23 00:17:52.564 EET
10.174.0.0/16, version 870, internal 0x4000001 (ptr 0x714d9a24) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Feb 20 15:57:49.770
 remote adjacency to TenGigE0/0/0/20.2103
 Prefix Len 16, traffic index 0, precedence n/a, priority 3
   via TenGigE0/0/0/20.2103
   via 10.20.240.49, 4 dependencies, recursive, bgp-ext [flags 0x6030]
    path-idx 0 [0x7146ef74 0x0]
    next hop VRF - 'SGI1', table - 0xe0000025
    next hop 10.20.240.49 via 10.20.240.49/32

RP/0/RSP0/CPU0:ASR-3#sh cef vrf SGI1 exact-route  95.140.229.166 10.174.219.227
Mon Feb 23 00:30:09.804 EET
10.174.0.0/16, version 17001, internal 0x4000001 (ptr 0x7144d784) [1], 0x0 (0x0), 0x400 (0x724e2ee8)
 Updated Feb 20 17:41:25.052
 remote adjacency to TenGigE0/0/0/20.2103
 Prefix Len 16, traffic index 0, precedence n/a, priority 3
   via TenGigE0/0/0/20.2103
   via 10.20.240.49, 4 dependencies, recursive, bgp-ext [flags 0x6020]
    path-idx 0 [0x7146ef74 0x0]
    next hop 10.20.240.49 via 10.20.240.49/32
     local label 16130
     next hop 10.20.237.1/32 Te0/0/0/14.2101 labels imposed {None}
 

Thank You in advance! I would appreciate any of Your help!

[xthuijs · Accepted solution]

hi marks,

based on this output I dont see the loadbalancing programmed. I see it tied to 14.2101.

It is a bit hard to follow what the precise design is you're having, so based on that there may be some options.

Could you share a topo and how the vrf's are interconnected, and where those 10G's are fitting in.

How is the IGP routing established? (static or ospf, if static, read the asr9000 ucmp doc here on support forums as it gives some extra pointers and shows on how to triage this).

cheers

xander

[Marks Maslovs]

Hi Alex!

Is it possible open a TAC case and assign in straight to You? ;)))) (joke!)

Yes, the situation that is mentioned in the ucmp doc is very close to the ours. In other words the provided solution also worked for us! :) Thank You very much!!!!

Anyway maybe someone will be interested, so I will try to explain what we faced.
I will attach picture of our topology.

1) There are 4 ebgp neighborships established between ASR3 and FNGB, (between loopbacks). Loopbacks are known via static routes.

router static
 vrf SGI1
  address-family ipv4 unicast
   10.20.240.49/32 10.20.237.17 tag 799 description FNG-B-1
   10.20.240.49/32 10.20.237.21 tag 799 description FNG-B-1
   10.20.240.49/32 10.20.237.25 tag 799 description FNG-B-1
   10.20.240.49/32 10.20.237.29 tag 799 description FNG-B-1
   10.20.240.50/32 10.20.237.17 tag 799 description FNG-B-2
   10.20.240.50/32 10.20.237.21 tag 799 description FNG-B-2
   10.20.240.50/32 10.20.237.25 tag 799 description FNG-B-2
   10.20.240.50/32 10.20.237.29 tag 799 description FNG-B-2
   10.20.240.51/32 10.20.237.17 tag 799 description FNG-B-3
   10.20.240.51/32 10.20.237.21 tag 799 description FNG-B-3
   10.20.240.51/32 10.20.237.25 tag 799 description FNG-B-3
   10.20.240.51/32 10.20.237.29 tag 799 description FNG-B-3
   10.20.240.52/32 10.20.237.17 tag 799 description FNG-B-4
   10.20.240.52/32 10.20.237.21 tag 799 description FNG-B-4
   10.20.240.52/32 10.20.237.25 tag 799 description FNG-B-4
   10.20.240.52/32 10.20.237.29 tag 799 description FNG-B-4

2) FNGB is distributing mobile subsribers prefixes to the ASR3. In other turn, ASR3 is just redistributing default route to FNGB.
3) Due to some reasons, that are not important here, it was decided to implement ABF on tengig ingress direction, to forward part of mobile subscribers straight to the C3/C4 vrf, towards CGN.

RP/0/RSP0/CPU0:ASR-3#sh run ipv4 access-list ACL_SCE_BYPASS
Fri Feb 27 20:47:34.011 EET
ipv4 access-list ACL_SCE_BYPASS
 19 remark 20-100 lines SCE bypass for FNGB users
 20 permit ipv4 host 10.172.24.34 any nexthop1 vrf C3
 30 permit ipv4 10.174.128.0/18 any nexthop1 vrf C3
 40 permit ipv4 10.182.128.0/18 any nexthop1 vrf C4
 50 permit ipv4 10.190.128.0/18 any nexthop1 vrf C3
 60 permit ipv4 10.158.128.0/18 any nexthop1 vrf C4
 70 permit ipv4 10.174.192.0/18 any nexthop1 vrf C3
 80 permit ipv4 10.182.192.0/18 any nexthop1 vrf C4
 90 permit ipv4 10.190.192.0/18 any nexthop1 vrf C3
 100 permit ipv4 10.158.192.0/18 any nexthop1 vrf C4
 900 permit ipv4 any any

4) Regarding C3 and C4. These vrf's only have one interface - Serviceapp. So in order to get any prefixes from FNGB, route-import was used from SGI1.

5) So here is the place, where the problem started. Now it turned out that , when that "special traffic", that matches ACL, comes back from the Internet from C3/C4, it goes through tengig0/0/0/14.2101, leaving other 3 interfaces empty.
All other traffic that is not matching ACL, and goes out different way to the I-net, and then of course comes different way back, is loadshared equaly between links.

6) So in the end, after reading ucmp doc, I figured out what is the root of all evil. :)

And also turned on multipath.
After all changes the cef looks like that:

RP/0/RSP0/CPU0:ASR-3#sh cef vrf C3 10.174.0.0/16 detail
Fri Feb 27 21:06:04.501 EET
10.174.0.0/16, version 1078, internal 0x4000001 (ptr 0x714d9a24) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Feb 27 19:03:01.318
 Prefix Len 16, traffic index 0, precedence n/a, priority 3
  gateway array (0x711d5e98) reference count 22, flags 0x8020, source rib (5), 0 backups
                [1 type 3 flags 0x80111 (0x712925e8) ext 0x0 (0x0)]
  LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]

  Level 1 - Load distribution: 0 1 2 3
  [0] via 10.20.240.49, recursive
  [1] via 10.20.240.50, recursive
  [2] via 10.20.240.51, recursive
  [3] via 10.20.240.52, recursive

   via 10.20.240.49, 7 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60b0]
    path-idx 0 [0x7146ef74 0x0]
    next hop VRF - 'SGI1', table - 0xe0000025
    next hop 10.20.240.49 via 10.20.240.49/32

    Load distribution: 0 1 2 3 (refcount 1)

    Hash  OK  Interface                 Address
    0     Y   TenGigE0/0/0/14.2101      remote
    1     Y   TenGigE0/0/0/20.2103      remote
    2     Y   TenGigE0/2/0/14.2102      remote
    3     Y   TenGigE0/2/0/20.2104      remote

   via 10.20.240.50, 5 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60b0]
    path-idx 1 [0x714f1424 0x0]
    next hop VRF - 'SGI1', table - 0xe0000025
    next hop 10.20.240.50 via 10.20.240.50/32

    Load distribution: 0 1 2 3 (refcount 1)

    Hash  OK  Interface                 Address
    4     Y   TenGigE0/0/0/14.2101      remote
    5     Y   TenGigE0/0/0/20.2103      remote
    6     Y   TenGigE0/2/0/14.2102      remote
    7     Y   TenGigE0/2/0/20.2104      remote

   via 10.20.240.51, 5 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60b0]
    path-idx 2 [0x7144d554 0x0]
    next hop VRF - 'SGI1', table - 0xe0000025
    next hop 10.20.240.51 via 10.20.240.51/32

    Load distribution: 0 1 2 3 (refcount 1)

    Hash  OK  Interface                 Address
    8     Y   TenGigE0/0/0/14.2101      remote
    9     Y   TenGigE0/0/0/20.2103      remote
    10    Y   TenGigE0/2/0/14.2102      remote
    11    Y   TenGigE0/2/0/20.2104      remote

   via 10.20.240.52, 5 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60b0]
    path-idx 3 [0x7144d5c4 0x0]
    next hop VRF - 'SGI1', table - 0xe0000025
    next hop 10.20.240.52 via 10.20.240.52/32

    Load distribution: 0 1 2 3 (refcount 1)

    Hash  OK  Interface                 Address
    12    Y   TenGigE0/0/0/14.2101      remote
    13    Y   TenGigE0/0/0/20.2103      remote
    14    Y   TenGigE0/2/0/14.2102      remote
    15    Y   TenGigE0/2/0/20.2104      remote

RP/0/RSP0/CPU0:ASR-3#sh cef vrf C3 exact-route  95.140.229.166 10.174.219.227
Fri Feb 27 21:07:06.321 EET
10.174.0.0/16, version 1078, internal 0x4000001 (ptr 0x714d9a24) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Feb 27 19:03:01.317
 remote adjacency to TenGigE0/2/0/14.2102
 Prefix Len 16, traffic index 0, precedence n/a, priority 3
   via TenGigE0/2/0/14.2102
   via 10.20.240.51, 5 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60b0]
    path-idx 2 [0x7144d554 0x0]
    next hop VRF - 'SGI1', table - 0xe0000025
    next hop 10.20.240.51 via 10.20.240.51/32
RP/0/RSP0/CPU0:ASR-3#sh cef vrf SGI1 exact-route  95.140.229.166 10.174.219.227
Fri Feb 27 21:07:37.228 EET
10.174.0.0/16, version 17121, internal 0x4000001 (ptr 0x7144d784) [1], 0x0 (0x0), 0x400 (0x72a2ab14)
 Updated Feb 27 19:08:59.974
 remote adjacency to TenGigE0/2/0/14.2102
 Prefix Len 16, traffic index 0, precedence n/a, priority 3
   via TenGigE0/2/0/14.2102
   via 10.20.240.51, 5 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60a0]
    path-idx 2 [0x7144d554 0x0]
    next hop 10.20.240.51 via 10.20.240.51/32
     local label 16130
     next hop 10.20.237.5/32 Te0/2/0/14.2102 labels imposed {None}

[xthuijs]

hey marks! thanks for following up! great to hear that did the trick!! very happy!

haha, ah mate, don't need a tac case, just a "open discussion" here on the forums :)

talk soon!

xander