Traffic exceeding the policy limit on ASR9K Bundle-Ether

[ONUR CICEK]

Hi,

I have a problem regarding an issue on ASR 9000 bundle interface.

I apply 20mbps input and output policy to the interface to limit our customers traffic but sometimes I see 40mbps of customer traffic. It doesn't happens always but it occurs on different customers on different bundle interface.

What would be the possible reason? Thanks in advance.

note: we have different burst values on other ASR9 devices  such as:

police rate 20000000 bps burst 3750000 bytes

police rate 20000000 bps burst 3200000 bytes

police rate 20000000 bps burst 3000000 bytes

police rate 20000000 bps burst 2500000 bytes

but not sure if indeed this is the problem,

Configurations:

policy-map 20mbps

class class-default

  police rate 20000000 bps burst 3200000 bytes

   conform-action transmit

   exceed-action drop

  !

interface Bundle-Ether1

mtu 9216

load-interval 30

!

interface Bundle-Ether1.100

service-policy input 20mbps

service-policy output 20mbps

ipv4 address 10.10.10.1

encapsulation dot1q 100

!

interface GigabitEthernet0/0/0/1

bundle id 1 mode active

!

interface GigabitEthernet0/0/0/2

bundle id 1 mode active

!

when i send traffic of 40mb to the customer:

ASR9#sh  int Bundle-Ether1.100  

Bundle-Ether1.100 is up, line protocol is up

......

  30 second input rate 11361000 bits/sec, 6642 packets/sec

  30 second output rate 40558000 bits/sec, 6877 packets/sec

[Rivalino Tamaela]

Hello,

Can you also check 'show policy-map interface bundle-ether 1.100 output' while sending 40mbps to customer?

thanks,

rivalino

[ONUR CICEK]

Hi Rivalino,

Here is the output, the thing is this time even if I've sent 40m, peak is 26m.

Bundle-Ether1.100

  30 second input rate 4291000 bits/sec, 886 packets/sec

  30 second output rate 24492000 bits/sec, 2574 packets/sec

     3763694312 packets input, 1626561697099 bytes, 108538 total input drops

     3365349 drops for unrecognized upper-level protocol

     Received 38 broadcast packets, 3365349 multicast packets

     4012226890 packets output, 3050913586640 bytes, 42481432 total output drops

     Output 123 broadcast packets, 0 multicast packets

ASR9#show policy-map interface Bundle-Ether1.100 output

Wed Jan 22 08:06:41.049 Turkiye

Bundle-Ether1.100 output: 20mb

Class class-default

  Classification statistics          (packets/bytes)     (rate - kbps)

    Matched             :           413918382/342268391637         42461

    Transmitted         :           N/A

    Total Dropped       :             5387815/7327884962           17684

  Policing statistics                (packets/bytes)     (rate - kbps)

    Policed(conform)    :           408530567/334940506675         24777

    Policed(exceed)     :             5387815/7327884962           17684

    Policed(violate)    :                   0/0                    0

    Policed and dropped :             5387815/7327884962        

[atif_hafeez]

Hi,

This is due to the default QOS behavior of ASR9K line cards i.e. every QOS policy applied on the bundle interface gets replicated to each member link. By default the traffic traversing the l3 bundle sub-interface gets load balanced based on the following parameters:

 IP source address

 IP destination address

 Router ID

 Layer 4 source port

 Layer 4 destination port

In most of the cases, the traffic will get load balance across each member link of the bundle and will get double the bandwidth configured in the policy.

In your case, 20mbps is set; each member link of the bundle is programmed to allow 20mbps ingress & egress. When you are sending 40mbps, it is getting load balanced across two member links i.e. each member link is traversing 20mbps as programmed. When you are sending 40mbps of traffic, please check the physical interface rates of each member link.

The only solution to this problem is to tweak the default load balancing mechanism of the bundle by using the command 'bundle load-balancing hash auto' which is only supported under l2transport sub-interfaces. This command will bind the bundle sub-interface with only one member link and hence all the traffic will be forced to traverse only one member link.

[ONUR CICEK]

Hi Atif,

thanks for explanation, but I have one question for your comment: When this load balancing issue occurs, traffic will be doubled always or sometimes?

Because in my case, this doesn't happen always. As you can see my reply to Rivalino, this time I got 26mb but last week it was 40mb. Beside that also I'm checking on another customer which is also on same ASR and same bundle, but it didn't exceed 20mb. Also check on other ASRs, same results. (also checked other policies like 5mb, 10mb etc)

You said tweak is a solution but what happens when the chosen member link gets saturated? is there any mechanism to switch to other member?

Cheers

[Rivalino Tamaela · Accepted solution]

Hello Eureka,

I guess the bundle has two members? Default behavior for QoS in bundle is that the member will inherit the policy from bundle. If you configure 20mbps policing on bundle, it will be applied to each member. So, if you have 2 member links of the bundle, the total traffic policing will be 40mbps (if the two members are active and doing load balancing). Please refer to this documentation:

http://www.cisco.com/en/US/docs/routers/asr9000/software/asr9k_r5.1/qos/configuration/guide/b_qos_cg51xasr_chapter_01000.html#ID105

Regarding the load balancing, ASR9k do loadbalance for ipv4 based on L3 and L4 information (router id too). If you have only one stream to consist same L3 and L4 info, then the traffic is to be forwarded on one link only. If you send that stream 21 mbps, then you get policed to 20 mbps. You should see total traffic in bundle is 20 mbps. Then, if you add the second stream 6 mbps, it take second link. You should see total traffic in bundle is 26mbps. This example is simplification of real world. The more granular your traffics are, the more chance both links are to be saturated. If you send random/Internet traffic, most likely you see total 40mbps on bundle.

For your question, what happens when the choses member link gets saturated? The exceeded packets are dropped. There is a way to shift hashing calculation. Read the following article about polarization:

https://supportforums.cisco.com/docs/DOC-26687

HTH,

Rivalino