BGP keeps flapping due to high bandwidth utilization.

[NetworkingGeek1]

Hello experts,

There is a router with interface with 10 Mbps bandwidth. It's MPLS link. Because of high bandwidth utilization of that interface on transmit direction, BGP is flapping frequently. Can I apply some QoS on interface to prioritize BGP control plane traffic? And I thought that Cisco has some default control plane packets bandwidth reservation, which should prioritize control plane traffic on the interface, even if QoS is not configured?

Thank you in advance.

[Joseph W. Doherty]

So, the BGP flapping is between your (one) CE and provider's PE?

There's a 10 Mbps cap on the bandwidth between your CE and PE, correct?  Does the cap apply for both CE=>PE and the PE=>CE?

Is the link saturated CE=>PE or PE=>CE or both directions?

Assuming there's saturation CE=>PE, try on your subinterface:

policy-map ParentShaper
class class-default
shape average 8500000 !if I didn't mess up my zeros, should be for 8.5 Mbps - I'm allowing 15% for L2 overhead - unsure all Cisco shapers count it too
service-policy ChildQoS

policy-map ChildQoS
class class-default
fair-queue

interface ? !i.e. subinterface
service-policy output ParentShaper

[NetworkingGeek1]

@Joseph W. Doherty  thank you.

"So, the BGP flapping is between your (one) CE and provider's PE?"   -  Yes

"There's a 10 Mbps cap on the bandwidth between your CE and PE, correct?" - Yes

 "Does the cap apply for both CE=>PE and the PE=>CE?" - Can you please clarify this? You mean if it's this 10 Mbps limitation for both up and down speed? I think, yes, it's 10 Mbps for up speed and 10 Mbps for down speed. But we don't apply any cap ourselves. This speed is provided to us by SP.

"Is the link saturated CE=>PE or PE=>CE or both directions?" - It's saturated only on CE=>PE direction, i.e. transmit direction from our side. Monitoring shows that during some hours it can go up to 100% utilization of the link.

Do I need to apply something on physical interface or only on subinterface?

[Joseph W. Doherty]

"Can you please clarify this? You mean if it's this 10 Mbps limitation for both up and down speed? I think, yes, it's 10 Mbps for up speed and 10 Mbps for down speed."

Correct, I was asking if there were up and down limits.

"Do I need to apply something on physical interface or only on subinterface?"

In your case, placing on either should do the trick.  As to which is the better choice, doing so on the main interface would be best if the 10 Mbps was an overall bandwidth limitation for the link.  Doing so on the subinterface would be best if the 10 Mbps was a to a specific destination bandwidth limit.  In your case, I believe both apply, so whichever you prefer, although in hindsight I now lean toward using the main interface, where before I was thinking subinterface.

[NetworkingGeek1]

I want to change physical speed of the interface from 100 Mbps to 10 Mbps. Because it will actually reflect real bandwidth which is provided by SP. In this case, your suggested QoS configuration will be the same? Or it should be changed?

"policy-map ParentShaper
class class-default
shape average 8500000 !if I didn't mess up my zeros, should be for 8.5 Mbps - I'm allowing 15% for L2 overhead - unsure all Cisco shapers count it too
service-policy ChildQoS

policy-map ChildQoS
class class-default
fair-queue

interface ? !i.e. subinterface
service-policy output ParentShaper"

[Joseph W. Doherty]

If you run the interface at 10 Mbps, you don't need a two level policy and shaper, i.e.:

policy-map QoS
class class-default
fair-queue

interface ? 
service-policy output QoS

BTW, before setting your interface to 10 Mbps, check whether provide does auto speed or whether it's hard coded.  Also, for future reference, find out whether bandwidth caps are, on the same link, implemented for both up and down, and if so, whether they can be asymmetrical.

[NetworkingGeek1]

Thank you for the reply, But what this configuration will do in this case? Sorry for the basic questions, I'm not good at QoS. I don't see any shaping here, what is the impact of this configuration?

policy-map QoS
class class-default
fair-queue

interface ? 
service-policy output QoS

Also, you mentioned that changing speed to 10 Mbps, may have some negative impact, what can it cause? "Well, by moving your 10 Mbps to your interface, it may, by default, do a better job of buffering the congestion than a downstream bottleneck of which you have no direct visibility into or control of.  Of course, there's also the chance of it making no change or actually making the situation worst.  I.e. possibly worth trying to see the result."

[Joseph W. Doherty · Accepted solution]

"But what this configuration will do in this case?"

Every flow, including BGP, will get an equal share of your 10 Mbps.  I.e. bandwidth hogs won't preclude/delay other flows from obtaining bandwidth.  (With FQ, bandwidth hogs tend to get their traffic packet's dropped, while other traffic is not often dropped.)

"I don't see any shaping here . . "

Correct, if you run your interface at 10 Mbps, and there's not logical sub cap, the interface is your "shaper".

"Also, you mentioned that changing speed to 10 Mbps, may have some negative impact, what can it cause?"

Depends on how your traffic, egress bandwidth and interface buffer resources work together.  Without knowing all three very hard to predict result, often just easier to try and see result.

If FQ, alone, doesn't appear to "protect" BGP, then placing it into its own class (as earlier suggested by another poster) would be the next thing to try.  Reason for trying FQ first, alone it might be enough and it tend to "improve" other network traffic too.

[NetworkingGeek1]

@Joseph W. Doherty  Thanks. I'll check which option which is more easy to implement. I'd like to go with changing physical speed, because it's more accurate. But if it needs involving of ISP to change it (I assume it's auto negotiation on both sides), then I'll probably implement shaper. Also I'll find out what traffic is congesting the link and will ask technician who manages that server to limit that traffic.

"shape average 8500000 ! if I didn't mess up my zeros, should be for 8.5 Mbps - I'm allowing 15% for L2 overhead - unsure all Cisco shapers count it too" - By L2 overhead here you mean Dot1q vlan header?

Also I noticed that during congestion, access points are rebooting. I think it's because they can't reach their wireless controller via CapWap tunnel, because it's reachable via congested link. But I'm not sure if this behavior is correct, what do you think?

[Joseph W. Doherty]

"Also I'll find out what traffic is congesting the link and will ask technician who manages that server to limit that traffic."

With FQ, good chance you don't need to ask for that.

"By L2 overhead here you mean Dot1q vlan header?"

Yes, that and all the other L2 overhead (that supports an Ethernet frame).

Also, again, unsure whether your shaper would also include L2 overhead while "counting" transmission rate.  Later/current IOS versions, I believe are more likely to do so.  What you can try, if 8.5 Mbps seems to cure the problem, increase rates, say in increments of 500 Kbps (not exceeding 10 Mbps), and see if problem reappears, if not, keep the higher rate.

Conversely, if L2 overhead isn't being including in shaping rate, sometimes you need to go even lower than 85%.  (Basically, L2 overhead is fixed per frame, so its percentage of overhead varies with packet size; smaller packets have a higher L2 overhead.)

"Also I noticed that during congestion, access points are rebooting. I think it's because they can't reach their wireless controller via CapWap tunnel, because it's reachable via congested link. But I'm not sure if this behavior is correct, what do you think?"

Yup, certainly possible.  Again, FQ might resolve.  FQ will see any one LWAP's tunnel traffic as a single flow, but should see different LWAP tunnels, each as their own flow.

At this point, FQ might sound like "snake oil", but as you've noted you're unfamiliar with QoS, have you every been in a long line with just one clerk vs. a long ling with multiple clerks, each taking customer from head of line as they finish with their current customer?  If so, the former is the default interface buffering, FIFO, the latter is (somewhat like) FQ.

Oh, if FQ doesn't seem to cure the problems, before getting involved with additional traffic classes, try tx-ring-limit on egress interface, say with a setting of 3.  (Tx-ring-limit is a hardware FIFO queue, attached to the interface.  When it overflows, it then places packets in the software interface queue[s] [where, as Cisco used to call it, we get to do "fancy queuing"].)

Also, BTW, in theory FQ provides a queue for every flow.  In practice, Cisco allocates a fixed number of flow queues and hashes flows across them.  I.e. it's possible a bandwidth hog and non-hog will share the same "flow" queue.

Some of the later IOSs support more configuration options, including, I recall (?) increasing number of flow queues.

[Joseph W. Doherty]

BTW, just in case you need it, a variant QoS policy, very similar to what @Georg Pauwen suggested might be:

class-map match-all BGP
match dscp CS6 !either of these match statements would likely suffice alone (DSCP matching would be more efficient)
match protocol BGP !NBAR might require a specific IOS or license

policy-map ParentShaper
class class-default
shape average 8500000 !if I didn't mess up my zeros, should be for 8.5 Mbps - I'm allowing 15% for L2 overhead - unsure all Cisco shapers count it (L2 overhead) too
service-policy ChildQoS

policy-map ChildQoS
class BGP
bandwidth percent 10 !doubt you'll need 10%, it doesn't use any of this bandwidth unless it must
class class-default
bandwidth percent 90
fair-queue

interface ?
service-policy output ParentShaper

The above differs from what Georg suggested in: 1) LLQ is best for traffic that that has strigent requirements for latency and jitter, like VoIP; for your BGP we just want to avoid drops and 2) even with BGP not in class-default, FQ, generally, will benefit other traffic, like your mention of your wireless disconnections. (Also Georg matched against, I believe, DSCP decimal value 6, but we want to match against CS6, or IPPrec 6.)

The advantage of using a separate class for BGP traffic, we have more control over managing that traffic.  Again, I would expect FQ, alone, to be sufficient, but as FQ does actually share flow queues, and not fully knowing your traffic mix, the above policy better guarantees BGP will not be adversely impacted by any other traffic (this also assumes the class bandwidth allocation is sufficient).

[NetworkingGeek1]

Hello @Joseph W. Doherty 

Actually, I forgot to mention, there is some old QoS applied to the physical interface (not sub interface). And what is interesting about it, it doesn't have any shaping. As I already mentioned, even though real bandwidth is 10 Mbps, physical speed is set to 100 Mbps. Does it mean that Router tries to send data at 100 Mbps rate, because neither shaping, nor correct speed set (10 Mbps)? I'll write some configuration facts and my assumptions, please correct me if I'm wrong:

1. Interface physical speed set to 100 Mbps.

2. Real bandwidth which we have from SP is 10 Mbps.

3. There is "bandwidth 10000000" (10 Mbps) configured under interface, but bandwidth doesn't limit actual speed, it's just for reference.

4. There is QoS configured under physical interface, but it doesn't have any shaping.

My assumptions: Since our CE router is not aware that real bandwidth is 10 Mbps, it tries to send data at 100 Mbps speed out of that interface and it might be that SP PE router is dropping the packets above 10 Mbps. - Am I right?

Also this assumption came from this fact: Recently, I just added "match protocol BGP" under class map for Voice traffic. This class map is configured under old policy-map which is configured (I already mentioned above about old QoS config). And after I added this config, for one day, BGP was stable, but the other day, it flapped again. I checked drops with command "policy-map interface {name}" and I didn't see drops for BGP. I think what might happen is, that during link congestion, SP was getting traffic at higher rate than 10 Mbps and it started to drop packets. But there one weak point in my assumptions: theoretically this class-map, where I placed BGP, should have priority queue, and it should leave interface as soon as possible, and it should arrive to the SP's router earlier than others and thus SP should not drop it.

[Joseph W. Doherty]

With the interface running at 100 Mbps, frames are always sent at 100 Mbps.  Without a shaper, QoS only "triggers" when the interface congests at 100 Mbps (which can be rare with traffic being policed downstream at 10 Mbps).

Even with a shaper, frames (again) are still sent at 100 Mbps, however some frames are held/queued by the shaper, even when the interface could transmit them.  Basically, a shaper will (when set at 10 Mbps on a 100 Mbps interface) try to average out releasing frames so your effective transmission (average) rate is 10 Mbps.  For QoS purposes, a shaper, again more-or-less, will queue packets "like" the physical interface would if running physically at shaper's bandwidth.

#2 if physical interface is 100 Mbps, your "real" bandwidth is 100 Mbps.  If SP is capping bandwidth at a lower rate (i.e. your 10 Mbps), that's often done with a policer (used by your SP).

#3 Correct.

#4 As described, without a shaper, your QoS is much, much less effective.

"My assumptions: Since our CE router is not aware that real bandwidth is 10 Mbps, it tries to send data at 100 Mbps speed out of that interface and it might be that SP PE router is dropping the packets above 10 Mbps. - Am I right?"

I believe so.

"I just added "match protocol BGP" under class map for Voice traffic."

Possibly okay for a quick test, to easily confirm whether "protecting" BGP solves BGP flapping issue (although w/o a shaper, as noted above, QoS much, much less effective), but as noted in another posting, you really don't want BGP mixed in with VoIP traffic.

"But there one weak point in my assumptions: theoretically this class-map, where I placed BGP, should have priority queue, and it should leave interface as soon as possible, and it should arrive to the SP's router earlier than others and thus SP should not drop it."

Sorry, actually your weak point is understanding QoS (which you admitted to earlier); I expect, though, you're learning.

First, even if you transmitted BGP packets first, as you suppose, but at 10x the contracted rate, a SP rate limiter usually doesn't  analyze packet contents or sequence received, just whether above contract rate, and if so, drop packets.

Second, QoS prioritizes when traffic is queued.  When there appears to be no congestion (i.e. physical interface running at 100 Mbps, but downstream policed at 10 Mbps) there's almost never any traffic queued on your interface (assuming your traffic is rate adaptive).  Also, as mentioned in one of my prior postings, there's a FIFO queued tied to the interface, only when it "overflows", into software managed queues (e.g. policy-map classes), can traffic be dequeued differently from how it was queued.

I.e. you really, really need to "control" your transmission rate to agree with contracted bandwidth limits.  Can be done when physical interface "speed" matches contracted rate limit or via a shaper.

BTW, once you match the contracted bandwidth, you likely now "see" much queuing and/or drops on your egress interface.  But, now that the queuing and/or drops are on your device, you can now "manage" the situation, via QoS.

Are you able to post you relevant QoS config statements being used?  If so, it might help provide me insight into your QoS policy, or at least your QoS policy someone, at sometime, thought you should use.

[NetworkingGeek1]

This QoS config which we have now, is quite old and needs to be changed anyway. I just wanted to do some "hot fix" adding "match protocol BGP" to it and see what is going on. Also, I'd like to add something regarding L2 header you mentioned. There is a feature called, "Ethernet Overhead Accounting". With this feature enabled, Cisco takes L2 headers (like dot1q) into consideration, while shaping traffic. This command doesn't exist on my router, perhaps because of IOS version, which not very new. Now I'm wondering, if this feature should be explicitly enabled, does it mean that default behavior of Cisco routers not count L2 headers in shaping calculation?

[Joseph W. Doherty]

Ah, I was unaware this feature had been created.

Over the years I suspected L2 overhead was not accounted for, but didn't find any documentation saying whether L2 was accounted for or not.

Later, I recall (?) some/few high end platforms started to account for L2 overhead, automatically (which seemed to further confirm my earlier suspicion).  This "Ethernet Overhead Accounting" feature seems to totally confirm my earlier suspicion, i.e. e.g. shapers do not "count" L2 overhead.

This feature (from briefly reading) appears to allow you to add a static constant for the L2 overhead per packet (which although not as "nice" as the device actually using actual frame size, would still work much, much better than just allowing for an average overhead percentage).

[NetworkingGeek1]

@Joseph W. Doherty  could please help me with another question regarding QoS. Recently, ISP asked to test the Internet circuit connected to our site (it's different site, not which we were discussing here), for a little while, for testing purposes they upgraded the link from 300 Mbps to 900 Mbps. Users checked speed via speedtest.net and it really showed 900/900 Mbps. But what bothers me, we had QoS configured on WAN Router which has shape average to 300 Mbps outbound. Why speedtest showed upload as 900 Mbps, not 300 Mbps? Is it some issue with QoS policy or it's a way how speedtest checks the speed?