ibgp peer stats recieving routes, then drops most...

[ferrari01]

Hey Guys, I have an interesting issue.

Setup:  Two 7000 series routers.

(Fictious ips used for example)

Router one: 5.5.5.1

                              ebgp Peers with Level3 (aprox 474,000 routes)

                              ebgp Peers with 5.5.5.2

Router two: 5.5.5.2

                              ebgp peers with Earthlink (aprox 474,000 routes)

                              ibgp peers with 5.5.5.1

Problem:

               The problem is with the ibgp session.  Router one will start to recieve routes from router two, I start the session...it starts getting routes in, but but when it hits a certian amount, usually around 200,000, it drops the routes down to around 46,000 routes...and stays there.  I cannot for the life of me understand why this is happening.  I should add the session isnt dropping, just the routes.

Router two sees 474K routes from both level3 and 5.5.5.1.  No problem.

Both routers had 512M of memory.  So i figured router1 may be running out, so I upgraded it to a NPE-G1 with 1GB memory.  Same problem is happening.  And since router2 only has 512MB and is getting full routes fine, I dont think its a memory issue.

Now, if somthing goes wrong with level3, then the routes from 5.5.5.2 start increasing until they reach full routes.  Its like im hitting some sort of max limits of routes....which is why i thought memory issue.

I really apreciate help with this as its driving me crazy!

--John

[JohnTylerPearce]

John,

Is 'maximum-prefixes' configured anywhere? I'm assume you probably checked that, just wanted to be sure. Also, has this always been going on, or did it happen all of a sudden.

What is the model number and IOS version of Router1?

And you have eBGP to the iBGP peer above, I'm assuming that was just a typo, and it's iBGP to iBGP from R1 to R2, then each has an eBGP to the provider?

[Harold Ritter]

Hi John,

Router 2 will not advertise a given prefix to router 1 if it considers router 1 to be the best path for this prefix. It sounds like router 2 considers router 1 to be the best path for most of the prefixes, hence it does not advertise these prefixes to router1.

Regards

[JohnTylerPearce]

Harold,

So, and iBGP neighbor will only send the route that it knows as best, to it's peers, unless that peer happens to be the best path according to the neighbor from whcih it was learned, which would make sense?

Do I have this correct, and does this go the same for eBGP neighbors?

[Harold Ritter]

Hi John,

A router does not send an update for a given prefix to a given neighbor if it considers that same neighbor as the best path for that prefix. This applies regardless it is iBGP or eBGP.

Regards

[ferrari01]

I understand BGP will not send an update to a peer if it considers that neighbor a better path, but in that case it causes the problem that when level3 goes down, its missing a lot of routes that were not sent.  Somthing isnt right here.

I did a debug ip bgp on router2, and notice that it sends the routes to router1, but once it hits around 400,000 or so it starts sending deletes.  Its like at part way through the session it gets somthing from router1 that causes it to say the majority of routes are unreachable, so it deletes them:

BGP(0): x.x.x.1 send UPDATE 24.112.88.0/22 -- unreachable

and that messages runs for a zillion networks and the prefixes on router1 goes down fast...till it hits around 47K.

Possibly this info from the routers will help (Before you ask, im not filering anything between them):

On router1 (level3):

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

level3             4  3356  210048     841  1552672    0    0 13:48:23      473458

router2           4 19193  364406  404145  1552693    0    0 10:02:16    43001

savvis             4  3561   26979     839  1552693    0    0 10:30:42        37005

(savvis is only sending partial routes, so that peer is correct)

Default information originate, default route-map see-default, default sent

                                 Sent       Rcvd

  Prefix activity:               ----       ----

    Prefixes Current:          447600      43001 (Consumes 2236052 bytes)

    Prefixes Total:            546893     484358

    Implicit Withdraw:          65313      26304

    Explicit Withdraw:          33980     415053

    Used as bestpath:             n/a      29387

    Used as multipath:            n/a          0

--------------------------------------------------------------------------------------------------------

On Router2 (Earthlink):

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

Earthlink        4 13407 12458091  192897 29221737    9    0 3w2d     476983

router1           4 19193  404226  364484 29221783    0    0 10:04:26   447600

It seems router1 is correctly getting full routes from both peers...

Default information originate, default route-map see-default, default sent

                                 Sent       Rcvd

  Prefix activity:               ----       ----

    Prefixes Current:           43007     447605 (Consumes 23275356 bytes)

    Prefixes Total:            484494     547095

    Implicit Withdraw:          26375      65451

    Explicit Withdraw:         415112      34039

    Used as bestpath:             n/a     433987

    Used as multipath:            n/a          0

------------------------------------------------------------------------------------------------------------------------------

Look at how many routes router2 is withdrawing......

--John

[Harold Ritter]

Hi John,

The path unreachable message (also known as withdraw message) seen from R2 is part of the convergence process. R2 initially advertises the routes it receives from Earthlink to R1 but after receiving those same routes from R1 and realizing that R1 is the best path, it sends a withdraw message to R1.

What I fail to understand is why it causes a problem in your scenario when Level3 goes down. In this case, when Level3 goes down, R1 should consider all of the paths received from Level3 as unreachable and send a withdraw message to R2, which in turn sees that R1 is not the best path anymore. R2 will then send its best paths received from Earthlink to R1.

So what is happening in your case? Is it just that the convergence time is too long?

Regards

[ferrari01]

Correct Harold, Im assuming the issue is that when Level3 goes down, the other routes arent there, so we are partially down until they come back from R2...

Im not understanding why the routes would be withdrawn....many times I look at a sho ip bgp route and see multiple routes to a network with a * showing the best.....why wouldnt it simply leave the routes and put a * for the best one?

--John

[Harold Ritter]

Hi John,

By default, BGP only advertise its best path and as explained before, that best path is not advertised back to the peer advertising it to us.

There is a way to change the default behavior and to make a BGP peer advertise its best path and its alternate path as well. This feature is called "BGP additional paths". It is not necessarily recommended in all scenarios though.

In your case, you should probably look at why the network takes too long to converge. Is it possible that the BGP routers are too small for the task they are being asked to do?

Regards

[Vishesh Verma]

Have you configured BGP conditional advertisements? Can you post the configuration of the BGP, from both routers? Along with any route-map etc. applied?

-Vishesh

[ferrari01]

Vishesh, thanks....here you go (all ips/AS ficticious) and i deleted the other neighbors for easier reading:

router1:

router bgp 19193

no synchronization

no bgp enforce-first-as

bgp log-neighbor-changes

network x.x.x.x mask 255.255.240.0

network x.x.x.x mask 255.255.252.0

network x.x.x.x mask 255.255.248.0

neighbor internal peer-group

neighbor internal remote-as 19193

neighbor internal filter-list 2 out

neighbor 192.168.72.2 remote-as 19193

neighbor 192.168.72.2 update-source GigabitEthernet0/0

neighbor 192.168.72.2 next-hop-self

neighbor 192.168.72.2 default-originate route-map see-default

no auto-summary

route-map see-default permit 10

match ip address 10

access-list 10 permit 0.0.0.0

--------------------------------------------------------------------------------------------------------------

router2

router bgp 19193

no synchronization

bgp router-id x.x.x.x

bgp log-neighbor-changes

network x.x.x.x mask 255.255.240.0

network x.x.x.x mask 255.255.252.0

network x.x.x.x mask 255.255.248.0

neighbor 192.168.72.1 remote-as 19193

neighbor 192.168.72.1 description ibgp-border1-non-loopback

neighbor 192.168.72.1 update-source FastEthernet0/0

neighbor 192.168.72.1 next-hop-self

neighbor 192.168.72.1 default-originate route-map see-default

no auto-summary

route-map see-default permit 10

match ip address 10

access-list 10 permit 0.0.0.0

[Vishesh Verma]

Hi John,

I want to confirm just in case you have any non-defaults, what is the Primary path for R1 for all 470K routes? And same question for R2?

I need following information from both routers -

• show version
• show ip bgp neighbor from R1 and R2 (for their ibgp neighborship)

I need some prefixes which are learnt from EarthLink on R2 and are sent to R1. Also, some prefixes which are learnt from EarthLink on R2 and are not sent(withdrawn) to R1. (Outputs required from both routers)

• show ip bgp

Use following command to get the neighbors info, if you see advertised to update-groups in prefix information.

• show ip bgp update-group

Also the outputs of debugs that you ran on both the routers.

-Vishesh

[ferrari01]

vishesh:  Emailed to you for privacy..but please post solution here for the benifit of the community with ip's and as# 's removed...

--John

[Vishesh Verma]

Hi John,

I have hidden your AS and IP's

           Level3                              EarthLink

              +                                    +

              |                                    |

              |                                    |

              |                                    |

              |eBGP                                |eBGP

              |                                    |

              |                                    |

              |                                    |

        +-----+------+                      +------+-----+

        |            |                      |            |

        |            |                      |            |

        |            |                      |            |

        |   Router1  +----------------------+   Router2  |

        |  (Uplink1) |        iBGP          |  (Border2) |

        |            |                      |            |

        |            |                      |            |

        +------------+                      +------------+

From Uplink1:

uplink1-7204#sho ip bgp 1.22.189.0/24

BGP routing table entry for 1.22.189.0/24, version 763275

Paths: (1 available, best #1, table Default-IP-Routing-Table)

  Advertised to update-groups:

     2        

  3356 6453 4755 45528

    Level3 from Level3 (Level3 Router-ID)

      Origin IGP, metric 0, localpref 100, valid, external, best

Uplink1#sho ip bgp sum

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

Level3          4  3356  323836    2761  1957830    0    0 1d21h      473501

Border2         4 OurAS  440754  483676  1957857    0    0 1d18h       42695

uplink1-7204#show ip bgp update-group 2

  Has 1 member (* indicates the members currently being sent updates):

   Border2 

• Uplink learns the prefix from Level3 and selects it as valid-best path and advertise to Border2

From Border2:

border2#sho ip bgp sum

Neighbor        V    AS MsgRcvd  MsgSent   TblVer   InQ OutQ Up/Down  State/PfxRcd

EarthLink       4 13407 12563568  194815  29579893    0    0  3w3d       477035

Uplink1         4 OurAS 483676    440756  29579911    0    0  1d18h      447900

border2#sho ip bgp 1.22.189.0/24

BGP routing table entry for 1.22.189.0/24, version 29063430

Paths: (2 available, best #1, table Default-IP-Routing-Table)

  Not advertised to any peer

  3356 6453 4755 45528

    Uplink1 from Uplink1 (Uplink1 Router-ID)

      Origin IGP, metric 0, localpref 100, valid, internal, best

  13407 3356 3356 6453 4755 45528

    EarthLink from EarthLink (EarthLink ROuter-ID)

      Origin IGP, localpref 100, valid, external

• Border2 learns about the prefix from Uplink1 and EarthLink, however it chooses the Uplink1 as best path because of smaller AS-path.
• BGP advertises only valid-best-path from the BGP table to its neighbor, but this best path is chosen from Uplink1
• Just like split-horizon rule in EIGRP, BGP do not advertise any prefix to a neighbor, for which the best path is already that neighbor
• And for the same reason it is not adveritsing this prefix or I should say it has withdrawn it from Uplink1 after selecting Uplink1 as bestpath.

We can say that BGP is working at its default behavior here, however, from this post I believe you want following -

• Level3 to be the primary path for Router1(Uplink1) and Earthlink would be primary path for Router2(Border2).
• Router1 and Router2 should learn all prefixes from each other via iBGP and in case of failure of primary path they should route traffic from each other with the least convergence time.

To achieve this we need to leverage weight to choose the isp's as preffered path and only after that can router1 and router2 would advertise all prefixes to each other.

neighbor { ip-address | peer-group-name }  weight number

Border2-

!

router bgp OurAS

neighbor EarthLink weight 1000

!

Upink1-

!

router bgp OurAS

neighbor Level3 weight 1000

!

Don't forget to soft clear inboud after making the change.

clear ip bgp EarthLink/Level3 soft in

Refer to following document for BGP best path selection algorithm -

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml

-Vishesh

[milan.kulik]

Hi Vishesh,

some comments:

1) " Just like split-horizon rule in EIGRP, BGP do not advertise any prefix to a neighbor, for which the best path is already that neighbor."

This is not correct!

The "split-horizon" feature is applied only on iBGP neighbors this way:

"When a BGP speaker receives an UPDATE message from an internal peer,

   the receiving BGP speaker SHALL NOT re-distribute the routing

   information contained in that UPDATE message to other internal peers

   (unless the speaker acts as a BGP Route Reflector [RFC 2796])."

See https://supportforums.cisco.com/thread/2159140

2) If you compare the AS_PATHs on

uplink1-7204#sho ip bgp 1.22.189.0/24

BGP routing table entry for 1.22.189.0/24, version 763275

Paths: (1 available, best #1, table Default-IP-Routing-Table)

  Advertised to update-groups:

     2        

  3356 6453 4755 45528

    Level3 from Level3 (Level3 Router-ID)

and

border2#sho ip bgp 1.22.189.0/24

BGP routing table entry for 1.22.189.0/24, version 29063430

Paths: (2 available, best #1, table Default-IP-Routing-Table)

  Not advertised to any peer

  3356 6453 4755 45528

    Uplink1 from Uplink1 (Uplink1 Router-ID)

      Origin IGP, metric 0, localpref 100, valid, internal, best

  13407 3356 3356 6453 4755 45528

    EarthLink from EarthLink (EarthLink ROuter-ID)

you can see the EarthLink router received the prefix from the Level3's AS.

So the EarthLink and  Level3 providers are peering somewhere in the Internet and the best path to the prefix destination is through  Level3 even while forwarded to EarthLink originally.

Now the question is:  Should each of the routers prefer his eBGP neighbor for all prefixes received?

If yes, wouldn't it be easier to ask the providers to send just the default route instead of full BGP tables?

I think John needs to answer this question together with his LAN topology - which router is used as the default GW by the LAN devices?

But generally, the router behaviour described in the original question is correct.

Bets regards,

Milan