OSPF, iBGP, EBGP across datacenters

[cjsattler]

I have two datacenters with two leased fibers running between them.  Currently one datacenter is just a slave off the main site with layer 3 switches connected by OSPF to routed interfaces.  The fibers themselves hook up to stacked 3750gs one each side with OSPF running between them.  The OSPF cost tells it what fiber to use as primarily.  The default-route is currently learned by OSPF from the core router at site A

Site A has 2 providers with full tables with ebgp and site B will have two providers with full routes and ebgp.  I would like to share internet routes by ibgp between both routers and i would like to also keep the fibers plugged into the stacked 3750s for redundnacy and not have the fibers plugged into the backbone routers.  

The problem i am having is when a packet comes in from the internet destined for a provider on site B, router A sends it to the switches at side A and it gets in a routing loop since the destination isnt on the layer 3 switch (only ospf with internal network routes) and sends it back to router A. 

I know i can just plug the fibers into the backbone routers to fix this but i really want it on the stacked 3750s since the likelihood i have to take down a single router is greater then both switches that have a port-channel to the access layer.

I was thinking of also doing a l2tpv3 pseudowire between both core routers to allow them to have layer two between them but i'd rather not have the added overhead and complexity.

Here are the questions i have and id love to hear people's recommendations:

A) Is there any way to do this other then a pseudowire or plugging the fibers into the 6500s?

B) When i do get this working, should both sites have default-route originate by ospf or would it be better to set one preference higher?

[Talha Ansari]

I think that there could be a way.

As per my understanding you have default route learned on both the 3750 stacks that is originated from router A in OSPF. So when the packet is destined for internet there is no specific route in 3750's routing table and it takes the default route pointing to router A. But router A's BGP says that the destination is reachable via router B and hence the packet gets looped(correct me if wrong). The strange thing here is that Router A must prefer the eBGP routes learned from the ISPs connected on Router A and not the iBGP routes pointing towards Router B unless Router B is advertising more specific routes.

I think the datacenters working here are like active-standby where site B assumes the role as Disaster Recovery site. In this case try the below steps if possible :

Increment the local preference of *ALL* the BGP routes on router A when it is advertising to Router B. Hence, the Router A should be the preffered choice in your iBGP. The routes learned via the ISPs on the site B would be eBGP and the same would be advertised to Router A as iBGP routes but Router A must not use them as the LP of the routes on Router A would be higher.

On Router B configure a default information originate in OSPF with a higher metric(you can achieve this by distribute-list/route-map). So the OSPF domain must contain a default route pointing towards Router A. Ideally this should solve the issue!

HTH

Talha

[Kishore Chennupati]

Marwan,

You  suggested a very good idea of GRE but using MP-BGP or something is an  overkill. All that needs to be done is to hide the destination from the  internal network which can be done by GRE and should be enough. I  dont  believe that you need any BGP sAFI's here. Also as Matt suggested no  redis is required between the BGP and IGP.

The problem  the poster is having is that the switches (IGP) dont know the destination address for eg : 168.215.5.0/19

and  hence when Router A tries to route via the 3750's the switches send the  packets back to the Router A because of the default route.

Talha,

I  see what you are saying . However I would say its suboptimal routing to  go out to the internet and then come back from Router A to Router B.   The network has a backdoor fibre link and it would make sense to route  those destinations internally via iBGP.

cjsattle,

I  will definetly recommend to use GRE. I have tested this for you just in  case you want some assurance. I will paste the results in the following  post.

HTH

Kishore

[Kishore Chennupati]

cjasttler,

Here is the lab results just for you.  I just used a diff ip addressing.

+++++ WIth GRE

R1#sh ip bgp

BGP table version is 2, local router ID is 22.22.22.22

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

              r RIB-failure, S Stale

Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path

*>i33.33.0.0/16     1.1.1.2                  0    100      0 i

R1#traceroute 33.33.33.33

Type escape sequence to abort.

Tracing the route to 33.33.33.33

1 1.1.1.2 72 msec *  60 msec                 <<<<  Trace succesful

R1#sh ip route 1.1.1.2

Routing entry for 1.1.1.0/24

  Known via "connected", distance 0, metric 0 (connected, via interface)

  Routing Descriptor Blocks:

  * directly connected, via Tunnel0

      Route metric is 0, traffic share count is 1

+++++ Without GRE

R1#sh ip bgp

BGP table version is 4, local router ID is 22.22.22.22

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

              r RIB-failure, S Stale

Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path

*>i33.33.0.0/16     2.2.2.2                  0    100      0 i

R1#traceroute 33.33.33.33

Type escape sequence to abort.

Tracing the route to 33.33.33.33

  1 192.168.1.3 40 msec 36 msec 24 msec

  2 192.168.1.1 32 msec 16 msec 24 msec

  3

*Dec  7 20:16:59.151: ICMP: time exceeded rcvd from 192.168.1.3

*Dec  7 20:16:59.191: ICMP: time exceeded rcvd from 192.168.1.3

*Dec  7 20:16:59.215: ICMP: time exceeded rcvd from 192.168.1.3

*Dec  7 20:16:59.235: ICMP: bogus redirect from 192.168.1.3 - for 33.33.33.33 use gw 192.168.1.1

*Dec  7 20:16:59.235:       gateway address is one of our addresses

So, just use a GRE Tunnel between both the Routers and run iBGP between them

HTH

Regards,

Kishore

[cjsattler]

I appreciate everyones replies and so far i've come up with the 3 scenarios that would work.  I am going to bring them back to my team and see what they want to do:

A) Create a tunnel across the routers

   benefits

    - Allows exactly what im trying to accomplish without any changes on the switches

   drawbacks

    - Added complexity

    - MTU issues since everythings set to 1500 bytes ill have to up the mtu on the switches which means rebooting them with the system mtu command

     - added overhead of encapsulating the packets in a tunnel of traffic & cpu

B)  Active /Passive datacenter egress

   Benefits

    - Allows for failover without topology changes

   Drawbacks

    - Have to make sure all costs are less then what go out the preferred default

    - Only can use egress bandwidth on one switch and other carriers sit idle

    - Traffic shaping to force out customers onto certain carriers becomes impossible for the failover router

C) Plug one fiber into the 6500s one into the 3750s on trunked vlans and run a separate vlan interface to create a layer 2 network across 6500s while still allowing the 3750s to talk at layer 3

    Benefits

     - Allows for no single point of failureredundancy if a single 6500 dies

     - No tunneling

   Drawbacks

    - Have to recable

    - Is there ever a occasion where the physical ports will remain up but the 6500s stop communicating by ibgp? that would create incongruent routing

   - have to create a layer 2 trunk for the ports to plug into on the switches and document a vlan

[Marwan ALshawi]

Good luck

And make sure to rate the helpful posts in this discussion

Sent from Cisco Technical Support iPhone App