OSPF and BGP

[skywalker_007]

FOr one of our customer , they have two router one in each DC

The router in DC1 is connected to COre Switch in DC1 >> This is a Layer 2 connection

Similarly router in DC2 is connected to Core switch in DC 2 ( again L2 connection)

The DCs are connected over a Physical underground link .

Between Router 1 and Router 2 , we have defined a VLAN 11 where we run OSPF . BGP is also configured between two routers.

They use BGP peer Group . I want to understand how the failover will work in case ISP link goes down .

Both the router have a default route pointing to their respective ISP .

In OspF it have defined the redistribute connected and static plus network address for4 subnets also .

Below is the config

interface Port-channel1.11
encapsulation dot1Q 11
ip address 10.2.2.2 255.255.255.248
ip ospf message-digest-key 1 md5 7 XXXXXXXXXXXXXXX

!
router ospf 11
router-id 10.2.2.10
area 0 authentication message-digest
redistribute connected subnets
redistribute static subnets
passive-interface default
no passive-interface Port-channel1.11
network 10.2.2.0 0.0.0.255 area 0
network 10.2.3.0 0.0.0.255 area 0
network 10.2.4.0 0.0.0.255 area 0
network 10.2.5.0 0.0.0.255 area 0
!
router bgp 27272
bgp router-id 10.2.2.10
bgp log-neighbor-changes
bgp graceful-restart
bgp maxas-limit 100
timers bgp 10 30
neighbor TEST_GROUP peer-group
neighbor TEST_GROUP remote-as 27272
neighbor TEST_GROUP password 7 XXXXXXXXXXXXXXXXXX
neighbor TEST_GROUP update-source Loopback0
neighbor 10.2.2.11 peer-group TEST_GROUP   >>>> this IS router 2 loop back address in DC2

neighbor 22.2.22.12 remote-as XXXXX
neighbor 22.2.22.12 description ISP
neighbor 22.2.22.12 password 7 XXXX
!
address-family ipv4
network 10.2.2.0 mask 255.255.255.0 route-map INTERNAL-ROUTES
network 10.2.3.0 mask 255.255.255.0 route-map INTERNAL-ROUTES
network 10.2.4.0 mask 255.255.255.0 route-map INTERNAL-ROUTES
network 10.2.5.0 mask 255.255.255.0 route-map INTERNAL-ROUTES
neighbor TEST_GROUP send-community both
neighbor TEST_GROUP next-hop-self
neighbor TEST_GROUP soft-reconfiguration inbound
neighbor 10.2.2.11 activate
neighbor 22.2.22.12 activate
neighbor 22.2.22.12 send-community both
neighbor 22.2.22.12 prefix-list FILTER-OUT out
exit-address-family

ip route 0.0.0.0 0.0.0.0 22.2.22.12
ip route 10.2.2.0 255.255.255.0 Null0
ip route 10.2.3.0 255.255.255.0 10.2.3.254
ip route 10.2.4.0 255.255.255.0 Null0
ip route 10.2.5.0 255.255.255.0 Null0

ip prefix-list FILTER-OUT seq 10 permit 10.2.2.0/24
ip prefix-list FILTER-OUT seq 20 permit 10.2.3.0/24
ip prefix-list FILTER-OUT seq 30 permit 10.2.4.0/24
ip prefix-list FILTER-OUT seq 40 permit 10.2.5.0/24

interface Loopback0
description Loopback OSPFBGP
ip address 10.2.2.10 255.255.255.255
!
interface Port-channel1.14
encapsulation dot1Q 14
ip address 10.2.3.226 255.255.255.248
glbp 1 ip 10.2.3.227
glbp 1 priority 200
no glbp 1 load-balancing
glbp 1 authentication md5 key-chain glbp

interface GigabitEthernet0/0/4
description ISP
ip address 22.2.22.11 255.255.255.252
no negotiation auto
!
!

[skywalker_007]

Also , is there a link between

if ISP1 goes down ,

does it make the GLB on VLAN 14 to shift to Router 2 so that Internal Traffic which is behind the core switches goes directly to Router 2 ?

[Richard Burts]

There are many things that I do not understand about your environment (and probably most of them do not really impact this discussion). But I do have a couple of comments:

- I do not understand why you are using peer groups if you have only 1 internal peer for BGP.

- whether it is peer groups or something else, if ISP 1 goes down how would that be communicated to router 2 and what difference would it make on router 2?

- why do both routers have a static default route configured? A static default route makes failover much more of a challenge.

- why not trust the default route that each ISP probably is advertising?

[skywalker_007]

I understand the point , and i will highlight this to customer. Also how about redistributing default route to ospf like mentioned by Georg .

Does that make difference .

In your opinion , if ISP1 goes down , in current case , there is no possibility to divert all traffic ( via OSPF and BGP) from Router R1 to router R2 and hence ISP2 ?  thats the goal ultimately .  

[dgrinceac]

There are more questions than answers indeed.
From what I can see in the configuration there is not redundancy between ISPs at this point for time because of default static route pointing towards ISP router.
As you are running BGP with ISP you can check what they are advertising to you using "show ip bgp neighbor X.X.X.X routes" command where X.X.X.X is the ISP ip address. Make sure you have 0.0.0.0/0 there before actually removing static route or you can break your network.

According to your scenario there is no point to send default using OSPF because you already have a iBGP session running between routers. Instead you need to make sure that default route that is installed into routing table is learnt from the ISP using BGP. If that condition is satisfied you don't really need to do anything else. Routes learnt from from external BGP (ISP) would be automatically propagated to iBGP(second router). You'll have failover out of the box.

So the plan high level recommendation would be:
1) Make sure ISP send you default route "show ip bgp", "show ip bgp neighbor X.X.X.X routes" on both routers
2) If previous condition is satisfied schedule an outage and remove static route from the configuration:
R1 - no ip route 0.0.0.0 0.0.0.0 X.X.X.X ! Validate connectivity in DC1
R2 - no ip route 0.0.0.0 0.0.0.0 Y.Y.Y.Y ! Validate connectivity in DC2
3) Confirm the default route is propagated over BGP.
4) Validate failover by shutting down one of the bgp peers and checking internet connectivity
router bgp xxx
neighbor X.X.X.X shut
end
ping 1.1.1.1
ping 8.8.8.8

router bgp xxx
no neighbor X.X.X.X shut

Be careful as you the changes might be quite disruptive.
Hope that helps.

[skywalker_007]

Hello , 

When i run below command , i see hell lot of routes and its a very big table , does not seem to end . But i cant see default route

x.x.x.x is my ISP 1 IP

R1#show ip bgp neighbors x.x.x.x routes   

t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
*> 1.0.0.0/24 x.x.x.x 0 0 3356 13335 i
*> 1.0.4.0/24 x.x.x.x 0 0 3356 4826 38803 i
*> 1.0.4.0/22 x.x.x.x 0 0 3356 4826 38803 i
*> 1.0.5.0/24 x.x.x.x 0 0 3356 4826 38803 i
*> 1.0.6.0/24 x.x.x.x 0 0 3356 4826 38803 i
*> 1.0.7.0/24 x.x.x.x 0 0 3356 4826 38803 i
* 1.0.16.0/24 x.x.x.x 0 0 3356 2516 2519 i
* 1.0.64.0/18 x.x.x.x 0 0 3356 2516 7670 18144 i
Network Next Hop Metric LocPrf Weight Path
* 1.0.128.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
*> 1.0.128.0/19 x.x.x.x 0 0 3356 38040 23969 i
*> 1.0.128.0/18 x.x.x.x 0 0 3356 38040 23969 i
*> 1.0.128.0/17 x.x.x.x 0 0 3356 38040 23969 i
* 1.0.129.0/24 x.x.x.x 0 0 3356 1299 4651 23969 i
* 1.0.132.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
* 1.0.133.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
* 1.0.134.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
* 1.0.135.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
* 1.0.136.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
* 1.0.137.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
Network Next Hop Metric LocPrf Weight Path
* 1.0.138.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
* 1.0.139.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
* 1.0.141.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
*> 1.0.144.0/20 x.x.x.x 0 0 3356 38040 23969 i
* 1.0.160.0/22 x.x.x.x 0 0 3356 1299 38040 23969 i
*> 1.0.160.0/19 x.x.x.x 0 0 3356 38040 23969 i
* 1.0.164.0/24 x.x.x.x 0 0 3356 174 38040 23969 i
* 1.0.165.0/24 x.x.x.x 0 0 3356 174 38040 23969 i

and so on ...

If i grep with 0.0.0.0 i dont see any default 

R1#show ip bgp neighbors x.x.x.x routes | i 0.0.0.0
*> 20.0.0.0/11 x.x.x.x 0 0 3356 8075 i
*> 40.0.0.0/19 x.x.x.x 0 0 3356 1299 7332 4249 i
*> 40.0.0.0/10 x.x.x.x 0 0 3356 7018 4249 4249 4249 4249 4249 i
* 60.0.0.0/13 x.x.x.x 0 0 3356 4837 4837 i
*> 70.0.0.0/17 x.x.x.x 0 0 3356 1239 10507 i
*> 70.0.0.0/13 x.x.x.x 0 0 3356 1239 i
*> 80.0.0.0/15 x.x.x.x 0 0 3356 6830 5089 i
*> 80.0.0.0/13 x.x.x.x 0 0 3356 6830 5089 i
*> 90.0.0.0/16 x.x.x.x 0 0 3356 5511 3215 i
*> 100.0.0.0/16 x.x.x.x 0 0 3356 701 i
*> 110.0.0.0/15 x.x.x.x 0 0 3356 2914 10013 i
* 120.0.0.0/12 x.x.x.x 0 0 3356 4837 4837 i
* 130.0.0.0/21 x.x.x.x 0 0 3356 2914 6908 i
*> 140.0.0.0/19 x.x.x.x 0 0 3356 1299 7632 23700 i
*> 140.0.0.0 x.x.x.x 0 0 3356 1299 7632 23700 i
*> 160.0.0.0/17 x.x.x.x 0 0 3356 42 715 i
*> 170.0.0.0/23 x.x.x.x 0 0 3356 16735 263536 264957 i
* 170.0.0.0/22 x.x.x.x 0 0 3356 16735 263536 264957 i
*> 180.0.0.0/10 x.x.x.x 0 0 3356 2914 4713 i
*> 190.0.0.0/18 x.x.x.x 0 0 3356 1299 13489 i
*> 210.0.0.0 x.x.x.x 0 0 3356 1299 7473 7474 i
* 220.0.0.0/16 x.x.x.x 0 0 3356 1299 17676 i
* 220.0.0.0/10 x.x.x.x 0 0 3356 1299 17676 i

and when i run the same command for another router IP as neighbor , again i see a very big routing 

R1#show ip bgp neighbors 10.2.52.240 routes

Network Next Hop Metric LocPrf Weight Path
* i 1.0.0.0/24 10.2.52.240 0 100 0 8220 13335 i
* i 1.0.4.0/24 10.2.52.240 0 100 0 8220 4826 38803 i
* i 1.0.4.0/22 10.2.52.240 0 100 0 8220 4826 38803 i
* i 1.0.5.0/24 10.2.52.240 0 100 0 8220 4826 38803 i
* i 1.0.6.0/24 10.2.52.240 0 100 0 8220 4826 38803 i
* i 1.0.7.0/24 10.2.52.240 0 100 0 8220 4826 38803 i
*>i 1.0.16.0/24 10.2.52.240 0 100 0 8220 2519 i
*>i 1.0.64.0/18 10.2.52.240 0 100 0 8220 7670 18144 i
Network Next Hop Metric LocPrf Weight Path
*>i 1.0.128.0/24 10.2.52.240 0 100 0 8220 38040 23969 ?
* i 1.0.128.0/19 10.2.52.240 0 100 0 8220 38040 23969 i
* i 1.0.128.0/18 10.2.52.240 0 100 0 8220 38040 23969 i
* i 1.0.128.0/17 10.2.52.240 0 100 0 8220 38040 23969 i
*>i 1.0.129.0/24 10.2.52.240 0 100 0 8220 4651 23969 ?
*>i 1.0.132.0/24 10.2.52.240 0 100 0 8220 38040 23969 ?
*>i 1.0.133.0/24 10.2.52.240 0 100 0 8220 38040 23969 ?
*>i 1.0.134.0/24 10.2.52.240 0 100 0 8220 38040 23969 ?
*>i 1.0.135.0/24 10.2.52.240 0 100 0 8220 38040 23969 ?
*>i 1.0.136.0/24 10.2.52.240 0 100 0 8220 38040 23969 ?
*>i 1.0.137.0/24 10.2.52.240 0 100 0 8220 38040 23969 ?

[mlund]

Hi

It seems that is is a little confusions here.

Let me give my understanding of this.

Ospf is used, in this case, mainly to advertise the loopback adresses. Because the ibgp is using the loopbacks to form neighbor ship with each other.
In router 1 you got routes received from your isp1. Those routes are forwarded to router 2 via ibgp.

In router 2 you got routes received from your isp2. Those routes are forwarded to router 1 via ibgp.
Now each router, 1 and 2, decides what routes it will be using, either the ebgp route or the ibgp route.

For those routes you receive from ISP:s, if ISP1 has the one with best metric, it will be inserted into the routing-table in both routers.
For those routes you receive from ISP:s, if ISP2 has the one with best metric, it will be inserted into the routing-table in both routers.

For those routes you receive from both ISP:s and if it is the same metric, both routers will install it's ebgp received route.
If ISP1 stops sending the route, the one from ISP2 will be inserted in the routing-table in both routers, and vice verse.

If you have the static default configured, that route will be in the routingtable as long as the interface towards ISP is up.
This may be dangerous, because sometimes link can be up even though the ISP router is not functioning. And in those cases you will blackhole your traffic.

If you are receiving the full table without default, then you can still have your static defult configured.

/Mikael

[dgrinceac]

According to the output and statement that routing table doesn’t seems to end we can make a conclusion that you are receiving full BGP table.

Having full table provides a lot of flexibility for the traffic engineering. Basically that means you have all routes of the internet which makes default route a bit redundant.

in that case it looks like you have proper setup. 
in case one of the ISPs fails the full routing table will be learnt from ibgp peer in different DC.
Default static route will send traffic towards any unknown destination that is not listed in the full table. There should be very limited to none routes matching this destination if you have full bgp table. If both routers have full bgp table coming from the ISP then you don’t really need it but it makes no harm to have it. 

[Richard Burts]

There are multiple aspects we could consider for this discussion (OSPF vs BGP, both ISP active and sharing load vs primary/backup, etc) but the most important aspect is failover. And I want to focus on that aspect for now. So you have 2 ISP and it looks like each of your routers is learning lots of routes from its ISP peer, and each of your routers uses IBGP to share its learned ISP routes with its internal peer. So if ISP 1 goes out of service and your router stops getting BGP from its ISP what will change? 

When router 1 stops receiving BGP from ISP 1 the BGP neighbor relationship will time out and router 1 will remove all routes learned from ISP 1 from its routing table. router 1 will then have the ISP 2 routes it learned via IBGP. So you would have partial failover. But what would not change if ISP 1 goes out of service? The default route would not change. As long as your router 1 outside interface is in the up state the default route will remain in the routing table and pointing to ISP 1 - which is no longer functional. So you do not have full failover.

How can you fix this so that you get complete fail over? You fix this by removing the static default route from both of your routers. When you remove the static default route you will begin to use the default route advertised by your ISP, and this will provide the full fail over.

I know that you have indicated that you are not sure that you are receiving a default route from your ISP. I have never seen an ISP send the full routing table to a customer that did not also send a default route. You certainly need to check with your ISP and verify that they are advertising a default route to you.

[skywalker_007]

@Richard Burts Great Thanks . This seems to be much more clear now .

I understood the iBGP part but what is the Purpose of having OS¨PF and below statements 

of reditribute static and connected and then calling 4 different networks using network command . How is it linked to iBGP 

router ospf 20
router-id 10.2.52.240
redistribute connected subnets
redistribute static subnets
passive-interface default
no passive-interface Port-channel1.15
network 10.2.52.0 0.0.0.255 area 0
network 10.2.53.0 0.0.0.255 area 0
network 10.2.54.0 0.0.0.255 area 0
network 10.2.55.0 0.0.0.255 area 0

Lastly i would like to pay my sincere regards and thanks to people who answered my queries . Thank you @Georg Pauwen @MHM Cisco World @mlund @dgrinceac @Richard Burts . Hats off to you for helping a learner to clear doubts

[Richard Burts]

I hope that now we have an understanding of BGP fail over and the default route implications. So we can move on to consider other aspects of your network. You ask this question "what is the Purpose of having OS¨PF and below statements". I do not have enough understanding of your network to provide complete responses. But based on what I  know so far I will suggest this as an answer: 

- there appear to be 4 networks/subnets that are the networks inside your organization.

- there are 2 core switches (and I assume at least 4 vlans on each core switch).

- each core switch connects to a router.

- each router connects to 1 core switch and to 1 ISP.

- it is not clear which device is doing the routing when traffic from one vlan/subnet wants to access another vlan/subnet but I am guessing that this routing logic is on the core switches and not on the routers. (is this correct?)

- the switches would be able to route between vlans/subnets because the subnets are locally connected. 

- I am assuming that the vlans/subnets are not locally connected on the routers. So the routers need some way to learn the subnets that are locally connected on the core switches. That is where OSPF becomes important. Running OSPF between each core switch and each router would allow the routers to be aware of the inside networks - and would allow the routers to advertise your inside subnets to each ISP.

[skywalker_007]

@dgrinceac  Thank  you . This seems to be the case . I will discuss about static default route .

Ofcourse ,  there is still although very little chance that some of the routes out of the big list the ISP is advertising could be missed 

[skywalker_007]

Hi ,

OSPF is running between routers only ( VLAN 15)  ;

Core switch is purely Layer 2 .

There is a FW connected to both the core switches which act as Layer 3

Primary FW connects to Core switch in DC1  , Secondary Firewall connects to core switch in DC2 .

Core switch DC1 also connected to Router DC1 

Core Switch DC2 also connected to Router DC2

Core switch DC1  is connected to Core Switch DC2 ( via underground cable )

My assumption is that OSPF vlan 15 acts as a carrier for IBGP . 

Vlan 15 which is OSPF vlan is only present on routers ( not on switch and not on  FW) . So this purely for cross talk between routers to carry ibgp

Also , ospf has redistribute command to redistribute static and connected . My assumption is that whatever ibgp routes i can see between two routers are actually coming via OSPF path of vlan 15

[skywalker_007]

I have another query ,

Which command in the configuration is allowing my side router to accept all the routes whatever is being sent by ISP1 ?.

Cant i do filter? . Normally we use prefix list , but i cant see any prefix list here , . Is it like it accepts everything which is being sent by default ?

I can see a prefix list out with specific networks which means we are only sending specifc but i cant see prefix list IN

neighbor x.x.x.x prefix-list FILTER-OUT out
ip prefix-list FILTER-OUT seq 10 permit 10.2.52.0/24
ip prefix-list FILTER-OUT seq 20 permit 10.2.53.0/24
ip prefix-list FILTER-OUT seq 30 permit 10.2.54.0/24
ip prefix-list FILTER-OUT seq 40 permit 10.2.55.0/24

[Richard Burts]

When you are running BGP the default is to accept all routes advertised to you by the BGP peer. It is possible to filter and to only accept certain of the routes which are advertised to you by the BGP peer. Prefix list is one way to do that filtering. A Route Map is another way to do the filtering of what is advertised to you.

Thanks for the clarification that the core switches are layer 2 only switches. So is the routing of traffic between inside subnets done on the router or on the firewall?

[skywalker_007]

Routing inside is done by firewall .