BGP Multihoming with 2 ISPs and 2 Routers

[swaroop.katargunde]

Hi All,

I need your help/suggestions in one of the issue with our BGP Multihoming environment.

We have 2 ISP's and 2 Router's and we have configure BGP with our owned /24 Public IP with AS number with IBGP over HSRP for failover.

Now the issue is whenever our Primary ISP failed , there is an outage for 2 minutes until the failover happens and the neighbors come up.

Is there any way where i can configured BGP with 2 ISP & 2 routers in multihoming environment where, if my Primary Internet or Router fails there will be no disconnect/outage like most of the Data Centers.

Can you also let me know if i configure the same with load sharing and best path.

Below is my current diagram and the configuration.

The IP address which i have mentioned in the diagram is our owned IP address.

Below is the configuration :

Primary Router 

!
track 1 ip sla 1 reachability
!
!
interface Port-channel30
no ip address
negotiation auto
!
interface Port-channel30.1
description Public_IP_pool
encapsulation dot1Q 30
ip address 109.811.61.3 255.255.255.0
standby 1 ip 109.811.61.1
standby 1 priority 110
standby 1 preempt delay minimum 60
standby 1 track 1 decrement 20
!
interface GigabitEthernet0/0/0
description Connected to Primary FTD
no ip address
negotiation auto
channel-group 30
!
interface GigabitEthernet0/0/1
description Connected to Secondary FTD
no ip address
negotiation auto
channel-group 30
!
interface GigabitEthernet0/0/2
description TATA_WAN
ip flow monitor PRTG input
ip flow monitor PRTG output
ip address 116.109.126.237 255.255.255.252
ip nat outside
negotiation auto
!
interface GigabitEthernet0/0/3
description LAN
ip address 10.7.1.7 255.255.255.0
ip nat inside
negotiation auto
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
no ip address
shutdown
negotiation auto
!
router bgp 134124
bgp log-neighbor-changes
network 109.811.61.0 mask 255.255.255.0
neighbor 109.811.61.4 remote-as 134124
neighbor 109.811.61.4 next-hop-self
neighbor 116.109.126.236 remote-as 4355
neighbor 116.109.126.236 description PRIMARY_ISP
neighbor 116.109.126.236 soft-reconfiguration inbound

ip sla 1
icmp-echo 116.109.126.236 source-interface GigabitEthernet0/0/2
ip sla schedule 1 life forever start-time now

Secondary Router :

!
interface Port-channel30
no ip address
negotiation auto
!
interface Port-channel30.1
description Public_IP_pool
encapsulation dot1Q 30
ip address 109.811.61.4 255.255.255.0
standby 1 ip 109.811.61.1
standby 1 preempt
!
interface GigabitEthernet0/0/0
description Connected to Primary FTD
no ip address
negotiation auto
channel-group 30
!
interface GigabitEthernet0/0/1
description Connected to Secondary FTD
no ip address
negotiation auto
channel-group 30
!
interface GigabitEthernet0/0/2
description WAN-AIRTEL
ip address 181.74.248.141 255.255.255.252
ip nat outside
negotiation auto
nat64 enable
ip virtual-reassembly
!
interface GigabitEthernet0/0/3
description LAN
ip address 10.7.1.8 255.255.255.0
ip nat inside
negotiation auto
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
no ip address
shutdown
negotiation auto
no cdp enable
!
router bgp 134124
bgp router-id 109.811.61.4
bgp log-neighbor-changes
network 109.811.61.0 mask 255.255.255.0
neighbor 109.811.61.3 remote-as 134124
neighbor 109.811.61.3 next-hop-self
neighbor 181.74.248.139 remote-as 9426
neighbor 181.74.248.139 description AIRTEL_ILL_ISP
neighbor 181.74.248.139 soft-reconfiguration inbound
neighbor 181.74.248.139 route-map PREPEND_OUT out
!

!
ip prefix-list as-path_prepend seq 5 permit 109.811.61.0/24

!
route-map back-up permit 10
match ip address prefix-list as-path_prepend
set as-path prepend 134124 134124 134124 134124 134124
!
route-map PREPEND_OUT permit 10
match ip address BACKUP_PREPEND
set as-path prepend 134124 134124

 ip access-list extended BACKUP_PREPEND
permit ip 109.811.61.0 0.0.0.255 any

[Peter Paluch]

Hello Swaroop,

Now the issue is whenever our Primary ISP failed , there is an outage for 2 minutes until the failover happens and the neighbors come up.

I am somewhat uncertain what you mean by "neighbors come up". Based on the configuration I could check, the iBGP peering between the .3 and .4 routers should be constantly up, and so should the eBGP peerings unless the ISPs themselves have an issue. Are you truly observing a flap of BGP adjacencies beyond the eBGP session to your Primary ISP when the Primary ISP goes down?

In any case, we need to understand the sequence of events that take place when your Primary ISP goes down - and we also need to understand exactly how it goes down. Can you describe it in as much detail as possible? Understanding the exact and complete sequence of events will allow us to understand where the delay in failing over is occurring.

There is only one thing I can suggest for the moment, but not knowing the exact details, it is only a blind shot: Since the delay in failover can be caused by the slowness in detecting that the BGP peer is down, you have a couple of options to consider:

• Shorten the keepalive and holdtime timers for your BGP neighbors, both internal and external. For example, you could use 5/15 sec as keepalive/holdtime. Not an ideal solution, but with just a couple of BGP neighbors, still acceptable.
• Use BFD with relaxed timers, for example, 500ms interval and multiplier of 3. This solution would be ideal.
• Use neighbor <IP-address> fall-over for the neighbor to tear down the adjacency whenever the loss of the route (in your case, due to the loss of the interface) is detected.

I am looking forward to hearing from you! :)

Best regards,
Peter

[swaroop.katargunde]

Hi Peter , Thank you so much for your reply !!!.

I am somewhat uncertain what you mean by "neighbors come up". Based on the configuration I could check, the iBGP peering between the .3 and .4 routers should be constantly up, and so should the eBGP peerings unless the ISPs themselves have an issue. Are you truly observing a flap of BGP adjacencies beyond the eBGP session to your Primary ISP when the Primary ISP goes down?

"neighbors come up" : I mean to say that once the HSRP failover completed ( 40 to 120 seconds) it take around 20 to 40 seconds to ISP (external) neighbor to come up. Yes i see a flap when my Internet Link goes down.

In any case, we need to understand the sequence of events that take place when your Primary ISP goes down - and we also need to understand exactly how it goes down. Can you describe it in as much detail as possible? Understanding the exact and complete sequence of events will allow us to understand where the delay in failing over is occurring.

There were multiple scenarios where the ISP goes down, Sometime Interface goes down, sometime Interface is up & the next hop is pingable but the reachability is down. Is there anything more that i need to check or i have missed to check.

There is only one thing I can suggest for the moment, but not knowing the exact details, it is only a blind shot: Since the delay in failover can be caused by the slowness in detecting that the BGP peer is down, you have a couple of options to consider:

Shorten the keepalive and holdtime timers for your BGP neighbors, both internal and external. For example, you could use 5/15 sec as keepalive/holdtime. Not an ideal solution, but with just a couple of BGP neighbors, still acceptable.
Use BFD with relaxed timers, for example, 500ms interval and multiplier of 3. This solution would be ideal.
Use neighbor <IP-address> fall-over for the neighbor to tear down the adjacency whenever the loss of the route (in your case, due to the loss of the interface) is detected.
Thank you so much for the solution but will this help me achieve my WAN connectivity to be up without any disruption. How can i configure in a way where i minimize the downtime / or to be up constantly.

Again Thank you so much for your input on this. I will wait to hear from you back.

[paul driver]

Hello

Looking at your topology another thing that could potentially occur is an asymmetric routing scenario where for some reason hrsp failovers over to your secondary router but the return path from your isp is still coming back via your primary path then you have a situation where the nat translation table now on the secondary router and not on the primary as such connectivity is lost.

I good solution to this would be to introduce stateful nat with hrsp so the active primary nat rtr will synchronise it nat table with the secondary rtr and no connectivity should then be lost refards this possible senario 

[swaroop.katargunde]

Hi Paul,

Thank you so much for your input.

I never checked the NAT.

I will do a test scenario today after hours and see if that the case.

I will let you know the output.

Thank you

[paul driver]

Hello

a couple of things you could  implement are:

1) shorten the bgp timers keepalive and hold-time

example:

timers bgp 10 30

2) apply bidirectional forwarding detection 

(BFD)

example:

bfd interval 100 min_rx 100 multiplier 5

BFD packets sent and expected every 100 milliseconds with a 5 packet threshold so after 5 missed packets equaling  500 milliseconds the bgp neighbour is deemed to be down