topic Re: BGP with Multiple ISP not functioning in Routing and SD-WAN

BGP with Multiple ISP not functioning

mbbmani — Tue, 05 Mar 2019 17:49:47 GMT

Hi,

I have 3 BGP session with 3 ISP using 3 routers.

The BGP session works but only 1 at a time.

Below are my configs

router bgp 22222
bgp log-neighbor-changes
neighbor YYY.YYY.YYY.YYY remote-as 123456
!
address-family ipv4
bgp dampening
network XXX.XXX.XXX.0 mask 255.255.255.252
network XXX.XXX.XXX.16 mask 255.255.255.240
network XXX.XXX.XXX.32 mask 255.255.255.224
network XXX.XXX.XXX.64 mask 255.255.255.192
network XXX.XXX.XXX.128 mask 255.255.255.128
network XXX.XXX.XXY.0 mask 255.255.255.0
network XXX.XXX.XXZ.0 mask 255.255.255.0
aggregate-address XXX.XXX.XXX.0 255.255.252.0 summary-only
neighbor YYY.YYY.YYY.YYY activate
neighbor YYY.YYY.YYY.YYY remove-private-as
neighbor YYY.YYY.YYY.YYY soft-reconfiguration inbound
neighbor YYY.YYY.YYY.YYY prefix-list inbound-route-filter in
neighbor YYY.YYY.YYY.YYY prefix-list route-advertise out
exit-address-family
!

What could it be the problem.

Please advice.

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Fri, 26 Jan 2018 17:29:58 GMT

Can you please more elaborate what do mean by "only 1 BGP work at a time" in which state they are (IDLE/Active) when they stop working ? can you please share the output of "show ip bgp summary" from 3 routers ? above configuration is showing only 1 router, appreciate if you can also share the topology table of your network as well.

Thank you.

Kindest regards,

uzzi

Re: BGP with Multiple ISP not functioning

mbbmani — Sat, 27 Jan 2018 05:33:08 GMT

Above is the network diagram. Blue color box is link balancer.

All 3 routers has the same config.

This is the sh ip bgp summary

Router 1 (Internet access good)

BGP router identifier 111.111.111.129, local AS number 111111
BGP table version is 15, main routing table version 15
8 network entries using 1088 bytes of memory
8 path entries using 448 bytes of memory
3/3 BGP path/bestpath attribute entries using 384 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1944 total bytes of memory
Dampening enabled. 0 history paths, 0 dampened paths
BGP activity 8/0 prefixes, 8/0 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
11x.11x.11x.77 4 111111 852 855 15 0 0 12:49:39 1

Router 2 (No Internet Access)

BGP router identifier 222.222.222.158, local AS number 222222
BGP table version is 24, main routing table version 24
8 network entries using 1088 bytes of memory
8 path entries using 448 bytes of memory
3/3 BGP path/bestpath attribute entries using 384 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1944 total bytes of memory
Dampening enabled. 0 history paths, 0 dampened paths
BGP activity 9/1 prefixes, 11/3 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
222.222.222.145 4 222222 1429 1441 24 0 0 21:38:24 1

Router 3 ( No internet access)

BGP router identifier 333.333.333.134, local AS number 333333
BGP table version is 16, main routing table version 16
8 network entries using 960 bytes of memory
8 path entries using 416 bytes of memory
3/3 BGP path/bestpath attribute entries using 372 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1772 total bytes of memory
Dampening enabled. 0 history paths, 0 dampened paths
BGP activity 8/0 prefixes, 8/0 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
333.333.333.133 4 333333 863 866 16 0 0 12:59:40 1

Re: BGP with Multiple ISP not functioning

mbbmani — Sat, 27 Jan 2018 05:38:51 GMT

hi,

Only Router 1 has internet access.

Router 2 and 3 does not have internet access.

router-1r#ping 8.8.8.8 so gi0/1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of NNN.NNN.NNN.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

router-2#ping 8.8.8.8 so fa0/1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of NNN.NNN.NNN.9

router-3#ping 8.8.8.8 so gi0/1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of NNN.NNN.NNN.5
.....
Success rate is 0 percent (0/5)

.....

Re: BGP with Multiple ISP not functioning

Dennis Mink — Sat, 27 Jan 2018 05:44:19 GMT

your configuration ofr AS22222 only contains one neighbour. why is there no 3 nieghbours if you have 3 ISPs?

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Sat, 27 Jan 2018 07:34:17 GMT

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Sat, 27 Jan 2018 07:42:25 GMT

With your shown output it seems you are advertising 1 prefix to all of 3 upstream, I don't know why you are doing this because in a usual scenario 1 edge router and then you configure 3 ebgp in one instance (bgp AS) or if you do multi homing yoi configure dual ISP setup but use MED if ISP agree or AS PREPAND which is known attribute and accepted by all ISP over the internet, are you advertising single /24 to all 3 upstream from your 3 routers? From where 8.8.8.8 is not reachable can you please share the output of "show ip bgp 8.8.8.8 " and also trace from same router from where internet is not working?

Thank you.

Kindest regards

Uzzi

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Sat, 27 Jan 2018 07:45:15 GMT

Further to below also can you please share "show ip bgp neighbor (neighbor ip) advertise-routes " from 3 routers as well.

Thank you.

Kindest regards

Uzzi

Re: BGP with Multiple ISP not functioning

paul driver — Sat, 27 Jan 2018 19:04:51 GMT

Hello

Your config shows you only have ebgp peering and no ibgp between each rtr

as such external routes cannot be shared between each rtr and also each rtr isn’t aware of each other

As for each rtr not able to each the internet can you confirm what interfaces you are basing your source on? - if this is the internal lan interface are you performing any NAT for this?

res

paul

Re: BGP with Multiple ISP not functioning

mbbmani — Sun, 28 Jan 2018 15:31:41 GMT

Hi,

I attached the sh ip bgp 8.8.8.8

Router 1

Router# sh ip bgp 8.8.8.8
BGP routing table entry for 0.0.0.0/0, version 2
Paths: (1 available, best #1, table default, RIB-failure(17))
Not advertised to any peer
9930, (received & used)
11x.11x.11x.77 from 11x.11x.11x.77 (yyy.yyy.yyy.16)
Origin IGP, localpref 100, valid, external, best

Router 2 - now it is able to access internet.

Router#sh ip bgp 8.8.8.8
BGP routing table entry for 0.0.0.0/0, version 9
Paths: (1 available, best #1, table default, RIB-failure(17))
Not advertised to any peer
10204, (received & used)
222.222.222.133 from 222.222.222.133 (yyy.yyy.yyy.6)
Origin IGP, metric 0, localpref 100, valid, external, best

Router 3

Router##sh ip bgp 8.8.8.8
BGP routing table entry for 0.0.0.0/0, version 2
Paths: (1 available, best #1, table default, RIB-failure(17))
Not advertised to any peer
38278, (received & used)
333.333.333.145 from 333.333.333.145 (yyy.yyy.yyy.32)
Origin IGP, localpref 100, valid, external, best

Re: BGP with Multiple ISP not functioning

mbbmani — Sun, 28 Jan 2018 15:35:56 GMT

Hi,

The source interface of the ping is the Internal ISP independent IP segment (public ip).

I have no NAT for this interface.

I have shortage of physical interface in this router. It is cisco 2911.

0/0 - WAN IP (connect to ISP)

0/1 - LAN IP (ISP independent IP - which is what I am working on)

0/2 - LAN IP (ISP provided IP segment)

Can I use secondary IP in int 0/1 or loopback for iBGP?

Thanks.

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Sun, 28 Jan 2018 17:13:29 GMT

My bad sir, show ip bgp will always show it as routing entry 0.0.0.0/0 means all your ISPs advertising you a default route, but we need to check the IGP, can you please share the output of "show ip route 8.8.8.8 " and also trace result from 3 routers to 8.8.8.8

Thank you.

Kindest regards

Uzzi

Re: BGP with Multiple ISP not functioning

mbbmani — Tue, 06 Feb 2018 18:33:26 GMT

Hi,

I attached the full config from all the 3 routers.

Issues :

1. Internet is only possible for ROUTER1 via ISP1.

2. Internet for ROUTER2 and ROUTER3 is via iBGP connection via ISP1.

Appreciate your help.

Re: BGP with Multiple ISP not functioning

mbbmani — Mon, 29 Jan 2018 08:44:14 GMT

Hi,

I attached the sh bgp ip 8.8.8.8 output.

Thank you for your advice.

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Mon, 29 Jan 2018 14:02:02 GMT

Why you configured static route when you are receiving default route from ISP? default route will be always preferred over BGP due to Admin distance. Usually the problem in reachability via 2 different sources (from 1 is reachable and from 2nd is not reachable is most probably due to NLRI of the reverse path to the source) appreciate if you can share the output of "show ip route 8.8.8.8 " and trace route 8.8.8.8 with source sub interfaces (.8 and .10).

Thank you.

Kindest regards,

Uzzi

Re: BGP with Multiple ISP not functioning

mbbmani — Mon, 29 Jan 2018 16:50:59 GMT

Hi,

I attached the traceroute and ip route output for router 2.

I have the default route for the ISP supplied IP address. Int gi0/3.

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Tue, 30 Jan 2018 09:45:39 GMT

My question is when you are learning default route from ISP why you defined is manually ? Also I cannot see any Gi0/3 in your provided router configuration moreover about the IP address of gi0/1.8 from where you are unable to reach over the internet, are you advertising this to your ISP ? Gi0/1.8 and .gI0/1.10 both belongs to public IP address ? if yes then are you advertising /28 of Gi0/1.10 to ISP ? you can check this by "show ip bgp neighbor (Neighbor IP) advertise-route.

203.115.224.189 is visible over the internet at route views however in your case it seems you are not advertising Gi0/1.10 prefix to your ISP and as part of BGP maximum prefix acceptance over the internet is /24 I have no idea why you are advertising in your network /27 /28 and /29 networks.

Re: BGP with Multiple ISP not functioning

mbbmani — Tue, 06 Feb 2018 18:34:12 GMT

Hi,

I attached the router config after some amendments.

I use private IP for iBGP peering.

I also remove most of the static routes.

ISP1 is working but ISP2 is not.

If I shutdown Router ISP1, then ISP2 works.

Sorry for the trouble. Appreciate your help.

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Tue, 30 Jan 2018 18:54:47 GMT

Please remove the static route from NTT-Router also one more thing I observed that out of 3 ISP's you are doing prepanding for 1 which I cannot see in your BGP config which you shared. strange thing is you are prepanding to AS 10204 and then they are handing over to AS 38278 which is one of your ISP and have direct peering with you but not advertising your /22 block over the internet, do you have any CCIE or CCNP to maintain your AS? please make sure you are advertising the block "Network 103.100.204.5 Mask 255.255.255.252" in your BGP AS by going "router bgp (AS#)"

Kindest regards,

Uzzi

Re: BGP with Multiple ISP not functioning

Muhammad Uzair — Tue, 30 Jan 2018 19:54:04 GMT

The main issue that you are facing is that R1 R2 R3 don't have complete information of each others connected networks and static routes. What is happening is that when you trace from R2 taking the source of your Gi0/1.8 interface the out traffic goes to the connected ISP and the return traffic till hop 3 also comes back through the same ISP to R2, from hop 4 onward the return path is through either ISP 1 or ISP 3 (whichever is most suitable through BGP path selection criteria) when the return traffic comes back through ISP 1 through router 1 router 1 doesn't have the route to reach the source IP of R2 interface Gi0/1.8, you can verify this by displaying "show ip route" ip address of R2 gi0/1.8 interface, R1 will have no entry for this IP address.

If the above is not true then you have to check the load balance config which might be dropping the traffic coming from R1 the source of R2 interface; due to reverse path check fail.

You can implement 3 solutions.

(1) Either run a IGP between R1 R2 R3 (most suitable for scalability ).

(2) You can redistribute connected/static routes into BGP on all R1 R2 R3 (But be careful to filter out any more specific than /24 prefix to your ISP)

(3) Use static route to define all connected networks of R1 R2 R3 to each other.

Kindest regards,

Uzzi