BGP Default route for two different ISP

[satish.txt1]

Folks, 

I have two ISP terminated on my two distinct edge routers (Its Cisco Nexus 93180 Switch) from here its connected to Border Leaf switch which is connected to EVPN fabric. This is new setup so my question is can I connect both border lead switches to two different ISP for redundancy and take default BGP route? Reason I am asking because my Edge routers are L3 switches and they can't take full BGP table. (Because of budget issue we can't replace them over night). 

All I want to know are there any major issue with this approach? I am only looking for redundancy if just incase ISP-A or B down so I have backup route.

[M02@rt37]

Hello @satish.txt1 

A lot of things to discuss here. Thanks for that sharing.

Ask both ISPs to announce only a default route. It is a practical approach for your scenario since this reduces the memory and CPU load on your Nexus 93180 switches, which are not designed to handle the full BGP routing table. Most ISPs can easily advertise the default route instead of the complete internet table. By receiving the default route from both ISPs, your devices can forward traffic to either ISP without maintaining a large routing table, simplifying the setup and reducing resource utilization...

Connecting both border leaf switches to different ISPs ensures redundancy in case of an ISP failure. If ISP-A or ISP-B goes down, the BGP session with the failed ISP will withdraw the default route, allowing traffic to flow through the remaining ISP. For effective failover, implement mechanisms such as BFD to detect failures quickly. Ensure that route preferences (via BGP atributes as LOCAL_PREF) are configured correctly to prioritize the preferred ISP when both are available. 

Also, one potential issue with dual ISP connections is asymmetric routing, where outgoing traffic uses one ISP, and return traffic arrives via another. This can cause problems with stateful devices like firewalls or NAT, which depend on consistent traffic flows. To address this, collaborate with your ISPs to ensure symmetric routing for your public IP addresses. You could use AS_PREPEND to force traffic to your ASN from one ISP...The one that send you a default route that your prioritize (LOCAL_PREF).

Your border leaf switches serve as the connection point between the ISPs and your EVPN fabric. It is essential to redistribute the default routes learned from the ISPs into the EVPN fabric while avoiding route loops. So, configure route tagging and apply policies to control how routes are advertised within the fabric. 

If you wish to utilize both ISPs simultaneously, you can configure equal cost multipath (ECMP) routing. Alternatively, as I said before, you can use BGP attributes like local preference or AS path prepending to influence the outbound traffic split between ISPs. While load balancing can optimize bandwidth utilization, achieving perfect symmetry for traffic flow is challenging and may require advanced routing configurations or coordination with ISPs to ensure optimal performance...

[satish.txt1]

Thank you so much for very detailed reply. I have follow up question. 

1. What If I plug both ISP and don't touch LOCAL_PERF or PRE_PENDING in that case what ISP it will pick? How does BGP will decide what circuit I should pick to route traffic out or in? 

2. Can I use Private ASN number between my Edge Routers and Border Leaf ? 

[M02@rt37]

You're so welcome @satish.txt1 

1. If you connect both ISPs to your network and leave BGP attributes like LOCAL_PREF or AS_PATH untouched, BGP will determine the best path based on its default path selection process...

-Outbound traffic will be influenced by the following criteria in order of preference:

• LOCAL_PREF: By default, the LOCAL_PREF attribute is typically set to 100 unless explicitly configured. If it is the same for both ISPs, the next criteria are evaluated.
• AS_PATH Length: BGP prefers the path with the shortest number of AS hops to the destination. For example, if ISP A advertises a route with 2 AS hops and ISP B advertises it with 3 AS hops, ISP A will be chosen.
• Multi-Exit Discriminator (MED): If MED values are advertised by the ISPs and accepted by your routers, the path with the lower MED will be preferred.
• eBGP over iBGP: If you are receiving routes via both external (eBGP) and internal (iBGP) neighbors, BGP will prioritize the eBGP path.
• Tiebreakers: If the above attributes are equal, additional criteria like router IDs or interface cost might come into play.

-Inbound traffic, however, is controlled by the ISPs. By default, each ISP will direct inbound traffic to your network based on their routing tables, and you have limited control over it unless you modify the AS_PATH or use mechanisms like BGP communities to influence the routing decisions of your upstream providers. Without any configuration changes, there is no guarantee that traffic distribution will be balanced across both ISPs.

2. If your edge routers are already using a public ASN to establish BGP sessions with the ISPs, using a private ASN for BGP peering between the edge routers and the border leaf switches would create issues. To make it easier use the same public ASN for BGP peering between the edge routers and the border leaf switches.

[satish.txt1]

I have question related ASN number between Border-leaf and edge-router. I am using EXT vrf on border-leaf to connect edge-router so in that case can I use private ASN number on border-leaf to peer edge router public ASN? 

For example this is my border-leaf BGP config. Interface e1/3 connected to Edge-router-1. Any issue if I use private ASN number 65535 to peer edge-1 using eBGP? 

interface Ethernet1/3
  description ** Connected to Edge-Router-1 **
  no switchport
  vrf member EXT
  ip address 1.1.1.1/30
  no shutdown


router bgp 65001
  router-id 10.255.1.3
  log-neighbor-changes
  template peer VXLAN_LEAF
    remote-as 65001
    update-source loopback0
    address-family ipv4 unicast
    address-family l2vpn evpn
      send-community
      send-community extended
      route-reflector-client
  neighbor 10.255.0.1
    inherit peer VXLAN_LEAF
    description ** iBGP Peer to spine-1 **
  vrf EXT
    log-neighbor-changes
    address-family ipv4 unicast
    neighbor 1.1.1.2
      remote-as 5678
      local-as 65535
      description ** Peer with Edge-Router-1 **
      address-family ipv4 unicast
        send-community
        send-community extended

. 

[M02@rt37]

Hello @satish.txt1 

Yes, th local-as configuration on your border-leaf ensures the edge router sees the private ASN 65535 in the AS_PATH, making this an eBGP relationship.

On edge routers, you have a public ASN in order to peer with ISPs ?

[satish.txt1]

Yes, I have public ASN number registered to peer with both ISP-A and ISP-B.  

I don't need to do anything on Edge Router related Default route I am going to received from ISP right? 

Assuming, that default route automatically populate in my EVPN fabric. 

[Flavio Miranda]

@satish.txt1 

Going to your question, yes, of course. You are already using eBGP on your Leaf, all you need to is stablish neighborship with both ISP. 

[MHM Cisco World]

please check this 

you can not use any SW as border GW

MHM 

[satish.txt1]

Hi, 

I don't know what you trying to say here with this diagram. What SW as border GW ? 

[MHM Cisco World]

Like campus' the vxlan use 

Leaf and spine and border GW' 

Border GW is use to connect your DC to other DC.

You mention about full bgp table linitation and I try to explain that only specific SW can work as border GW of vxlan.

MHM

[Joseph W. Doherty]

BTW, taking full Internet route tables, especially for a non ISP, is generally way overrated except in the case where one ISP can reach a destination but another cannot.  Think about the latter, if it happens, generally one ISP has a major issue.

So, just taking defaults, is likely "good enough", almost all the time, and if you want to go one step further, consider taking networks, from ISP, just one hop away.

A common consideration with what you want to do is whether your public IPs belong to your ISPs or to you.  Unsure obtaining you own public IPv4 IPs is still possible.  Who "owns" the public IPs you'll be using, matters a lot in how you setup the connecting topology.

[satish.txt1]

Thank so so much for reply folks. I have created more clear diagram so I can explain my config and get some input from experts. 

This is how my topo will look end of the day. 

1. eBGP peer between Edge <---> ISP

2. eBGP peer between Edge <---> Border-Leaf  (I am using 65355 to peer eBGP with Edge )

3. Inside EVPN/VXLAN fabric I have BGP 65001 

I am planning to do following config

[Edge]

router bgp 3500
  log-neighbor-changes
  neighbor 1.1.1.1
    remote-as 1500
    description ** eBGP peer with ISP-1 **
    address-family ipv4 unicast
      send-community
      send-community extended
  neighbor 192.168.1.1
    remote-as 65535
    address-family ipv4 unicast
      send-community
      send-community extended

[Border-Leaf]

router bgp 65001
  router-id 10.254.1.2
  log-neighbor-changes
  template peer VXLAN_SPINE
    remote-as 65001
    update-source loopback0
    address-family ipv4 unicast
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.254.0.1
    inherit peer VXLAN_SPINE
    description ** iBGP Peer to Spine-1 **
    no shutdown
  neighbor 10.254.0.2
    inherit peer VXLAN_SPINE
    description ** iBGP Peer to Spine-2 **
    no shutdown
  vrf CUST
    log-neighbor-changes
    address-family ipv4 unicast
      redistribute static route-map RTBH
      aggregate-address XX.XX.XX.0/23 summary-only
      aggregate-address XX.XX.XX.0/24 summary-only
    neighbor 192.168.1.2
      description ** eBGP Peer to Edge1 **
      remote-as 3500
      local-as 65535
      address-family ipv4 unicast
        send-community
        send-community extended

Question:

Q1. Both ISP-1 and ISP-2 sending BGP default routes (instead of full BGP). Assuming Edge will forward that default route to my EVPN fabric, right? 

Q2.Are there any issue using 65535 AS between br-leaf and edge router?  

Q3. Where do I apply local-preference or AS_PATH prepend to do IP shift to ISP-1 to ISP-2? 

[M02@rt37]

Hello @satish.txt1 

Thanks for that diagram.

Yes, edge routeurs will forward the default routes to the border Leafs, and you can further propagate that route into your EVPN fabric if needed.

Use as-path prepend in outbound from your edge routers towards the ISP you want to de-prioritize.

And, use local-pref on the Border Leaf in the evpn fabric, where you receive BGP routes from edge...

As concerned, asn 65535... technically it's allowed. 

[satish.txt1]

Thank you for the reply, 

Does as-path prepend works with default routes? I thought it only work with full BGP table.

Toward what direction I should put local-pref?  

I believe in following peer on Border-Leaf right?  route-map BGP-LOCAL-PREF-150 in 

neighbor 192.168.1.2
      description ** eBGP Peer to Edge1 **
      remote-as 3500
      local-as 65535
      address-family ipv4 unicast
        send-community
        send-community extended
        route-map BGP-LOCAL-PREF-150 in