Nexus 9K iBGP

[billy_maclin]

I have 2 Nx9Ks each with a VRF to an Internet provider. I want the 2 Nexus VRFs to advertise/reflect the routes they learn from their connected ISP to the other VRF over iBGP. I have followed the NX-OS configuration guide and cannot get this to work.

VRF 1 is AT&T ***************************

interface Ethernet1/47
description << ATT Int LAN >>
vrf member ATT-Internet
ip flow monitor flowmon input
no ip redirects
ip address 209.36.218.97/28
ip address 12.49.166.145/29 secondary
ip address 12.49.166.148/29 secondary
ip address 12.133.188.2/24 secondary
ip address 64.132.122.3/24 secondary
hsrp 0
preempt
priority 250
ip 64.132.122.1
no shutdown


interface Ethernet1/48
description << ATT Int WAN >>
bandwidth 500000
vrf member ATT-Internet
ip flow monitor flowmon input
no ip redirects
ip address 32.142.144.46/30
no shutdown

router bgp 32300
address-family ipv4 unicast
vrf ATT-Internet
cluster-id 1000
address-family ipv4 unicast
network 12.49.166.144/29
network 12.133.188.0/24
network 64.132.122.0/24
network 209.36.218.96/28
neighbor 32.142.144.45
remote-as 7018
local-as 32300
address-family ipv4 unicast
neighbor 64.132.122.7
remote-as 32300
log-neighbor-changes
address-family ipv4 unicast
route-reflector-client
next-hop-self




********************************************


VRF 2 is Lumen - 64.132.122.7

interface Ethernet1/47
description << Lumen Int LAN >>
vrf member Lumen-Internet
no ip redirects
ip address 64.132.122.7/24
ip address 12.133.188.3/24 secondary
hsrp 0
ip 64.132.122.1
no shutdown

interface Ethernet1/48
description << Lumen Int WAN >>
speed 1000
duplex full
no negotiate auto
bandwidth 500000
vrf member Lumen-Internet
ip flow monitor flowmon input
no ip redirects
ip address 66.162.22.26/30
no shutdown

router bgp 32300
address-family ipv4 unicast
vrf Lumen-Internet
cluster-id 1000
address-family ipv4 unicast
network 12.133.188.0/24
network 64.132.122.0/24
neighbor 64.132.122.3
remote-as 32300
log-neighbor-changes
address-family ipv4 unicast
route-reflector-client
next-hop-self
neighbor 66.162.22.25
remote-as 3356
log-neighbor-changes
local-as 32300
address-family ipv4 unicast

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/213402-understand-next-hop-set-in-ibgp-advertis.html

NSK have special behave for

NEXT-HOP-SELF

 
please check the link above 

[billy_maclin]

Thanks for your reply. I'll test using the

route-map 

as shown in the article next week.

[Harold Ritter]

Hi @billy_maclin ,

Try with the following modified configs:

AT&T router:

router bgp 32300
vrf ATT-Internet
address-family ipv4 unicast
network 12.49.166.144/29
network 12.133.188.0/24
network 64.132.122.0/24
network 209.36.218.96/28
neighbor 32.142.144.45
remote-as 7018
address-family ipv4 unicast
neighbor 64.132.122.7
remote-as 32300
log-neighbor-changes
address-family ipv4 unicast
next-hop-self




Lumen Router:

router bgp 32300
vrf Lumen-Internet
address-family ipv4 unicast
network 12.133.188.0/24
network 64.132.122.0/24
neighbor 64.132.122.3
remote-as 32300
log-neighbor-changes
address-family ipv4 unicast
next-hop-self
neighbor 66.162.22.25
remote-as 3356
log-neighbor-changes
address-family ipv4 unicast

Regards,

[Harold Ritter]

Hi @billy_maclin ,

I would also suggest adding a filter to avoid advertising the routes received from AT&T to Lumen and vice versa.

AT&T router:

ip as-path access-list MyASNONLY seq 10 permit "^$"

router bgp 32300

vrf ATT-Internet

neighbor 32.142.144.45

address-family ipv4 unicast

filter-list MyASNONLY out




Lumen router:

ip as-path access-list MyASNONLY seq 10 permit "^$"

router bgp 32300

  vrf Lumen-Internet

    neighbor 66.162.22.25

      address-family ipv4 unicast

        filter-list MyASNONLY out

Regards,

[billy_maclin]

Hi Harold,


Thanks for your reply, however, you misunderstand what I'm trying to do. I WANT the 2 routers to advertise their respective received ISP routes to each other. In IOS and IOS-XE, this works without issue by specifying

 next-hop-self

on the iBGP peers. In NX-OS, the 2 routers only advertise the routes in their respective network statements.


In my scenario, the firewall has only a default route, so it forwards traffic to the AT&T router, who then just sends it out to the Internet. Traffic comes back on both carriers as it should because I'm advertising my public subnets to both carriers without any modification. But I want the AT&T router to learn about the Lumen routes, so that if Lumen has a better route to a subnet, the AT&T router will forward the traffic to the Lumen router which will then forward it to the Internet.


So I'm trying to send traffic outbound on whichever carrier is seen as the best route rather than sending everything out via AT&T. And I don't want my firewall to participate in BGP with my Internet routers.


Thanks.


Billy

[Harold Ritter]

Hi @billy_maclin ,

> Thanks for your reply, however, you misunderstand what I'm trying to do. I WANT the 2 routers to advertise their respective

> received ISP routes to each other. In IOS and IOS-XE, this works without issue by specifying

 next-hop-self 

on the iBGP peers.

I understood exactly what you are trying to achieve.

I tested the configuration I provided and it works as expected. AT&T advertises 192.168.100.1/32 and Lumen advertises 192.168.100.2/32. Here's a snapshot of both border routers:

Border1# sh bgp vrf ATT-Internet all
BGP routing table information for VRF ATT-Internet, address family IPv4 Unicast
BGP table version is 33, Local Router ID is 32.142.144.46
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

Network Next Hop Metric LocPrf Weight Path
*>l12.49.166.144/29 0.0.0.0 100 32768 i
* i12.133.188.0/24 64.132.122.7 100 0 i
*>l 0.0.0.0 100 32768 i
* i64.132.122.0/24 64.132.122.7 100 0 i
*>l 0.0.0.0 100 32768 i
*>e192.168.100.1/32 32.142.144.45 0 0 7018 i
*>i192.168.100.2/32 64.132.122.7 0 100 0 3356 i
*>l209.36.218.96/28 0.0.0.0 100 32768 i

Border1#




Border2# sh bgp vrf Lumen-Internet all 
BGP routing table information for VRF Lumen-Internet, address family IPv4 Unicast
BGP table version is 42, Local Router ID is 66.162.22.26
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

Network Next Hop Metric LocPrf Weight Path
*>i12.49.166.144/29 64.132.122.3 100 0 i
* i12.133.188.0/24 64.132.122.3 100 0 i
*>l 0.0.0.0 100 32768 i
* i64.132.122.0/24 64.132.122.3 100 0 i
*>l 0.0.0.0 100 32768 i
*>i192.168.100.1/32 64.132.122.3 0 100 0 7018 i
*>e192.168.100.2/32 66.162.22.25 0 0 3356 i
*>i209.36.218.96/28 64.132.122.3 100 0 i

Border2#

As you can see, both routers have the route from the opposite side.

Please let us know what does not work.

Regards,

[billy_maclin]

Sorry, Harold, but you had a comment yesterday saying "I would also suggest adding a filter to avoid advertising the routes received from AT&T to Lumen and vice versa." Maybe I misinterpreted your statement????


I want them to advertise their ISP learned routes to each other, but they will not.


In any case, I started with my routers configured exactly as your sample config, and they do not advertise any learned routes to each other. They advertise only the routes that are configured with network statements.

Here's my current config and a couple show commands:


AT&T

router bgp 32300
address-family ipv4 unicast
vrf ATT-Internet
address-family ipv4 unicast
network 12.49.166.144/29
network 12.133.188.0/24
network 64.132.122.0/24
network 209.36.218.96/28
neighbor 32.142.144.45
remote-as 7018
local-as 32300
address-family ipv4 unicast
neighbor 64.132.122.7
remote-as 32300
log-neighbor-changes
address-family ipv4 unicast
next-hop-self

RWay-Core-Nx93240-1# sh ip bgp sum vrf ATT-Internet
BGP summary information for VRF ATT-Internet, address family IPv4 Unicast
BGP router identifier 32.142.144.46, local AS number 32300
BGP table version is 778684, IPv4 Unicast config peers 2, capable peers 2
43601 network entries and 43603 paths using 11162120 bytes of memory
BGP attribute entries [5595/1029480], BGP AS path entries [4118/77578]
BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
32.142.144.45 4 7018 447857 52110 778684 0 0 5w1d 43597
64.132.122.7 4 32300 1259 1251 778684 0 0 00:06:16 2

RWay-Core-Nx93240-1# sh ip bgp neigh 64.132.122.7 routes vrf ATT-Internet

Peer 64.132.122.7 routes for address family IPv4 Unicast:
BGP table version is 778693, Local Router ID is 32.142.144.46
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

Network Next Hop Metric LocPrf Weight Path
* i12.133.188.0/24 64.132.122.7 100 0 i
* i64.132.122.0/24 64.132.122.7 100 0 i



Lumen

router bgp 32300
address-family ipv4 unicast
vrf Lumen-Internet
address-family ipv4 unicast
network 12.133.188.0/24
network 64.132.122.0/24
neighbor 64.132.122.3
remote-as 32300
log-neighbor-changes
address-family ipv4 unicast
next-hop-self
neighbor 66.162.22.25
remote-as 3356
log-neighbor-changes
local-as 32300
address-family ipv4 unicast

RWay-Core-Nx93240-2# sh ip bgp sum vrf Lumen-Internet
BGP summary information for VRF Lumen-Internet, address family IPv4 Unicast
BGP router identifier 64.132.122.7, local AS number 32300
BGP table version is 11421299, IPv4 Unicast config peers 2, capable peers 2
419210 network entries and 419212 paths using 107318024 bytes of memory
BGP attribute entries [85956/15815904], BGP AS path entries [63276/1490046]
BGP community entries [21119/1591276], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
64.132.122.3 4 32300 1264 1251 11421299 0 0 00:09:51 4
66.162.22.25 4 3356 3853460 85472 11421299 0 0 21:07:47 419205

RWay-Core-Nx93240-2# sh ip bgp neigh 64.132.122.3 routes vrf Lumen-Internet

Peer 64.132.122.3 routes for address family IPv4 Unicast:
BGP table version is 11421322, Local Router ID is 64.132.122.7
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

Network Next Hop Metric LocPrf Weight Path
*>i12.49.166.144/29 64.132.122.3 100 0 i
* i12.133.188.0/24 64.132.122.3 100 0 i
* i64.132.122.0/24 64.132.122.3 100 0 i
*>i209.36.218.96/28 64.132.122.3 100 0 i

Another commenter suggested there is a

route-map

that will fix the

next-hop-self 

issue in NX-OS and get the iBGP peers to act right, so I'm going to try that next. Here's that link: https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/213402-understand-next-hop-set-in-ibgp-advertis.html


Thanks very much for your help.

[Harold Ritter · Accepted solution]

Hi @billy_maclin ,

> Sorry, Harold, but you had a comment yesterday saying "I would also suggest adding a filter to avoid advertising the routes

> received from AT&T to Lumen and vice versa." Maybe I misinterpreted your statement????

This suggestion was about not advertising routes from AT&T to Lumen and vice versa, so that you don't become a transit AS. It was not about blocking routes between your border routers.

> Another commenter suggested there is a

route-map

that will fix the

 next-hop-self 

issue in NX-OS and get the iBGP peers to act

> right, so I'm going to try that next. Here's that link: https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/213402-understand-next-hop-set-in-ibgp-advertis.html

This is not required and will work just fine with a simple

next-hop-self

as the routes are received from eBGP and propagated to iBGP.

You need to remove the

 local-as 

command from the configuration. It is not required in your scenario and will prevent routes from being propagated between your border routers.

Regards,

[billy_maclin]

Removing the

local-as

command from the eBGP peer fixed it. As soon as I removed it from the Lumen eBGP peer, the Lumen router advertised all of the Lumen learned routes to the AT&T router via iBGP.

Harold, thank you again for your help. I'm learning to hate NX-OS a little less every day.

[Harold Ritter]

I am glad I could help @billy_maclin . Do not hesitate to let us know if there is anything else we can help with.

Have a great week end