Multiple BGP instances on n9k

[VK20]

Hello,

In a lab setup, I want to set up 2 bgp instances in 2 vrfs to test eBGP feature among those 2 instances.

I have one n9k is this possible to set up.

How would I leak routes between 2 vrfs? Is it possible to have 2 AS numbers on the same n9k or do I need 2 routers to test this setup.

Can someone provide me some config or some guidance?

Any feedback will be helpful.

[Giuseppe Larosa]

Hello @VK20 ,

a single device can be in a single BGP AS.

In the case of Nexus you can have VDC that is a form of partitioning .

However, there are tricks that can be used to pretend to be part of a different AS

router bgp 100

address-family ipv4 vrf VPNA

neighor xx.x.x local-as 65000

neighbor x.x.x.x remote-as 65001

redistribute connected

address-family ipv4 vrf VPNB

neighor xx.x.y local-as 65000

neighbor x.x.x.y remote-as 65001

redistribute connected

>> How would I leak routes between 2 vrfs?

If you want to really use an eBGP session you need to connect with a cable two interface one belonging to vrf VPNA and one belonging to vrf VPNB.

The two interfaces have to :

a) share a common IPv4 subnet to build the eBGP session

b) you need to ensure that they use different MAC addresses at OSI layer 2. You may need to change the MAC address used by one of them if you are using SVIs. Routed interfaces might work with no changes but again you need to check.

For sharing routes between different VRFs the more efficient and elegant is to play with route targets but in this case you would not set up an eBGP session

Hope to help

Giuseppe

[VK20]

I tried this ? but it seems 

address-family ipv4 vrf VPNA     dosen't work at all

vrf is not there , I only see multicast and unicast. this is nexus os 9.3

Can you provide me some more info?

[VK20]

Has anyone tried something like this:

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/xe-16/irg-xe-16-book/per-vrf-assignment-of-bgp-router-id.html 

[Giuseppe Larosa]

Hello @VK20 ,

I'm sorry I have given you incomplete information

you need to create the VRF objects before you can invoke them under MP BGP.

vrf definition VPNA

rd   65000:100

route-target import 65000:200

route-target export 65000:200

address-family ipv4 unicast

!

vrf definition VPNB

rd   65000:101

route-target import 65000:300

route-target export 65000:300

address-family ipv4 unicast

!

Hope to help

Giuseppe

[VK20]

Some more followup.

I am unable to establish eBGP session with peer it stays in an ideal state.

# sh ip bgp vrf 108 neighbors
BGP neighbor is (ipB), remote AS 65000, ebgp link, Peer index 3
BGP version 4, remote router ID 0.0.0.0
Neighbor previous state = Idle
BGP state = Idle (Missing address-family config), down for 02:15:51
Neighbor vrf: 108, retry in 00:01:19
Peer is directly attached, interface Ethernet1/46
External BGP peer might be up to 2 hops away
Last read never, hold time = 180, keepalive interval is 60 seconds
Last written never, keepalive timer not running
Received 0 messages, 0 notifications, 0 bytes in queue
Sent 0 messages, 0 notifications, 0(0) bytes in queue
Enhanced error processing: On
0 discarded attributes
Connections established 0, dropped 0
Connection attempts 105
Last reset by us never, due to No error
Last error length sent: 0
Reset error value sent: 0
Reset error sent major: 0 minor: 0
Notification data sent:
Last reset by peer never, due to No error
Last error length received: 0
Reset error value received 0
Reset error received major: 0 minor: 0
Notification data received:

Message statistics:
Sent Rcvd
Opens: 0 0
Notifications: 0 0
Updates: 0 0
Keepalives: 0 0
Route Refresh: 0 0
Capability: 0 0
Total: 0 0
Total bytes: 0 0
Bytes in queue: 0 0

For address family: IPv4 Unicast
BGP table version 2, neighbor version 0
0 accepted prefixes (0 paths), consuming 0 bytes of memory
0 received prefixes treated as withdrawn
0 sent prefixes (0 paths)
Last End-of-RIB sent 0.000000 after session start
First convergence 0.000000 after session start with 0 routes sent

No established BGP session with peer

Config : 

vrf context 107
rd 65000:100
address-family ipv4 unicast
route-target import 65000:200
route-target export 65000:200
vrf context 108
rd 65000:101
address-family ipv4 unicast
route-target import 65000:300
route-target export 65000:300

interface Ethernet1/45
no switchport
vrf member 107
ip address (ipA)/25
no shutdown

interface Ethernet1/46
no switchport
vrf member 108
ip address (ipB)/25
no shutdown

router bgp 100
address-family ipv4 unicast
vrf 107
local-as 65000
router-id (ipA)
address-family ipv4 unicast
neighbor (ipB)
remote-as 65001
ebgp-multihop 2
address-family ipv4 unicast
vrf 108
local-as 65001
router-id (ipB)
address-family ipv4 unicast
neighbor (ipA)
remote-as 65000
ebgp-multihop 2
address-family ipv4 unicast

I have configured the routed interfaces and have them connected directly.

ipA and ipB are not reachable as well??

The commands that you gave won't directly work on n9k so I did the best I could.

Are the misconfiguration ?? Anything I am missing? 

[Giuseppe Larosa]

Hello @VK20 ,

I realize you have a Nexus check if you need to enable features

>> Missing address-family config), down for 02:15:51

you may need a neighbor x.x.x.x activate in the address-family context as we are in MP BGP

feature bgp

feature vrf

check if the two interfaces are using different MAC addresses

show int eth1/45

show int eth1/46

if they have different MAC addresses they should be able to ping each other using

ping vrf 107  <ipB>

and

ping vrf 108 <ipA>

if they have the same MAC address for any reason you need to change it on one of them in interface configuration mode.

Hope to help

Giuseppe

[VK20]

Thank you for your response.

I did change the mac address they were the same. 

But still eBGP won't peer, Gives the same error. I have connectivity among the vrfs. 

vrf context 107
rd 65000:100
address-family ipv4 unicast
route-target import 65000:300
route-target export 65000:200
export map test
import map test
vrf context 108
rd 65000:101
address-family ipv4 unicast
route-target import 65000:200
route-target export 65000:300
export map test
import map test

router bgp 100
address-family ipv4 unicast
redistribute direct route-map test
redistribute static route-map test
vrf 107
local-as 65000
router-id <ipA>
address-family ipv4 unicast
redistribute direct route-map test
redistribute static route-map test
neighbor <ipB>
remote-as 65001
ebgp-multihop 2
address-family ipv4 unicast
vrf 108
local-as 65001
router-id <ipB>
address-family ipv4 unicast
redistribute direct route-map test
redistribute static route-map test
neighbor <ipA>
remote-as 65000
ebgp-multihop 2
address-family ipv4 unicast

[Giuseppe Larosa]

Hello @VK20 ,

a different MAC address is needed to have a working communication.

However, try the following

router bgp 100

vrf 107

address-family ipv4 unicast

neighbor <ipB>

activate

vrf 108

address-family ipv4 unicast

neighbor <ipA>

activate

Hope to help

Giuseppe

[VK20]

I am sorry active command is not there under bgp/vrf/address family

Also I changed the config a little to see if it changes anything.

I have made ipA and ipB in a different subnet. I do have connectivity between them

The message has changed a little 

BGP state = Idle (Connect failure), down for 23:22:15

Thank you so much for replying.

[Giuseppe Larosa · Accepted solution]

Hello @VK20 ,

>> I have made ipA and ipB in a different subnet. I do have connectivity between them

In this way it cannot work.

However, the error message is changed and now it says as we could expect

BGP state = Idle (Connect failure), down for 23:22:15

What version of NX-OS is running on your Nexus 9000?

It is high time to have a look at the correct configuration guide and/or command reference.

Please have IPA and IPB in the same IP subnet again as this is a basic requirement for what you want to achieve

Edit:

looking at NX OS 9.3 Configuration guide. It looks like you need an address-family ipv4 unicast statement under each neighbor

see

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/unicast/configuration/guide/b-cisco-nexus-9000-series-nx-os-unicast-routing-configuration-guide-93x/b-cisco-nexus-9000-series-nx-os-unicast-routing-configuration-guide-93x_cha...

Hope to help

Giuseppe

[VK20 · Accepted solution]

I was able to set up following your advice.

BGP neighbor is ipA, remote AS 65001, ebgp link, Peer index 3
BGP version 4, remote router ID ipB
Neighbor previous state = OpenConfirm
BGP state = Established, up for 00:01:56
Neighbor vrf: 107

Thank you so much. It feels pretty awesome that some remote on a community website will help someone multiple times, completely unknown.

This is just great. There is still lots of goodness left here

If it's ok I will like to ask few more questions.

How do I pass unicast traffic to this bgp setup from a Linux box or VM? DNS traffic. 

How do I add different routers to these instances? Just network command will add the network, Do I need to define routes in vrf or on the n9k level.

Here is a basic config that's running on n9k in future if someone wants to try, Those interfaces are in the same network and are connected directly.

vrf context 107
rd 65000:100
address-family ipv4 unicast
route-target import 65000:300
route-target export 65000:200
export map test
import map test
vrf context 108
rd 65000:101
address-family ipv4 unicast
route-target import 65000:200
route-target export 65000:300
export map test
import map test

interface Ethernet1/45
no switchport
mac-address 3c13.cc78.8e69
vrf member 107
ip address ipA
no shutdown

interface Ethernet1/46
no switchport
mac-address 3c13.cc78.8e67
vrf member 108
ip address ipB
no shutdown

router bgp 100
address-family ipv4 unicast
redistribute direct route-map test
redistribute static route-map test
vrf 107
local-as 65000
router-id ipA
address-family ipv4 unicast
redistribute direct route-map test
redistribute static route-map test
neighbor ipB
remote-as 65001
address-family ipv4 unicast
vrf 108
local-as 65001
router-id ipB
address-family ipv4 unicast
redistribute direct route-map test
redistribute static route-map test
neighbor ipA
remote-as 65000
ebgp-multihop 2
address-family ipv4 unicast

[Giuseppe Larosa]

Hello @VK20 ,

I am happy we have found the correct configuration for this device. And you have been kind in reporting the working configuration.

Let's see your further questions:

>> How do I pass unicast traffic to this bgp setup from a Linux box or VM? DNS traffic

You need to build and end to end L3 topology with at least one customer / user oriented VLAN/IP subnet in each VRF vrf 107 and vrf 108. These two subnets must be unique to be able to route across the link between the two VRFs.

>> How do I add different routers to these instances? Just network command will add the network, Do I need to define routes in vrf or on the n9k level.

At VRF level you can add either different eBGP neighbors or you can use an IGP like OSPF or EIGRP . In the second case you need to redistribute the IGP (OSPF/EIGRP) into BGP in the address family of the vrf and also BGP in the routing protocol mapped to VRF 107 or 108.

In any case these configurations need unique subnets in each VRFs for communications exactly as in previuos case. Also the prefixes advertised by other network devices in eBGP or IGP need to be unique and not overlapping.

Hope to help

Giuseppe

[VK20]

Hello,

If I may ask for a followup question.

I want to make router id of vrf 107 accessible to networks outside switch. Is this possible ? As any interface I try to configure with the vrf it has to be in routed mode and its not reachable outside. 

I tried to leake routes but that doesn't seem to work as my routes are from mgmt vrf that doesn't allow lots of commands.