nexus 9k eBGP multi-hop problem

[rastislavpetro]

Hi gyus,

I am trying to configure ebgp multihop session from nexus 9k, but the far end is not connected to this VxLAN fabric. I am using border-spine topology. Traffic is going through spine (ECMP) into our MPLS network (there is also my far end). I tried to configure SVI and Loopback as update-source with no success.  The L3 connection looks good, but BGP peering is still down. 

ping:

leaf2-dc# ping a.b.c.d source-interface loopback 10
PING a.b.c.d (a.b.c.d): 56 data bytes
64 bytes from a.b.c.d: icmp_seq=0 ttl=252 time=1.618 ms
64 bytes from a.b.c.d: icmp_seq=1 ttl=252 time=1.386 ms
64 bytes from a.b.c.d: icmp_seq=2 ttl=252 time=1.318 ms
64 bytes from a.b.c.d: icmp_seq=3 ttl=252 time=1.243 ms
64 bytes from a.b.c.d: icmp_seq=4 ttl=252 time=1.201 ms

--- a.b.c.d ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss

telnet to port 179:

leaf2-dc# telnet a.b.c.d 179 source loopback10 vrf inet
Trying a.b.c.d...
Connected to a.b.c.d.

Connfiguration on my nexus switch:

interface loopback10
description Loopback in vrf_inet
vrf member inet
ip address e.f.g.h/32

router bgp X
neighbor a.b.c.d
remote-as Y
update-source loopback10
disable-connected-check
ebgp-multihop 255
address-family ipv4 unicast
route-map drop in
route-map drop out

BGP logs from nexus:

2023 Jun 21 13:44:55.028199: (inet) EVT: Starting timer (60 sec 0 nsec) for a.b.c.d connection retry
2023 Jun 21 13:44:55.028173: (inet) EVT: a.b.c.d cleaning up active peer setup, thread id 0x0
2023 Jun 21 13:44:55.028167: (inet) EVT: a.b.c.d session setup (active) timed out, setup state Active busy 0
2023 Jun 21 13:44:55.018111: (inet) EVT: a.b.c.d Wait (0 sec) for session setup response
2023 Jun 21 13:44:55.018104: (inet) EVT: Peer a.b.c.d state Active TCP connection failed
2023 Jun 21 13:44:55.017471: (inet) EVT: a.b.c.d went from Idle to Active (Active setup)
2023 Jun 21 13:44:55.017283: (inet) EVT: a.b.c.d Triggered active open for peer
2023 Jun 21 13:44:55.017205: (inet) EVT: a.b.c.d peer connection retry timer expired
2023 Jun 21 13:44:55.017201: (inet) EVT: a.b.c.d remote iod 0 skip resolving l3 addr

Configuration from far end:

router bgp Y
neighbor e.f.g.h
remote-as X
ebgp-multihop 255
update-source Bundle-Ether15.1653
address-family ipv4 unicast
send-community-ebgp
route-policy drop in
route-policy drop out

ping:

RP/0/RSP0/CPU0:gw#ping e.f.g.h source Bundle-Ether15.1653
Wed Jun 21 15:33:56.792 CEST
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to e.f.g.h, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

telnet does not work - I am not sure why, because there is no firewall or ACL.

RP/0/RSP0/CPU0:gw#telnet e.f.g.h 179 source-interface Bundle-Ether15.1653
Trying e.f.g.h...
Use specified source interface(Bundle-Ether15.1653).
Use a.b.c.d as local address.
telnet: Unable to connect to remote host: Connection timed out

I can see incomming BGP session on far end, but I do not get reply on a nexus switch. I tried to configure ebgp-multihop session from my far end to another router (outside VxLAN fabric) and peering was up. So it means that configuration of my far end is ok.

Do you have any idea what is wrong with my setup?

Thank you. Rasto

[MHM Cisco World]

router bgp Y
neighbor e.f.g.h
remote-as X
address-family ipv4 unicast 

Neigbor x.x.x.x activate 

Add this in both side 

If not work then using it issue can from using LO in vrf but the bgp config which use LO as update source in global.

Check first solution then we discuss second one 

[Harold Ritter]

Hi @MHM Cisco World ,

The "neighbor x.x.x.x activate" is for IOS, not nxos.

Regards,

[Harold Ritter]

Hi @rastislavpetro ,

Your BGP configuration on the Nexus 9k is currently in the global context. It needs to be in the VRF inet context, as follow:

router bgp X

vrf inet
neighbor a.b.c.d
remote-as Y
update-source loopback10
disable-connected-check
ebgp-multihop 255
address-family ipv4 unicast
route-map drop in
route-map drop out

Regards,

[rastislavpetro]

Hi guys,

my mistake, I forgot to add vrf inet for BGP configuration im my first post. The correct and actual configuration is with "vrf inet". But peering is still down.

BR, Rasto

[MHM Cisco World]

You need neighbor active under address family ipv4 vrf inet 

Also you need 

Peer-gateway and layer3 router-peer 

In both nsk.

[rastislavpetro]

Hi @MHM Cisco World ,

my VPC configuration:
vpc domain 56
 peer-switch
 role priority 100
 peer-keepalive destination 10.0.0.2 source 10.0.0.1 vrf PeerKeepAlive
 delay restore 40
 peer-gateway
 layer3 peer-router
 no layer3 peer-router syslog
 auto-recovery
 ip arp synchronize

But this configuration or mentioned commands do not have any impact on this BGP peering :-(.

BR, Rastislav

[Harold Ritter]

Hi @rastislavpetro ,

You are missing another important part, which is the activation of the "address-family ipv4 unicast" under the vrf inet:

router bgp X

vrf inet

address-family ipv4 unicast

router bgp Y

address-family ipv4 unicast

Regards,

[Harold Ritter]

Hi @rastislavpetro ,

Forget the last comment about activating "address-family ipv4 unicast" under the VRF. It is not required in nxos, unless you need to enable specific parameters. I was thinking about the IOS-XR behavior.

Can you provide more information, such as:

show bgp vrf inet ipv4 unicast summ

and

show bgp vrf inet ipv4 unicast neighbor a.b.c.d

and

sh sockets connection tcp | egrep 179

Regards,

[rastislavpetro]

Hi @Harold Ritter ,

leaf2-dc# show bgp vrf inet ipv4 unicast summ
BGP summary information for VRF inet, address family IPv4 Unicast
BGP router identifier x.x.x.x, local AS number X
BGP table version is 907413, IPv4 Unicast config peers 14, capable peers 7
15595 network entries and 31028 paths using 1831232 bytes of memory
BGP attribute entries [1620/259200], BGP AS path entries [749/8934]
BGP community entries [43/1660], BGP clusterlist entries [10/40]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
a.a.a.a 4 A 3800473 3342737 907413 0 0 1w0d 1
a.a.a.b 4 B 3340594 3339423 907413 0 0 5w0d 2
a.a.a.c 4 B 3340681 3339403 907413 0 0 5w0d 2
a.a.a.d 4 C 1709564 1748734 907413 0 0 14w3d 4
a.a.a.e 4 D 268377 348332 907413 0 0 1w6d 0
a.a.a.f 4 F 124564 128657 907413 0 0 1w5d 0
a.b.c.d 4 Y 0 0 0 0 0 17:38:24 Idle
a.a.a.g 4 F 929541 929445 907413 0 0 4w5d 0

leaf2-dc# show bgp vrf inet ipv4 unicast neighbor a.b.c.d
BGP neighbor is a.b.c.d, remote AS X, ebgp link, Peer index 21
BGP version 4, remote router ID 0.0.0.0
BGP state = Idle, down for 17:38:58, retry in 00:00:13
Using loopback10 as update source for this peer
Connected check is disabled
Enable logging neighbor events
External BGP peer might be up to 255 hops away
Last read never, hold time = 30, keepalive interval is 10 seconds
Last written never, keepalive timer not running
Received 0 messages, 0 notifications, 0 bytes in queue
Sent 0 messages, 0 notifications, 0(0) bytes in queue
Connections established 0, dropped 0
Connection attempts 850
Last reset by us never, due to No error
Last reset by peer never, due to No error

Message statistics:
Sent Rcvd
Opens: 0 0
Notifications: 0 0
Updates: 0 0
Keepalives: 0 0
Route Refresh: 0 0
Capability: 0 0
Total: 0 0
Total bytes: 0 0
Bytes in queue: 0 0

For address family: IPv4 Unicast
BGP table version 907416, neighbor version 0
0 accepted prefixes (0 paths), consuming 0 bytes of memory
0 sent prefixes (0 paths)
Inbound route-map configured is drop, handle obtained
Outbound route-map configured is drop, handle obtained
Last End-of-RIB sent 0.000000 after session start
First convergence 0.000000 after session start with 0 routes sent

No established BGP session with peer

leaf2-dc# sh sockets connection tcp | egrep 179
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
ESTAB 0 0 a.a.a.a:179 a.a.b.a:33559
ESTAB 0 0 a.a.a.b:179 a.a.b.a:44234
ESTAB 0 0 a.a.a.c:27059 a.a.c.a:179
ESTAB 0 0 a.a.a.d:179 a.a.d.a:40242
ESTAB 0 0 g.g.g.a:50904 g.g.g.b:179 - peers in global routing table
ESTAB 0 0 g.g.g.a:35769 g.g.g.c:179 - peers in global routing table
ESTAB 0 0 a.a.a.e:42113 a.a.e.a:179
ESTAB 0 0 a.a.a.f:179 a.a.f.a:35034
ESTAB 0 0 a.a.a.g:179 a.a.g.a:53999

There is no line with ESTAB for peer a.b.c.d or e.f.g.h (from far end). 

BR, Rastislav

[Harold Ritter]

Hi @rastislavpetro ,

The "sh sockets connection tcp | egrep 179" output looks really weird, with all of the duplicate sockets listening to port 179. It would definitely be worth getting TAC to look at it if possible, as your configuration looks good and it might be a software issue.

Regards,

[MHM Cisco World]

[MHM Cisco World]

As @Harold Ritter mention there is no neighbor active.

I can run bgp' fact that I see issue in update but the bgp session is open'

What I do config vrf and config static route under vrf in each NSK for reachability of LO of other NSK.

Check ping and then config bgp as shown in attach photo

[rastislavpetro]

Hi guys,

I got information from TAC that this feature or design is unsupported according to Cisco internal documentation :(.

Thank you for your help.

BR, Rastislav

[MHM Cisco World]

First thanks for update us, 

Can you share topology you use

Thanks 

MHM