topic Re: nexus 9k eBGP multi-hop problem in Routing and SD-WAN

nexus 9k eBGP multi-hop problem

rastislavpetro — Wed, 21 Jun 2023 14:00:16 GMT

Hi gyus,

I am trying to configure ebgp multihop session from nexus 9k, but the far end is not connected to this VxLAN fabric. I am using border-spine topology. Traffic is going through spine (ECMP) into our MPLS network (there is also my far end). I tried to configure SVI and Loopback as update-source with no success. The L3 connection looks good, but BGP peering is still down.

ping:

leaf2-dc# ping a.b.c.d source-interface loopback 10
PING a.b.c.d (a.b.c.d): 56 data bytes
64 bytes from a.b.c.d: icmp_seq=0 ttl=252 time=1.618 ms
64 bytes from a.b.c.d: icmp_seq=1 ttl=252 time=1.386 ms
64 bytes from a.b.c.d: icmp_seq=2 ttl=252 time=1.318 ms
64 bytes from a.b.c.d: icmp_seq=3 ttl=252 time=1.243 ms
64 bytes from a.b.c.d: icmp_seq=4 ttl=252 time=1.201 ms

--- a.b.c.d ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss

telnet to port 179:

leaf2-dc# telnet a.b.c.d 179 source loopback10 vrf inet
Trying a.b.c.d...
Connected to a.b.c.d.

Connfiguration on my nexus switch:

interface loopback10
description Loopback in vrf_inet
vrf member inet
ip address e.f.g.h/32

router bgp X
neighbor a.b.c.d
remote-as Y
update-source loopback10
disable-connected-check
ebgp-multihop 255
address-family ipv4 unicast
route-map drop in
route-map drop out

BGP logs from nexus:

2023 Jun 21 13:44:55.028199: (inet) EVT: Starting timer (60 sec 0 nsec) for a.b.c.d connection retry
2023 Jun 21 13:44:55.028173: (inet) EVT: a.b.c.d cleaning up active peer setup, thread id 0x0
2023 Jun 21 13:44:55.028167: (inet) EVT: a.b.c.d session setup (active) timed out, setup state Active busy 0
2023 Jun 21 13:44:55.018111: (inet) EVT: a.b.c.d Wait (0 sec) for session setup response
2023 Jun 21 13:44:55.018104: (inet) EVT: Peer a.b.c.d state Active TCP connection failed
2023 Jun 21 13:44:55.017471: (inet) EVT: a.b.c.d went from Idle to Active (Active setup)
2023 Jun 21 13:44:55.017283: (inet) EVT: a.b.c.d Triggered active open for peer
2023 Jun 21 13:44:55.017205: (inet) EVT: a.b.c.d peer connection retry timer expired
2023 Jun 21 13:44:55.017201: (inet) EVT: a.b.c.d remote iod 0 skip resolving l3 addr

Configuration from far end:

router bgp Y
neighbor e.f.g.h
remote-as X
ebgp-multihop 255
update-source Bundle-Ether15.1653
address-family ipv4 unicast
send-community-ebgp
route-policy drop in
route-policy drop out

ping:

RP/0/RSP0/CPU0:gw#ping e.f.g.h source Bundle-Ether15.1653
Wed Jun 21 15:33:56.792 CEST
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to e.f.g.h, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

telnet does not work - I am not sure why, because there is no firewall or ACL.

RP/0/RSP0/CPU0:gw#telnet e.f.g.h 179 source-interface Bundle-Ether15.1653
Trying e.f.g.h...
Use specified source interface(Bundle-Ether15.1653).
Use a.b.c.d as local address.
telnet: Unable to connect to remote host: Connection timed out

I can see incomming BGP session on far end, but I do not get reply on a nexus switch. I tried to configure ebgp-multihop session from my far end to another router (outside VxLAN fabric) and peering was up. So it means that configuration of my far end is ok.

Do you have any idea what is wrong with my setup?

Thank you. Rasto

Re: nexus 9k eBGP multi-hop problem

MHM Cisco World — Wed, 21 Jun 2023 14:21:14 GMT

router bgp Y
neighbor e.f.g.h
remote-as X
address-family ipv4 unicast

Neigbor x.x.x.x activate

Add this in both side

If not work then using it issue can from using LO in vrf but the bgp config which use LO as update source in global.

Check first solution then we discuss second one

Re: nexus 9k eBGP multi-hop problem

Harold Ritter — Wed, 21 Jun 2023 14:40:11 GMT

Hi @rastislavpetro ,

Your BGP configuration on the Nexus 9k is currently in the global context. It needs to be in the VRF inet context, as follow:

router bgp X

vrf inet
neighbor a.b.c.d
remote-as Y
update-source loopback10
disable-connected-check
ebgp-multihop 255
address-family ipv4 unicast
route-map drop in
route-map drop out

Regards,

Re: nexus 9k eBGP multi-hop problem

Harold Ritter — Wed, 21 Jun 2023 14:46:43 GMT

Hi @MHM Cisco World ,

The "neighbor x.x.x.x activate" is for IOS, not nxos.

Regards,

Re: nexus 9k eBGP multi-hop problem

rastislavpetro — Wed, 21 Jun 2023 15:11:05 GMT

Hi guys,

my mistake, I forgot to add vrf inet for BGP configuration im my first post. The correct and actual configuration is with "vrf inet". But peering is still down.

BR, Rasto

Re: nexus 9k eBGP multi-hop problem

MHM Cisco World — Wed, 21 Jun 2023 15:15:43 GMT

You need neighbor active under address family ipv4 vrf inet

Also you need

Peer-gateway and layer3 router-peer

In both nsk.

Re: nexus 9k eBGP multi-hop problem

Harold Ritter — Wed, 21 Jun 2023 16:37:36 GMT

Hi @rastislavpetro ,

You are missing another important part, which is the activation of the "address-family ipv4 unicast" under the vrf inet:

router bgp X

vrf inet

address-family ipv4 unicast

router bgp Y

address-family ipv4 unicast

Regards,

Re: nexus 9k eBGP multi-hop problem

Harold Ritter — Wed, 21 Jun 2023 17:23:36 GMT

Hi @rastislavpetro ,

Forget the last comment about activating "address-family ipv4 unicast" under the VRF. It is not required in nxos, unless you need to enable specific parameters. I was thinking about the IOS-XR behavior.

Can you provide more information, such as:

show bgp vrf inet ipv4 unicast summ

and

show bgp vrf inet ipv4 unicast neighbor a.b.c.d

and

sh sockets connection tcp | egrep 179

Regards,

Re: nexus 9k eBGP multi-hop problem

rastislavpetro — Thu, 22 Jun 2023 07:07:23 GMT

Hi @Harold Ritter ,

leaf2-dc# show bgp vrf inet ipv4 unicast summ
BGP summary information for VRF inet, address family IPv4 Unicast
BGP router identifier x.x.x.x, local AS number X
BGP table version is 907413, IPv4 Unicast config peers 14, capable peers 7
15595 network entries and 31028 paths using 1831232 bytes of memory
BGP attribute entries [1620/259200], BGP AS path entries [749/8934]
BGP community entries [43/1660], BGP clusterlist entries [10/40]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
a.a.a.a 4 A 3800473 3342737 907413 0 0 1w0d 1
a.a.a.b 4 B 3340594 3339423 907413 0 0 5w0d 2
a.a.a.c 4 B 3340681 3339403 907413 0 0 5w0d 2
a.a.a.d 4 C 1709564 1748734 907413 0 0 14w3d 4
a.a.a.e 4 D 268377 348332 907413 0 0 1w6d 0
a.a.a.f 4 F 124564 128657 907413 0 0 1w5d 0
a.b.c.d 4 Y 0 0 0 0 0 17:38:24 Idle
a.a.a.g 4 F 929541 929445 907413 0 0 4w5d 0

leaf2-dc# show bgp vrf inet ipv4 unicast neighbor a.b.c.d
BGP neighbor is a.b.c.d, remote AS X, ebgp link, Peer index 21
BGP version 4, remote router ID 0.0.0.0
BGP state = Idle, down for 17:38:58, retry in 00:00:13
Using loopback10 as update source for this peer
Connected check is disabled
Enable logging neighbor events
External BGP peer might be up to 255 hops away
Last read never, hold time = 30, keepalive interval is 10 seconds
Last written never, keepalive timer not running
Received 0 messages, 0 notifications, 0 bytes in queue
Sent 0 messages, 0 notifications, 0(0) bytes in queue
Connections established 0, dropped 0
Connection attempts 850
Last reset by us never, due to No error
Last reset by peer never, due to No error

Message statistics:
Sent Rcvd
Opens: 0 0
Notifications: 0 0
Updates: 0 0
Keepalives: 0 0
Route Refresh: 0 0
Capability: 0 0
Total: 0 0
Total bytes: 0 0
Bytes in queue: 0 0

For address family: IPv4 Unicast
BGP table version 907416, neighbor version 0
0 accepted prefixes (0 paths), consuming 0 bytes of memory
0 sent prefixes (0 paths)
Inbound route-map configured is drop, handle obtained
Outbound route-map configured is drop, handle obtained
Last End-of-RIB sent 0.000000 after session start
First convergence 0.000000 after session start with 0 routes sent

No established BGP session with peer

leaf2-dc# sh sockets connection tcp | egrep 179
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
LISTEN 0 40 :::179 :::*
LISTEN 0 40 *:179 *:*
ESTAB 0 0 a.a.a.a:179 a.a.b.a:33559
ESTAB 0 0 a.a.a.b:179 a.a.b.a:44234
ESTAB 0 0 a.a.a.c:27059 a.a.c.a:179
ESTAB 0 0 a.a.a.d:179 a.a.d.a:40242
ESTAB 0 0 g.g.g.a:50904 g.g.g.b:179 - peers in global routing table
ESTAB 0 0 g.g.g.a:35769 g.g.g.c:179 - peers in global routing table
ESTAB 0 0 a.a.a.e:42113 a.a.e.a:179
ESTAB 0 0 a.a.a.f:179 a.a.f.a:35034
ESTAB 0 0 a.a.a.g:179 a.a.g.a:53999

There is no line with ESTAB for peer a.b.c.d or e.f.g.h (from far end).

BR, Rastislav

Re: nexus 9k eBGP multi-hop problem

rastislavpetro — Thu, 22 Jun 2023 07:13:56 GMT

Hi @MHM Cisco World ,

my VPC configuration:
vpc domain 56
peer-switch
role priority 100
peer-keepalive destination 10.0.0.2 source 10.0.0.1 vrf PeerKeepAlive
delay restore 40
peer-gateway
layer3 peer-router
no layer3 peer-router syslog
auto-recovery
ip arp synchronize

But this configuration or mentioned commands do not have any impact on this BGP peering :-(.

BR, Rastislav

Re: nexus 9k eBGP multi-hop problem

MHM Cisco World — Fri, 23 Jun 2023 15:33:11 GMT

Re: nexus 9k eBGP multi-hop problem

MHM Cisco World — Fri, 23 Jun 2023 15:32:30 GMT

As @Harold Ritter mention there is no neighbor active.

I can run bgp' fact that I see issue in update but the bgp session is open'

What I do config vrf and config static route under vrf in each NSK for reachability of LO of other NSK.

Check ping and then config bgp as shown in attach photo

Re: nexus 9k eBGP multi-hop problem

Harold Ritter — Fri, 23 Jun 2023 01:05:09 GMT

Hi @rastislavpetro ,

The "sh sockets connection tcp | egrep 179" output looks really weird, with all of the duplicate sockets listening to port 179. It would definitely be worth getting TAC to look at it if possible, as your configuration looks good and it might be a software issue.

Regards,

Re: nexus 9k eBGP multi-hop problem

rastislavpetro — Tue, 27 Jun 2023 08:33:42 GMT

Hi guys,

I got information from TAC that this feature or design is unsupported according to Cisco internal documentation :(.

Thank you for your help.

BR, Rastislav

Re: nexus 9k eBGP multi-hop problem

MHM Cisco World — Tue, 27 Jun 2023 09:03:21 GMT

First thanks for update us,

Can you share topology you use

Thanks

MHM

Re: nexus 9k eBGP multi-hop problem

rastislavpetro — Tue, 27 Jun 2023 09:22:15 GMT

my topolgy and configuration of our spine switches

Re: nexus 9k eBGP multi-hop problem

MHM Cisco World — Tue, 27 Jun 2023 09:37:05 GMT

so it part from VXLAN, I will make check, but why you not config any GW in your VXLAN ?

Re: nexus 9k eBGP multi-hop problem

rastislavpetro — Tue, 27 Jun 2023 12:44:25 GMT

gateway is on my leaf switch which is part of my vxlan fabric. I try to configure ebgp multihop from this gateway's IP address (or loopback) to the remote site - IOSXR which is connected on Internet. This looks like that BGP session which should be terminated direct on nexus 9k switch and this traffic (session) was encapsulated as vxlan traffic is dropped somewhere.

telnet from the leaf switch to the remote router

leaf2-dc# telnet a.b.c.d 179 source loopback10 vrf inet
Trying a.b.c.d...
Connected to a.b.c.d.

telnet to the leaf switch:

and ping is OK in both directions.

Re: nexus 9k eBGP multi-hop problem

MHM Cisco World — Thu, 29 Jun 2023 09:41:20 GMT

why we called Node in VXLAN GW ?
GW in VXLAN is Node that have two address family,
l2vpn and IPv4 unicast,
in this Doc. in end there is one example about config GW in VXLAN
https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9500/software/release/17-6/configuration_guide/vxlan/b_176_bgp_evpn_vxlan_9500_cg/configuring_evpn_vxlan_integrated_routing_and_bridging.html