Solved: Firewall Insertion 2nd BGP connection not coming up.

bunjiega · ‎10-31-2023

I am mocking up putting an HA-Firewall (FTD) pair into a fabric.
I am taking the eBGP approach, building a BGP session to the loopback of each VTEP. (see attached diagram)
There are no vPC devices.
The BGP Session to VTEP 3 comes up just fine, but the BGP session to VTEP 4 does not come up.
I can ping the loopback from the FTD, so I know I have reachability. But when I run debugs on the FTD and Nexus, it just looks like timeouts.
Wondering if anyone has done this or something similar or can see an obvious mistake.

FTD Config:

router bgp 65106
 bgp log-neighbor-changes
 bgp graceful-restart
 address-family ipv4 unicast
  neighbor 2.2.2.2 remote-as 65111
  neighbor 2.2.2.2 ebgp-multihop 5
  neighbor 2.2.2.2 ha-mode graceful-restart
  neighbor 2.2.2.2 activate
  neighbor 3.3.3.3 remote-as 65111
  neighbor 3.3.3.3 ebgp-multihop 5
  neighbor 3.3.3.3 ha-mode graceful-restart
  neighbor 3.3.3.3 activate
  network 192.168.1.0
  no auto-summary
  no synchronization
 exit-address-family
!
route inside 2.2.2.2 255.255.255.255 10.50.50.1 1
route inside 3.3.3.3 255.255.255.255 10.50.50.1 1
route inside 10.0.0.0 255.0.0.0 10.50.50.1 1
!
ASA# ping 3.3.3.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 10/24/80 ms
!
ASA# sh bgp ipv4 un sum
BGP router identifier 192.168.1.1, local AS number 65106
BGP table version is 5, main routing table version 5
3 network entries using 600 bytes of memory
3 path entries using 240 bytes of memory
3/3 BGP path/bestpath attribute entries using 624 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1512 total bytes of memory
BGP activity 5/2 prefixes, 11/8 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2.2.2.2         4        65111 1243    1120           5    0    0 20:38:20  2       
3.3.3.3         4        65111 0       0              1    0    0 never  Active

NX-4 Config (not working)

router bgp 65111
  router-id 10.2.0.2
  neighbor 10.2.0.3
    remote-as 65111
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf myvrf_50000
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map fabric-rmap-redist-subnet
      maximum-paths ibgp 2
    address-family ipv6 unicast
      advertise l2vpn evpn
      redistribute direct route-map fabric-rmap-redist-subnet
      maximum-paths ibgp 2
    neighbor 10.50.50.10                               !<--This is the FTD inside interface IP
      remote-as 65106
      update-source loopback3
      ebgp-multihop 5
      address-family ipv4 unicast
        send-community
        send-community extended
        route-map extcon-rmap-filter out
!
interface loopback3
  vrf member myvrf_50000
  ip address 3.3.3.3/32 tag 12345
!
NX-4# sh bgp ipv4 unicast summary vrf myvrf_50000 
BGP summary information for VRF myvrf_50000, address family IPv4 Unicast
BGP router identifier 3.3.3.3, local AS number 65111
BGP table version is 57, IPv4 Unicast config peers 1, capable peers 0
7 network entries and 8 paths using 1452 bytes of memory
BGP attribute entries [6/2112], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [2/8]

Neighbor        V    AS    MsgRcvd    MsgSent   TblVer  InQ OutQ Up/Down  State/
PfxRcd
10.50.50.10     4 65106         14         16        0    0    0 20:36:22 Active

NX-3 Config (working)

router bgp 65111
  router-id 10.2.0.4
  neighbor 10.2.0.3
    remote-as 65111
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf myvrf_50000
    address-family ipv4 unicast
      advertise l2vpn evpn
      redistribute direct route-map fabric-rmap-redist-subnet
      maximum-paths ibgp 2
    address-family ipv6 unicast
      advertise l2vpn evpn
      redistribute direct route-map fabric-rmap-redist-subnet
      maximum-paths ibgp 2
    neighbor 10.50.50.10
      remote-as 65106
      update-source loopback2
      ebgp-multihop 5
      address-family ipv4 unicast
        send-community
        send-community extended
        route-map extcon-rmap-filter out
!
interface loopback2
  vrf member myvrf_50000
  ip address 2.2.2.2/32 tag 12345
!
NX-3# sh bgp ipv4 un sum vrf myvrf_50000 
BGP summary information for VRF myvrf_50000, address family IPv4 Unicast
BGP router identifier 2.2.2.2, local AS number 65111
BGP table version is 39, IPv4 Unicast config peers 1, capable peers 1
7 network entries and 8 paths using 1228 bytes of memory
BGP attribute entries [6/1032], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [2/8]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.50.50.10     4 65106    1137    1262       39    0    0 20:39:50 1

Debugs from FTD

BGP: 3.3.3.3 open failed: Connection refused by remote host    !<-- This looks like the Nexus refusing
BGP: 3.3.3.3 Active open failed - tcb is not available, open active delayed 22528ms (35000ms max, 60% jitter)
BGP: ses global 3.3.3.3 (0x00007f6879c51530:0) act Reset (Active open failed).
BGP: 3.3.3.3 active went from Active to Idle
BGP: nbr global 3.3.3.3 Active open failed - open timer running
BGP: nbr global 3.3.3.3 Active open failed - open timer running
BGP: topo global:IPv4 Unicast:base Scanning routing tables
BGP: 3.3.3.3 active went from Idle to Active
BGP: 3.3.3.3 open active, local address 10.50.50.10sock_bgp_create: socket created successfully
sock_bgp_connect: connect issued
sock_bgp_util_notify_cb: NP_SOCK_CLOSED - Connect Failed
sock_bgp_util_wait_for_connect: Connect event received

Debugs from NX-4

2023 Oct 31 16:43:01.778173 bgp: libhmm  [22318] Get forwarding mode svi Vlan3000, phy_intf -
2023 Oct 31 16:43:01.778235 bgp: libhmm  [22318] Reading Forwarding mode for key: Vlan3000  from SDB
2023 Oct 31 16:43:01.778307 bgp: libhmm  [22318] hmm_get_fwd_mode(): Client bgp has not registered for mtype: 9, doing an SDB based lookup
2023 Oct 31 16:43:01.778365 bgp: libhmm  [22318] hmm_get_fwd_mode(): Client bgp has not registered for mtype: 18, doing an SDB based lookup
2023 Oct 31 16:43:01.778390 bgp: libhmm  [22318] Doing SDB based lookup for 2
2023 Oct 31 16:43:01.779056 bgp: libhmm  [22318] Reading Forwarding mode: Anycast Gateway from SDB
2023 Oct 31 16:43:01.779197 bgp: libhmm  [22318] Forwarding mode Anycast Gateway for svi Vlan3000, phy_intf -
2023 Oct 31 16:43:01.780569 bgp:  [22318] (myvrf_50000) PEER: 10.50.50.10 peer connection retry timer expired
2023 Oct 31 16:43:01.784069 bgp:  [22318] (myvrf_50000) PEER: 10.50.50.10 Triggered active open for peer
2023 Oct 31 16:43:01.784631 bgp:  [22318] (myvrf_50000) PEER: 10.50.50.10 went from Idle to Active (Active setup)
2023 Oct 31 16:43:01.784868 bgp:  [22318] (myvrf_50000) ADJ: bgp_tcp_connect: Peer 10.50.50.10 remote i/f Vlan3000 
2023 Oct 31 16:43:01.792672 bgp:  [22318] (myvrf_50000) ADJ: Local addr for peer 10.50.50.10 is 3.3.3.3 
2023 Oct 31 16:43:01.793321 bgp:  [22318] (myvrf_50000) ADJ: set_local_port: Peer 10.50.50.10, remote iod Vlan3000 
2023 Oct 31 16:43:01.807947 bgp:  [22318] (myvrf_50000) PEER: 10.50.50.10 Schedule wait for connect
2023 Oct 31 16:43:01.808249 bgp:  [22318] (myvrf_50000) EVT: 10.50.50.10 Wait (30 sec) for session setup response

From wireshark (on NX-3's uplink to the spine) I see 2-way traffic, even though there are tons of retransmissions (also attached)

Thanks!

bunjiega · ‎11-01-2023

Ended up getting it to work by going back and using 9.x Nexus code - no other changes. I was on 10.2.4. Maybe its fixed in 10.2.6? But feel better knowing it wasn't a config thing. Thanks for the input.

View solution in original post

f00z · ‎10-31-2023

Why is 2.2.2.2 and 3.3.3.3 routed to the same IP on the ASA? I'd think that 2.2.2.2 would be routed to one nexus and 3.3.3.3 to the other. If they are in VPC and have peer gateway set and you are using the 10.50.50.1 as the HSRP for example, this will fail unless nx3/4 have layer3 routes to each other. It's difficult to tell what the problem is, the only way would be to post the entire configs of the asas and the nexuses. The response could be going to the second ASA and it's not letting it get back to the first one if it's not synching state tables and routes.

Again without full configs it's hard to know. Wireshark on every link would tell where the problem is also

bunjiega · ‎10-31-2023

Why is 2.2.2.2 and 3.3.3.3 routed to the same IP on the ASA?
Because both destinations have the same next-hop.
These are not in vPC as mentioned in the original post.
I don't think the response is going to the secondary ASA because I can ping the remote loopback (3.3.3.3). This is an active/standby setup on the firewalls. Also I can see some bidirectional traffic in wireshark on NX-3's uplink.

f00z · ‎10-31-2023

Well you are trying to connect from ASA1 to NX4 , so what physical path is it taking? through the other asa, or through nx3 to nx1 to nx4 ?

bunjiega · ‎10-31-2023

Yes, ASA1->NX3->NX1->NX4
For simplicity you can ignore ASA2, that will only take over these BGP sessions during a failover scenario.

bunjiega · ‎11-01-2023

Ended up getting it to work by going back and using 9.x Nexus code - no other changes. I was on 10.2.4. Maybe its fixed in 10.2.6? But feel better knowing it wasn't a config thing. Thanks for the input.