Solved: Re: Can't establish connection between BGP neighbours

DStringfield · ‎04-20-2021

I have two geographically distinct sites, each with an ASA5506, that I am trying to connect via BGP. Please see the attached diagram below

My work so far has been succesful. Both ASA's have BGP enabled, and have Established connections to the common AS (the details of which were given to me by our provider). I next thing I did was advertise the network 192.168.170.0/24 from 64513. When I login to the other ASA (under 64514) and I can view the routes, I can see the following entry in the table:

Protocol | Type | Destination IP | Netmask       | Gateway    | Interface
BGP      |      | 192.168.170.0  | 255.255.255.0 | 10.252.0.9 |

When I view the BGP IPv4 Routes I see the following entry:

Status Code | Network       | Next Hop   | Metric | Local Pref | Weight | Path
*>          | 192.168.170.0 | 10.252.0.9 |                     | 0      | 2764 64513 i

Despite this, when I try and send ping or tracert requests from 192.168.36.XXX addresses to 192.168.170.0/24, I don't get any response. I'm trying to understand why, as I have the right routes in place, so I can't see why traffic isn't going through.

So far my thoughts are:

A misconfiguration for iBGP to eBGP. As we can see in the diagram this is an eBGP connection, but the route in the BGP Routes has an i for the origin code.
- I've ensured that Allow connections with neighbour that is not directly connected is checked in the BGP Neighbour settings
- I've tried at various points to add a second neighbour connection to each AS (directly across to the other AS) but I can't get that configured properly, and when I spoke to the provider about it they suggested this shouldn't be necessary (and I have the routes, so seems unecessary).
A security issue. On both ASA's the traffic is going between two interfaces (the inside interface, and an interface dedicated to BGP connections). Both interfaces have the same security level and the Enable traffic between two or more interfaces... is checked.
- An access rule is in place for both ASAs enabling all IP traffic on both interfaces.

I've tried everything I can think of. I can provide more configuration as needed, happy to answer more questions or take further reading recommendations to increase my knowledge on this topic.

Thanks!

Giuseppe Larosa · ‎04-23-2021

Hello David,

if your provider is an MPLS provider and it is giving you an MPLS L3 VPN, there can be an issue in the forwarding plane that they need to investigate a Label Switched Path in one direction may be not working correctly.

Hope to help

Giuseppe

View solution in original post

Giuseppe Larosa · ‎04-21-2021

Hello,

the devices are Firewall not routers !

>>Despite this, when I try and send ping or tracert requests from 192.168.36.XXX addresses to 192.168.170.0/24, I don't get any response. I'm trying to understand why, as I have the right routes in place, so I can't see why traffic isn't going through.

Then also the return path needs a route so you need to advertise subnet 192.168.36.0 in BGP as well from the other side that is in AS 64514.

In addition to this the outside ACL applied to each FW to the outside interface must allow traffic coming from the other site.

example of a line:

access-list outside extended permit ip 192.168.36.0 255.255.255.0 192.167.170.0 255.255.255.0

Hope to help

Giuseppe

DStringfield · ‎04-21-2021

Hi Giuseppe,

Thanks for your response. Okay I didn't have the other network advertised: that makes sense. I've now added it. As I said: I was fairly confident I had the correct ACL's in place, but I've gone ahead an added more for good measure. On both devices, for both relevant interfaces (inside & internal_routing) I've added rules allowing traffic in both directions for both networks:

show access-list internal_routing
access-list internal_routing; 6 elements; name hash: 0xff1e75c0
access-list internal_routing line 1 extended permit object-group DM_INLINE_PROTOCOL_5 192.168.33.0 255.255.255.0 192.168.170.0 255.255.255.0 (hitcnt=0) 0x22a03213
  access-list internal_routing line 1 extended permit ip 192.168.33.0 255.255.255.0 192.168.170.0 255.255.255.0 (hitcnt=0) 0x8358f5ea
  access-list internal_routing line 1 extended permit icmp 192.168.33.0 255.255.255.0 192.168.170.0 255.255.255.0 (hitcnt=0) 0x7c83c670
  access-list internal_routing line 1 extended permit icmp6 192.168.33.0 255.255.255.0 192.168.170.0 255.255.255.0 (hitcnt=0) 0xb5fcc511
access-list internal_routing line 2 extended permit object-group DM_INLINE_PROTOCOL_6 192.168.170.0 255.255.255.0 192.168.33.0 255.255.255.0 (hitcnt=0) 0x27c062c7
  access-list internal_routing line 2 extended permit ip 192.168.170.0 255.255.255.0 192.168.33.0 255.255.255.0 (hitcnt=0) 0x74803d9b
  access-list internal_routing line 2 extended permit icmp 192.168.170.0 255.255.255.0 192.168.33.0 255.255.255.0 (hitcnt=0) 0x31ad0d7b
  access-list internal_routing line 2 extended permit icmp6 192.168.170.0 255.255.255.0 192.168.33.0 255.255.255.0 (hitcnt=0) 0xe061016a

show access-list inside
access-list inside; 6 elements; name hash: 0x45467dcb
access-list inside line 1 extended permit object-group DM_INLINE_PROTOCOL_4 192.168.33.0 255.255.255.0 192.168.170.0 255.255.255.0 (hitcnt=0) 0xb10ac335
access-list inside line 1 extended permit ip 192.168.33.0 255.255.255.0 192.168.170.0 255.255.255.0 (hitcnt=0) 0x88e5cdd9
access-list inside line 1 extended permit icmp 192.168.33.0 255.255.255.0 192.168.170.0 255.255.255.0 (hitcnt=0) 0xeee90664
access-list inside line 1 extended permit icmp6 192.168.33.0 255.255.255.0 192.168.170.0 255.255.255.0 (hitcnt=0) 0x07e4c468
access-list inside line 2 extended permit object-group DM_INLINE_PROTOCOL_7 192.168.170.0 255.255.255.0 192.168.33.0 255.255.255.0 (hitcnt=0) 0x500493d4
access-list inside line 2 extended permit ip 192.168.170.0 255.255.255.0 192.168.33.0 255.255.255.0 (hitcnt=0) 0xfe205f06
access-list inside line 2 extended permit icmp 192.168.170.0 255.255.255.0 192.168.33.0 255.255.255.0 (hitcnt=0) 0x71a829d5
access-list inside line 2 extended permit icmp6 192.168.170.0 255.255.255.0 192.168.33.0 255.255.255.0 (hitcnt=0) 0x78a0f635

(Note, these rules are the same on both devices & the target network is not 192.168.33.0/24 unlike the in initial example).

Having done this and double checked the routes, I began running packet traces to see what was happening to the packets. I can confirm the following:

Packets sent from 192.168.170.0/24 reach the inside interface for 64513
Packets sent from 192.168.33.0/24 fail to reach the internal_routing interface for 64514

This is all tested using ping. My understanding is that because packets are being dropped at the internal_routing interface on the ASA for 64514, that echo responses cannot be sent from 192.168.33.0/24 back to 192.168.170/24.

At this point I tried doing captures with the asp_drop capture flag enabled on both devices. However I was unable to find any record of the packets being dropped. I tried a range of packet capture types but nothing seemed to give me more info about what is going on. Feel like I'm getting very close if I can just stop these packets being dropped. I'm happy to show configs or more captures as requested.

Thanks,

David

Giuseppe Larosa · ‎04-21-2021

Hello ,

>> On both devices, for both relevant interfaces (inside & internal_routing) I've added rules allowing traffic in both directions for both networks:

What we can note is that all lines in each ACL have hit count 0

From a device in subnet 192.168.33.0/24 from a shell you can try to do

telnet 192.168.170.X 80

if you have a web server on host 192.168.170.X

This is just to make a test with a protocol based on TCP instead of ICMP

It may be possible that you need to modify the global inspect policy for ICMP as suggested by Paul Driver.

Hope to help

Giuseppe

DStringfield · ‎04-22-2021

Hi Giuseppe,

Okay I performed those steps. From a device 192.168.33.91 I tried to telnet to 192.168.170.35 on port 80, but this was unsuccesful. I also tried this command from another host (one routed without BGP) and was able to connect no problem. I have entered the commands as suggested by Paul. Is there a way I can verify they registered correctly on the ASA?

Thanks,

David

Giuseppe Larosa · ‎04-22-2021

Hello ,

so TCP based connections work.

>> I have entered the commands as suggested by Paul. Is there a way I can verify they registered correctly on the ASA?

You can check the ASA config to confirm that the inspec icmp line is present under

policy-map global_policy
class inspection_default

At this point you should be able also to ping

Hope to help

Giuseppe

DStringfield · ‎04-22-2021

Hi again Giuseppe,

By showing the running-config on both devices I found the following (the icmp entry definitely appears in both):

policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect esmtp
  inspect sqlnet
  inspect skinny
  inspect sunrpc
  inspect xdmcp
  inspect sip
  inspect netbios
  inspect tftp
  inspect ip-options
  inspect snmp
  inspect icmp
!

I'm wondering if at this point I need to bring it up with my provider (who administers AS 2764) to see if they have misconfigured anything, or can provide some info about the packets being dropped. Is there anything else I can try to diagnose my problem further?

Thanks,

David

DStringfield · ‎04-22-2021

Hi again Guiseppe,

Sorry I realise my post before was vague. The telnet connection was not succesful, I still don't have connection.

Thanks!

David

Giuseppe Larosa · ‎04-22-2021

Hello David,

I apologize for my misunderstanding about TCP connection test result.

At this point from a host in 192.168.33.0/24 perform a tracert to 192.168.170.35 if your other site ASA is not reached the issue is inside the ISP and you should open a ticket with them.

Edit:

your packet-tracer tests on both ASA show that now they should allow packets coming from the other site.

Hope to help

Giuseppe

DStringfield · ‎04-22-2021

Hi again,

All good on the misunderstanding, I can see that my post was vague, and it was only after I edited it that it really conveyed what I intended.

I have done a tracert as you suggested, with no luck:

~>tracert 192.168.170.135

Tracing route to 192.168.170.135 over a maximum of 30 hops

  1    24 ms    36 ms     6 ms  10.252.0.9
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
etc......

I actually this morning already have raised a case with the ISP as I was so out of ideas, so hopefully that will turn something out. Always frustrating when you're so close to getting it right but the people you pay can't do their job

Thanks for your help so far, hopefully soon I'll be able to mark an answer.

Giuseppe Larosa · ‎04-23-2021

Hello David,

if your provider is an MPLS provider and it is giving you an MPLS L3 VPN, there can be an issue in the forwarding plane that they need to investigate a Label Switched Path in one direction may be not working correctly.

Hope to help

Giuseppe

paul driver · ‎04-21-2021

Hello
By default icmp is blocked from lower level interfaces to higher level interface, (icmp echo-reply) also icmp inspection is disable, so try enabling it gloably and test again.

ASA
policy-map global_policy
class inspection_default
inspect icmp

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

DStringfield · ‎04-21-2021

Thanks Paul. I've run this command on both devices.

paul driver · ‎04-21-2021

Hello

if you have bgp peering but cannot ping them it does seem it a icmp inspection issue

Can you perform a packet trace to see where it is failing and post the output:

packet trace input <inside interface> icmp <source ip><des ip>

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

DStringfield · ‎04-22-2021

Hi Paul.

I certainly can't ping nor other types of network connection. I made some slight modifications to the command as your initial one didn't work, based on further research, I hope this is correct.

Please find attached the output of each command. From 64514:

packet-tracer input inside icmp 192.168.170.35 8 0 192.168.33.91

Phase: 1
Type: INPUT-ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 10.252.0.1 using egress ifc  internal_routing

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit ip any any
Additional Information:

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp
service-policy global_policy global
Additional Information:

Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 50351298, packet dispatched to next module

Phase: 12
Type: INPUT-ROUTE-LOOKUP-FROM-OUTPUT-ROUTE-LOOKUP
Subtype: Resolve Preferred Egress interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 10.252.0.1 using egress ifc  internal_routing

Phase: 13
Type: ADJACENCY-LOOKUP
Subtype: Resolve Nexthop IP address to MAC
Result: ALLOW
Config:
Additional Information:
Found adjacency entry for Next-hop 10.252.0.1 on interface  internal_routing
Adjacency :Active
MAC address b026.8020.1286 hits 1919 reference 2

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: internal_routing
output-status: up
output-line-status: up
Action: allow

And from 64513

packet-tracer input GuestWireless icmp 192.168.33.91 8 0 192.168.170.135

Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: INPUT-ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 10.252.0.9 using egress ifc  internal_routing

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group GuestWireless_access_in in interface GuestWireless
access-list GuestWireless_access_in extended permit object-group DM_INLINE_PROTOCOL_8 any any
object-group protocol DM_INLINE_PROTOCOL_8
 protocol-object ip
 protocol-object icmp
 protocol-object icmp6
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp
service-policy global_policy global
Additional Information:

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 13
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 14
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 15
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 16664309, packet dispatched to next module

Phase: 16
Type: INPUT-ROUTE-LOOKUP-FROM-OUTPUT-ROUTE-LOOKUP
Subtype: Resolve Preferred Egress interface
Result: ALLOW
Config:
Additional Information:
Found next-hop 10.252.0.9 using egress ifc  internal_routing

Phase: 17
Type: ADJACENCY-LOOKUP
Subtype: Resolve Nexthop IP address to MAC
Result: ALLOW
Config:
Additional Information:
Found adjacency entry for Next-hop 10.252.0.9 on interface  internal_routing
Adjacency :Active
MAC address 0021.059c.59d2 hits 2813 reference 2

Result:
input-interface: GuestWireless
input-status: up
input-line-status: up
output-interface: internal_routing
output-status: up
output-line-status: up
Action: allow

Thanks!

David