Routing/BGP Issue

[jimmlegs]

I have an issue trying to route from my PC at 192.168.150.202 successfully to 62.10.10.66. The 192.168.150/24 and 192.168.68.0/24 networks are connected via cross-connected sites using the 172.30.254.0/29 network to communicate.

62.10.10.66 and 192.168.68.43 are two interfaces on the same server, the private IP has a BGP neighborship with 192.168.68.253 advertising the 62.10.10.66/28 network. This is what my router sees:

B 62.10.10.10 255.255.255.240
[20/1] via 192.168.68.43, 5w1d

A traceroute gets me to the private IP of the correct server but dies there. I do not think that the other router is aware of the path back but I have not been able to figure out how to fix this.

tracert 62.10.10.66

Tracing route to 62.10.10.66 over a maximum of 30 hops

1 4 ms * 3 ms 172.30.254.2
2 3 ms 3 ms 3 ms 192.168.68.43
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 * * * Request timed out.
8 * * * Request timed out.

BGP on the remote side looks like this:

Network Next Hop Metric LocPrf Weight Path
192.168.68.253 0 0 65518 i
* 172.30.0.0 192.168.68.253 0 0 65518 i
* 172.30.254.0/28 192.168.68.253 0 0 65518 i
* 192.168.150.0 192.168.68.253 0 0 65518 i

192.168.68.252 is an L3 switch and the default gateway on the 192.168.68.0/24 network. 192.168.68.253 is the router and gateway of last resort for the L3 switch. I've tried to explicitly add a route for 172.30.254.0/28 over the lan2-if but I get a message saying: "ERROR: Cannot add route, connected route exists"

Any help would be much appreciated.

Relevent Configurations:

interface GigabitEthernet1/2.68
vlan 68
nameif inside-68
security-level 100
ip address 192.168.68.253 255.255.255.0 standby 192.168.68.254

interface GigabitEthernet1/2.254
vlan 254
nameif lan2-if
security-level 100
ip address 172.30.254.4 255.255.255.240
!

router bgp 65518
bgp log-neighbor-changes
bgp router-id x.x.x.x
address-family ipv4 unicast
neighbor 192.168.68.42 remote-as 15518
neighbor 192.168.68.42 activate
neighbor 192.168.68.43 remote-as 15518
neighbor 192.168.68.43 activate
network 192.168.150.0
network 192.168.152.0
network 192.168.160.0
network 192.168.162.0
network 172.30.254.0 mask 255.255.255.240
auto-summary
synchronization
exit-address-family
!

route outside 0.0.0.0 0.0.0.0 x.x.x.x 1
route inside-68 62.10.10.66 255.255.255.255 192.168.68.43 1
route lan2-if 192.168.150.0 255.255.255.0 172.30.254.2 1
route lan2-if 192.168.152.0 255.255.255.0 172.30.254.2 1
route lan2-if 192.168.160.0 255.255.255.0 172.30.254.2 1
route lan2-if 192.168.162.0 255.255.255.0 172.30.254.2 1

[Jon Marshall]

Okay the OP has posted that on the next page ie the first output of

sh ip bgp 192.168.150.0

was from the firewall not the remote server. 

Looking at that output (the remote server) is not the best route the one with 192.168.68.253 as the

next hop ? 

Jon

[Harold Ritter]

Hi @jimmlegs ,

> I don't understand why 172.30.254.2 is showing as preferred in the

show bgp ipv4 uni 192.168.150.0

This is because the BGP path is originated from a static route with 172.30.254.2 as the

next hop

You can configure

neighbor 192.168.68.42 next-hop-self

on the FW side to advertise its peering address as the

next hop. 

>  "network 192.168.150.0" in BGP, I guess the subnet is implied?

Yes, the classful subnet (class C) is implied.

Regards,

[jimmlegs]

Thank you, I will give this a shot tonight after close of business.

[jimmlegs]

Being that 192.168.68.252 (switch IP) is the default gateway of the servers on the network, would "TCP State Bypass" be an option as a temporary work-around until the gateways can be updated to the point towards ASA and I can remove the IP from the switch?

Thanks

[Jon Marshall]

I was going to mention that as an option but didn't as it basically circumvents the firewall. 

It could be but first maybe have a look at the issue Harold raised as that could solve it for you. 

Jon

[jimmlegs]

Just for completeness I wanted to post the results of

sho bgp ipv4 uni 192.168.150.0

from the remote device

sho bgp ipv4 uni 192.168.150.0
BGP routing table entry for 192.168.150.0/24, version 840
Paths: (4 available, best #3, table default)
Multipath: eBGP
Advertised to update-groups:
2 
Refresh Epoch 1
65518
142.3.109.201 from 142.3.109.201 (142.2.107.169)
Origin IGP, metric 0, localpref 100, valid, internal
Community: 17063:298 17063:3590
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
65518, (received-only)
142.3.109.201 from 142.3.109.201 (142.2.107.169)
Origin IGP, metric 0, localpref 100, valid, internal
Community: 17063:298
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
65518
192.168.68.253 from 192.168.68.253 (149.197.17.6)
Origin IGP, metric 0, localpref 100, valid, external, best
Community: 17063:298
rx pathid: 0, tx pathid: 0x0
Refresh Epoch 1
65518, (received-only)
192.168.68.253 from 192.168.68.253 (149.197.17.6)
Origin IGP, metric 0, localpref 100, valid, external
rx pathid: 0, tx pathid: 0

Thanks again for all of the assistance, thus far.

[jimmlegs]

FIrstly, thank you for all of the assistance and guidance provided so far, this issue is beyond my ability and Cisco support is not helping much. The issue isn't yet fixed but I believe progress is being made.

Unfortunately adding

neighbor 192.168.68.43 next-hop-self

to the configuration did not do the trick. I ended up configuring tcp_bypass and I received a new log message which was previously not there:

%ASA-ip-4-313005: No matching connection for ICMP error message: icmp src inside-68:192.168.68.43 dst abel-noser-if:192.168.150.202 (type 11, code 0) on inside-68 interface.  Original IP payload: icmp src 192.168.150.202 dst 62.10.10.66 (type 8, code 0).

I ran a packet-tracer and it appears the packets are being evaluated against the TCP bypass ACL properly however the packets are still unable to make the return trip. Is there perhaps something more I need to route for the TCP bypass to function correctly? (Please note this TCP bypass is a temporary fix until we can decommission the L3 interface on the switch).

Here are the packet-tracer restults. I had also tried configuring some packet captures but they did not appear to contain much useful info.

packet-tracer input inside-68 tcp 192.168.150.202 $

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.68.43 using egress ifc  inside-68

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_out in interface inside-68
access-list inside_access_out extended permit ip any4 any4
Additional Information:

Phase: 4
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map tcp_bypass
 match access-list tcp_bypass
policy-map tcp_bypass_policy
 class tcp_bypass
  set connection timeout idle 0:10:00 embryonic 0:00:30 half-closed 0:10:00
        idle 0:10:00
        DCD: disabled, retry-interval 0:00:15, max-retries 5
        DCD: client-probe 0, server-probe 0, conn-expiration 0
  set connection advanced-options tcp-state-bypass
service-policy tcp_bypass_policy interface inside-68
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map tcp_bypass
 match access-list tcp_bypass
policy-map tcp_bypass_policy
 class tcp_bypass
  set connection timeout idle 0:10:00 embryonic 0:00:30 half-closed 0:10:00
        idle 0:10:00
        DCD: disabled, retry-interval 0:00:15, max-retries 5
        DCD: client-probe 0, server-probe 0, conn-expiration 0
  set connection advanced-options tcp-state-bypass
service-policy tcp_bypass_policy interface inside-68
Additional Information:

Phase: 9
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 22952626, packet dispatched to next module

Result:
input-interface: inside-68
input-status: up
input-line-status: up
output-interface: inside-68
output-status: up
output-line-status: up
Action: allow

As always, thank you.

[Jon Marshall · Accepted solution]

Not sure ICMP will work even with TCP bypass turned on. 

As a test can you not add a host route to the L3 switch for 62.10.10.66 pointing to 172.30.254.4 ie.  -

ip route 62.10.10.66 255.255.255.255 172.30.254.4

so the traffic is symmetric both ways. 

Jon

[jimmlegs]

Would TCP connections work? I'm not too concerned with ICMP.

Thank you. I will try this tonight after business hours. 

[Jon Marshall]

TCP connections should still work but it may be worth trying that route tonight just to see if that is the issue. 

Jon

[jimmlegs]

Thank you Jon! Once I added the route on the switch and applied the correct ACL to the interface I am able to communicate end-to-end as required.

I am forever greatful.

[Jon Marshall]

No problem, glad you got it working. 

Obviously that route is really just a short term solution, but at least you know what the issue is now. 

Jon