Hi,

pan.systems · ‎12-10-2015

I have a 5525-X running 9.4.2 in multi context mode. In a test context I am configuring PBR - the topology is as follows -

The main routed path is learned via eBGP but I am selecting some Ingress traffic based on its source and pushing it to the PBR path via setting the next-hop.

Connections inbound Outside->Inside work, are policy routed correctly, which is verified by packet-tracer -

PBR-FW/TEST/pri/act# packet-tracer input TEST tcp 1.1.1.14 19100 8.8.8.2 5001 
 
Phase: 1 
Type: PBR-LOOKUP 
Subtype: policy-route 
Result: ALLOW 
Config: 
route-map TEST_PBR permit 10 
 match ip address BAHRAINx 
 set ip next-hop 192.168.8.1 
 set interface TEST_SITEA 
Additional Information: 
 Matched route-map TEST_PBR, sequence 10, permit 
 Found next-hop 192.168.8.1 using egress ifc TEST_SITEA 
 
Phase: 2 
Type: ACCESS-LIST 
Subtype: log 
Result: ALLOW 
Config: 
access-group VF-Peering-V10-IN in interface TEST 
access-list VF-Peering-V10-IN extended permit tcp any any eq 5001 
Additional Information: 
 
pbr: policy based route lookup called for 1.1.1.14/19100 to 8.8.8.2/5001 proto 6 sub_proto 0 received on interface TEST 
pbr: First matching rule from ACL(3) 
pbr: route map TEST_PBR, sequence 10, permit; proceed with policy routing 
pbr: evaluating next-hop 192.168.8.1 
pbr: policy based routing applied; egress_ifc = TEST_SITEA : next_hop = 192.168.8.1 
 
Phase: 3 
Type: NAT

Egress connections from the Inside->Outside between the same two endpoints fails. TCP as well as pings fail. So testing with ICMP for simplicity -

Dec 10 2015 09:45:21: %ASA-6-302020: Built outbound ICMP connection for faddr 1.1.1.14/0 gaddr 8.8.8.2/13974 laddr 8.8.8.2/13974 
Dec 10 2015 09:45:21: %ASA-6-110003: Routing failed to locate next hop for ICMP from TEST:1.1.1.14/0 to TEST_SITEA:8.8.8.2/13974 
Dec 10 2015 09:45:21: %ASA-6-302021: Teardown ICMP connection for faddr 1.1.1.14/0 gaddr 8.8.8.2/13974 laddr 8.8.8.2/13974

However packet-tracer says -

PBR-FW/TEST/pri/act# packet-tracer input TEST icmp 1.1.1.14 0 0 8.8.8.2 
 
Phase: 1 
Type: PBR-LOOKUP 
Subtype: policy-route 
Result: ALLOW 
Config: 
route-map TEST_PBR permit 10 
 match ip address BAHRAINx 
 set ip next-hop 192.168.8.1 
 set interface TEST_SITEA 
Additional Information: 
 Matched route-map TEST_PBR, sequence 10, permit 
 Found next-hop 192.168.8.1 using egress ifc TEST_SITEA 
 
pbr: policy based route lookup called for 1.1.1.14/0 to 8.8.8.2/0 proto 1 sub_proto 0 received on interface TEST 
pbr: First matching rule from ACL(3) 
pbr: route map TEST_PBR, sequence 10, permit; proceed with policy routing 
pbr: evaluating next-hop 192.168.8.1 
pbr: policy based routing applied; egress_ifc = TEST_SITEA : next_hop = 192.168.8.1 
 
Phase: 2 
Type: ACCESS-LIST 
Subtype: log 
Result: ALLOW 
Config: 
access-group VF-Peering-V10-IN in interface TEST 
access-list VF-Peering-V10-IN extended permit icmp any any 
Additional Information: 
 
Phase: 3

So the firewall appears to be dropping the ingress ICMP echo-replies because it can't find a route to the destination network on the interface to which PBR has sent it, even though PBR has provided it with a next hop already.

The whole point of PBR is that the route table DOES NOT have a route to the destination network via the next-hop that we are manually pushing the traffic to.

Packet tracer believes the packets should be forwarded.

Suggestions?

--------------------------------------

Update

A usable workaround is to place the PBR exit interfaces into a zone. The zone based interfaces are allowed up to 8 ECMP routes, so you can place a static summary route to your destination network via each zone member, leaving your more specific learned route to provide the primary routed path. The next hop will be correctly provided by PBR while the summary ECMP routes allow the destination check to complete.

Rishabh Seth · ‎12-10-2015

Hi,

Here is an explanation for the suslog message that you see:

110003

Error Message %ASA-6-110003: Routing failed to locate next-hop for protocol from src interface : src IP/src portto dest interface : dest IP/dest port

Explanation.An error occurred when the ASA tried to find the next hop on an interface routing table.

protocol —The protocol of the packet
src interface —The interface from which the packet was received
src IP —The source IP address of the packet
src port —The source port number
dest IP —The destination IP address of the packet
dest port —The destination port number

Recommended Action Copy the error message, the configuration, and any details about the events leading up to the error, and contact Cisco TAC. During debugging, use the show asp table routing command to view the routing table details.

Now going by the definition of the syslog, i think ASA is not able to find interface for the next-hop 192.168.8.1.

So you can start checking by pinging this next-hop IP from ASA. If it is not reachable then create route for this next-hop so that ASA can use it to find egress interface and ARP for next-hop.

Hope it helps.

Thanks,

Rishabh Seth

pan.systems · ‎12-10-2015

Hi Rishabh,

Thanks for the quick response.

192.168.8.0/29 is the directly connected network on interface TEST_SITEA. The firewall has a valid ARP entry for 192.168.8.1 on that interface, and the firewall can ping it.

When the ICMP packets are outside->inside they are are correctly Policy Routed. If the ICMP packets are outside->inside but are part of a response, i.e. state exists for the packets, PBR fails because of a route lookup for the destination network.

If I place a summary route to 8.0.0.0/8 pointing out the TEST_SITEA interface it all begins to work normally even though the longest match route for the destination is still via the routed path. This is not a work around however as you can't duplicate routes to different interfaces.

So it appears to be checking the policy routed packets destination network against the route table to make sure that the destination network is routed via the interface set by PBR as next-hop.

This should not be happening. By definition PBR is sending the packet to a next-hop that is not the normal routed path for the destination.

Cheers

Pete

Rishabh Seth · ‎12-10-2015

HI Pete,

So the issue is seen only for ICMP reply traffic. And based on your observation it looks like ASA

is treating the reply packet as a new packet and processing it using route lookup.

Can you check if you have ICMP inspection enabled? If not then enable ICMP inspection and check the behaviour.

Refer following link:

http://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/I-R/cmdref2/i2.html#pgfId-1760396

Thanks,

Rishabh Seth

pan.systems · ‎12-10-2015

Hi Rishabh,

It's happening with all return traffic, ICMP is just the simplest demonstration. ICMP inspection is enabled.

An inside->outside TCP connection dies when the SYN Ack from the outside host fails to get back to the inside host due to the route table lookup issue.

pan.systems · ‎12-11-2015

If I remove "inspect icmp" from my global policy (making it stateless) packets are PBRd as I expect.

Adding it back causes the return packets to be checked against the route table and dropped.

With inspect icmp -

Packet tracer shows correct next hop = yes

With real ping

Routing failed log message = yes

policy-route debug output = no

Packet is actually being routed, not PBRd

Without inspect icmp -

Packet tracer shows correct next hop = yes

With real ping

Routing failed log message = no

policy-route debug output = yes

Packet is being PBRd correctly.

So it appears that the same packet is treated differently depending on weather it has existing state. Packets that have no existing state are PBR's correctly.

Packets that have state are not PBRd even though packet-tracer says they should be.

pan.systems · ‎12-11-2015

OK, it would appear I have discovered this issue for myself - ASA: PBR policies should be applied for output route-lookup CSCuv00272 Cheers Pete

guibarati · ‎08-31-2016

If anybody comes here looking for a solution, I was having the same problem. ICMP would work but anything with connection state wouldn't.

What I had to do was to add a second default route, with a higher Administrative Distance to the interface where I wanted to send traffic with PBR.

This allowed the PBR to work and the secondary route wouldn't be installed in the routing table unless the primary interface was down.

ASA Policy Based Routing in a Context (9.4.2)

110003