ASA 5510 'bounces' UDP packets. Why?

[fmaas]

Hi. I hope to find someone who can shed a light on something that is bugging me for days now. We have an ASA (5510, running 7.2(2)) to connect our subnets to the backbone of the ISP. I received complaints about one specific connection, which I'll "draw" here:

[ remote host 192.87.x.y (A) ] -- {"Internet"} -- [ Telecity Router ] -- [ Our ASA ] -- [ switch ] -- [ fibre ] -- [ switch ] -- [ our host 82.199.c.d (B) ]

What happens is the following: I can successfully ping from A to B and back. I can also traceroute (UDP) from A to B and back. But a specific UDP packet sent by A (port 9001) to B (port 5432) is causing a problem. "Our ASA" does not route the packet to the correct interface, but sends it back to the Telecity router instead. Which in turn sends it back to "Our ASA" (since it is destined for our subnet). This goes back and forth for 60 times and then the TTL is expired.

I have no idear what is happening here. The access-lists are correctly configured (as far as I know), but even if they weren't I would expect the packets to get dropped rather than put back on the sending interface. This packet is bounced (on ethernetlevel, the IP part remains the same apart from the TTL) to the sending router.

Any pointers as to where to look for what is causing this and how to investigate this further are highly appreciated.

Frank

[Federico Coto Fajardo]

Frank,

The ASA just do regular routing (not PBR or advanced routing decisions), so if you can PING from A to B and vice versa, the ASA has its routing table correct (i don't think is a routing issue).

If nothing seems to be causing this problem on the configuration on the ASA try to do a Packet Tracer test from the ASA.

You launch Packet Tracer from ASDM or CLI and simultate the packet coming from A to B (on those ports) and vice versa.

The ASA will show if there's any process on the ASA itself causing that behavior that you're seeing.

Let us know the results.

Federico.

[fmaas]

Hi Frederico. Of course I tried the packet tracer as one of the first tools :-). It showed a complete path (I used ASDM). Now I retried it (via the CLI) and something strange happens. This is a packet trace A to B with udp from port 9001 to 5431 (not 5432):

Phase: 1
Type: FLOW-LOOKUP / ALLOW / Found no matching flow, creating a new flow

Phase: 2
Type: ROUTE-LOOKUP / ALLOW / in   my_DMZ    255.255.255.240 my_DMZ

Phase: 3
Type: ACCESS-LIST / ALLOW / access-group my_backbone_access_in in interface my_backbone
                                                access-list my_backbone_access_in extended permit ip any my_DMZ 255.255.255.240

Phase: 4
Type: IP-OPTIONS / ALLOW

Phase: 5
Type: ACCESS-LIST / ALLOW / access-group my_DMZ_access_out out interface my_DMZ
                                                access-list my_DMZ_access_out extended permit ip any object-group host_group
                                                object-group network host_group
                                                network-object host myHost

Phase: 6
Type: IP-OPTIONS / ALLOW

Phase: 7
Type: FLOW-CREATION / ALLOW / New flow created with id 7342770, packet dispatched to next module

Phase: 8
Type: ROUTE-LOOKUP / ALLOW / found next-hop myHost using egress ifc my_DMZ
                                                    adjacency Active
                                                    next-hop mac address 00bb.ccdd.eeff hits 1

Result:
input-interface:   my_backbone / input-status: up / input-line-status: up
output-interface: my_DMZ / output-status: up / output-line-status: up

Action: allow

As you can see it is a complete path. But this is what happens if I try it for A/9001 to B/5432:

Phase: 1
Type: FLOW-LOOKUP / ALLOW / Found flow with id 12, using existing flow

Module information for forward flow: snp_fp_inspect_ip_options, snp_fp_adjacency, snp_fp_fragment, snp_ifc_stat

Module information for reverse flow: snp_fp_inspect_ip_options, snp_fp_adjacency, snp_fp_fragment, snp_ifc_stat

Result:
input-interface: my_backbone / input-status: up / input-line-status: up / Action: allow

Apparently the packet is send into an existing flow (is there a command to see what this flow is?) and that is where it ends. Now I need to find out why this is happening... This firewall has been reloaded a few times, but that did not solve the problem. Any pointers are highly appreciated as was this hint.

Frank

[fmaas]

Until now I came as far as

# sh conn 192.87.x.y detail long

...

UDP my_backbone:192.87.x.y/9001 (192.87.x.y/9001) my_backbone:myHost/5432 (myHost/5432)

...

Is there some way to give detailed information about this flow? Because I am very interested to know why a flow exists that routes the traffic back...

Frank

[Federico Coto Fajardo]

Frank,

Let's try to see what the problem is with this particular flow.
The connection is from A/9001 to B/5432
We've seen that the connection apparently is send into an existing flow.

I want to make sure of one thing...
Where do you see the packet is not being routed out the correct interface?
If you check the ''sh route'' for B, the packet is sent through that interface?

Federico.

[fmaas]

Frederico,

I did a capture (on the ASA) for traffic between A and B on the incoming and the outgoing interface. That showed that the packet was received on the incoming interface and immediately sent back to that interface. After your helpful hint to try the packet tracer (again) I dug a bit further and found that the existing flow seems to route traffic from the backbone back to the backbone (see my previous addition to this thread).

The routing table is correct, I just checked. And it should be as ping, telnet, ssh, http, everything is working flawlessly from the outside to the inside and back.

I have been looking for information on either showing more details about the existing flow or removing that existing flow, but no luck so far.

Your help is in any way greatly appreciated!

Frank

[Federico Coto Fajardo · Accepted solution]

Frank,

I honestly don't see why would the ASA reroute the traffic backout the same interface for this particular flow.
(if the routing table points to another interface for the destination IP address of the flow).
Anyway, we can try to clear the existing connection entry for this flow ''clear conn...'' and clear the local host ''clear local-host...''

Try this:
Before attempting that connection again (A/9001 to B/5432), make sure the ''sh conn'' does not show that connection entry no more.
Try again the connection and if it fails again, check if it's building again the same connection.

Note: On the ASAs, nothing better to follow the behavior of a flow than analysing the captures.

Federico.

[fmaas]

Frederico,

The problem is that I have no real control over host A. That host is continuously trying to send those UDP packets. So I executed the 'clear local-host' and that seems to have done the trick. It has apparently removed the wrong flow and (given the existing traffic) a new flow has been built that routes the traffic as it is supposed to.

Now the question remains how the wrong flow was created. Given that it had a very low 'id' I am thinking that this flow came to live when the router was not completely up and running. If at that moment the routing table was nog complete, the ASA could have thought that it could not route that traffic through. Why the flow remained existant, even when the remote host was restarted (and the traffic stopped for a short time), is a question as well. Perhaps this is simply a bug in 7.x?

In any case: the connection between A and B now seems OK. I'll keep an eye on this and try to remember the steps taken for the future.

Regards!
Frank

[Federico Coto Fajardo]

Frank you're correct.
The incorrect flow could have happened because of a failure at that particular moment.
The flows (translations and connections) have a timeout so they should live for as long as they are allowed to.

As you said, keep an eye on this because even if there was a failure, the ASA should not have created a flow backout the same interface
unless something in the configuration instructs it to do so.

Thank you,

Federico.

[fmaas]

Thanks for your information. The only reason I can think of is the absence of a complete routing table. Any rule or setting could make the packets being dropped, but not sent back - or am I wrong here. Do you know what the timeout for a flow is?

Frank

[Federico Coto Fajardo]

You can check your current timeout settings doing:

sh run timeout

It will show the timeout for the XLATEs, conn, half-open, udp, icmp, etc....

It seems strange to me that it was caused by an incomplete routing table because the ASA normally requires NAT as well to build the flow. So, if there's no NAT rule to send the traffic received on one interface backout the same interface, the flow won't be created even if there's a routing problem.

However, I could be wrong and you might have nat-control disabled or indeed have a NAT rule for this connection.

Federico.

[fmaas]

sh run 
timeout
It
 will show the timeout for the XLATEs, conn, half-open, udp, icmp, 
etc....

Timeout for UDP connections is 2 minutes... I know for sure that the appliances were never switched off for that long, so the flow was kept in place.

It
 seems strange to me that it was caused by an incomplete routing table 
because the ASA normally requires NAT as well to build the flow. So, if 
there's no NAT rule to send the traffic received on one interface 
backout the same interface, the flow won't be created even if there's a 
routing problem.
However, I could be wrong and you might have nat-control 
disabled or indeed have a NAT rule for this connection.

Does it help if I say that we do not use NAT ;-) We use 'real' IP's at the inside of our network. So the ASA is really only routing the traffic.

Frank

[Federico Coto Fajardo]

Hmmm...
Do you know for how long was the connection kept alive?
The connection will remain on the ASA as long as there are packets matching the flow... the timeout applies from the moment when the
connection is idle (to not keep the connections forever on the table)

Anyway, as you said it's a bit difficult to tell, please continue monitoring and let us know if something happens again :-)

Federico.

[d-fillmore]

I've had the same problem on my ASA, the connection entry showed the packet coming in and going back out the same interface. It's also a 5510, running 9.1 code. I reset the interface which fixed it temporariliy, but the traffic built up again, it sems the traffic is duplicated and just bounces between two firewalls in my case. My firewall shouldbe routing it out a dfifferent interface but doesn't, it routes it back towards the ASA it just came from.

I don't know the cause, but for me the fix was to uncheck the box "enable traffic between two or more hosts connected to the same interface" under the interfaces config section

[Jouni Forss]

Hi,

I imagine its possible that a NAT configuration configured in the wrong way might affect how the traffic is forwarded as NAT can override routing table choises in the new softwares.

The configuration change you did probably stop the traffic coming towards the firewall but will automatically drop it because now lacking the configuration command "same-security-traffic permit intra-interface"

- Jouni