ASA 5505 8.4(3) not responding to ARP requests from different subnet

[mika_cesim]

Hi,

We have a problem of setting up an ASA for our network's edge:

We have IP address

XX.150.85.91/255.255.255.0 setup for our ASA's outside interface. We have another IP-address in same block XX.150.85.85 setup for static nat for one of our internal servers. We have a static route setup via default gateway XX.150.85.254.

Everything is fine and working for the outside interface IP, but the natted IP does not work. It WORKS if we test it from host directly attached to the outside interface (using a switch), but it does not work further from internet. IP traffic does not get routed to its destination (the outside interface of ASA).

Proxy arp is enabled and we can see from the host directly attached to outside interface that the IP XX.150.85.85 resolves to outside interface's mac address as it is supposed to.

We have isolated the problem to the following using "debug arp" which outputs:

> arp-in: Arp packet received from XX.145.193.133 which is in different subnet than the connected interface XX.150.85.91/255.255.255.0

It seems that they want us to respond to an ARP request from a router in different subnet. This seems to be quite evil behaviour and I understand that ASA does not respond to the request by default. Is there a way to change this behavior of ASA? I think that operator is not willing to change their setup.

I suppose that this also means that our outgoing traffic is routed through the default gw XX.150.85.254, but incoming traffic comes directly from the next hop router revealed by "debug arp": XX.145.193.133.

-Mika

[mirober2]

Hi Mika,

The ASA's behavior in 8.4(3) was made more strict to no longer allow this behavior. The problem is technically a mis-configuration on the operator's end as they should be routing the traffic to your ASA's outside interface instead of trying to forward the traffic directly and send ARP requests for non-connected subnets.

There is currently no way to change this behavior in 8.4(3). If they are unwilling to make this change, the ASA will need to run 8.4(2) or lower.

-Mike

[mika_cesim]

Actually this case was solved by operator changing their config and us changing our IP-addresses.

Our previous setup was quite special. We had non-contiguous sequence of IPs assigned from /24-subnet like following:

XX.150.85.95

XX.150.85.102

XX.150.85.125

XX.150.85.127

Opeartor's gateway was in address:

XX.150.85.254

But it was not ARPing to us, but the next hop router XX.145.193.133 did. So the traffic was flowing (at least logically) different routes outbound/inbound.

Now they gave us a real /28-block of addresses and with this config the ARP-requsts are coming from the operator's gateway address in the same block.

BTW. quite evil from Cisco to change this kind of behavior in a minor ASA version update and not change it back, but of course the operator's config is quite evil too.

[BOFH-2012]

Hello there,

I was shocked when I upgraded one of my ASAs to 8.4(4.1) today and found out that most of the things didn't work anymore. By reverting back to 8.4(2) and upgrading step by step I narrowed the problem down and it started to occur with 8.4(3) due to the following changelog entry for 8.4(3):

Change: CSCts68268

Desc:     PIX-ASA: Route command should validate next hop IP before accepting

We are using multiple different subnets on our outside interfaces on many firewalls, this is common behaviour and every other major firewall vendor is not having any issues with that.

The reason why we use this is pretty simple: When you are operating Cisco ASA firewalls in a hosting environment and you start up small, you get a /29 for example, once you grow, you may be in the need for additional ip addresses and then the provider assigns another small subnet to you. Most of the time they cannot just extend your current subnet, because consecutive ip addresses are already given to another customer or it's just not possible to create a /28 out of your /29 because you're at the end of a /28 or something like this.

So with 8.4(2), we just created the route entry and set static arp alias entries for all the new additional ip addresses and everything was fine.

If this change is not going to be reverted again in the next version or if there is no way of doing things like that with Cisco ASAs anymore, we and many others, will have to switch the platform.

I do not have a desperate need to upgrade to something post 8.4(2) right now, just wanted to see if some of the minor issues we're encountering on our 100+ ASAs are fixed with the new version, but anything above 8.4(2) is useless to us now due to this stupid change.

Does anyone have a workaround for getting this done with 8.4(3) and above or can anyone from Cisco shed some light on the reason behind this change and tell us if it's going to be reverted back to the original behaviour again?

Thanks,

Alex

[CSCO11377298]

Hi Alexander,

I had the same issue as you last night trying to go to 8.4(4).

I think i will go to 8.4(2) and noted that works with static arp alias entries for static nat but could you clarify what route i need to add in addition?

I have a /30 on the outside interface with another /28 routed into that network by the ISP but to my knowledge not statically routed to my Outside interface ip:

Outside interface

xx.40.145.126/30

route Outside 0.0.0.0 0.0.0.0 xx.40.145.125 1

Additional range xx.84.92.64/28

static nat example:

object network man_exchange1

nat (inside,Outside) static xx.84.92.67

Cheers,

Dave

[BOFH-2012]

Hi David,

sure, no problem.

Usually, your provider has something like this set up:

interface VlanXXX

ip address 133.133.133.129 255.255.255.248 secondary

ip address 122.122.122.121 255.255.255.248

no ip redirects

end

So the main subnet is 122.122.122.120/29, the additional subnet you got is 133.133.133.128/29.

122.122.122.121 is the IP of your provider's gateway for you, so the usable ip addresses on your ASA are 122 - 126.

All these ip addresses will work right away, but none of the other ip addresses (133.133.133.130 - 133.133.133.134) will work for you just because of this issue.

All you have to do now is to add the following lines to the configuration of your ASA (assuming that 122.122.122.122 is the main ip of your ASA in the primary subnet):

route outside 133.133.133.128 255.255.255.248 122.122.122.122 1

Now the network is routed on your ASA and you need to add static ARP alias entries for all possible IPs in this subnet, that is:

arp outside 133.133.133.130 mac.of.outside alias

arp outside 133.133.133.131 mac.of.outside alias

arp outside 133.133.133.132 mac.of.outside alias

arp outside 133.133.133.133 mac.of.outside alias

arp outside 133.133.133.134 mac.of.outside alias

where mac.of.outside is the mac address of your outside interface on the ASA.

That's all that is needed and you're up and running with the new subnet.

Please note that your ASA does _NOT_ need an ip address in this network, so all the available IP addresses in this subnet are also available for your virtual services.

That's the basic configuration, you can then add port forwardings, static NAT rules or just protected services with these ip addresses on your ASA.

[CSCO11377298]

Hi Alexander,

Thanks fror you detailed reply above. I am actually going to go ahead with changing the way the ISP presents the range to us as it avoids the wasted IP and messy arp entries along with allowing more time next time i upgrade to troubleshoot something else which has been silently changed in a minor version.

FYI: I also have the same issue on new pair of 5515-X's running 8.6 so this appears to be standard behaviour now and in the future.

Can't wait for both these to be rectified, i will let you know the outcome.

Thanks again!

[Cristian Iconaru]

Hi Alexander,

shouldn't be the static route:

route outside 133.133.133.128 255.255.255.248 122.122.122.122 1

... configured on the ISP Router?

Thanks.

Regards

[aacole]

Hi,

Last night I also ran into this problem, upgrading from 8.4.1 to 8.4.3 ASA5540.

Customer has two public IP ranges allocated, one on the connected subnet on the outside of the ASA and another that doesnt have an IP address in the outside address range of the ASA. In my experience, whilst its not an ideal configuration its common practice, at least here in the UK.

Following the upgrade all connectivity to the directly connected public IP addresses worked, but connections to the non directly connected second range failed.

Checking ARP debug I saw a similar message to that posted above:

> arp-in: Arp packet received from XX.145.193.133 which is in  different subnet than the connected interface XX.150.85.91/255.255.255.0

Searching on the support community form I found this thread, which showed a workaround using a route and static ARP entries for the non directly connected subnet. I configured up a patch, and tested but it didnt work. As we had spent nearly 10 hours (!!) on this simple upgrade I didnt troubleshoot this any furthur, was loosing the will to live at this point and customer wanted the service back up.

Downgraded to 8.4.2, all worked as per the previous 8.4.1 release.

So, did the static ARP entries fix this problem, as the thread doesnt say. I'm not sure at this stage how to proceed, so would appreciate the experience of others who have run into this issue.

[rchockeelopez]

Hi,

Saturday night I had to do a rollback in the windows cause of this issue. The router had secondary address and the ASA with version 8.4.4(1) had static NAT to those subnets. I made a test today as aacole in lab with the static ARP on the ASA and it didn´t work.

I know that the feature can be disabled on version 8.4.4 (5) but that is an interim version so I would not like to ugprade to a beta version and then do an upgrade once again.

Let me know how you guys fix it up.

I will appreciate your help.

Thanks a lot,

Regards,

[CSCO11377298]

This is what i found to be the issue:

https://supportforums.cisco.com/docs/DOC-24549#comment-12588

Had the ISP change the way secondary subnets were presented to us and all worked fine on 8.4(4) with proxy arp

[Cristian Iconaru]

thanks.

it will work if the ISP removes the secondary IP and configures only the static route.

tried yesterday.

[michaellambert1980]

This new 'feature' was introduced in 8.4(3) but can be disabled in new ASA code 8.4(4.5)

See bug/enhancement:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCty95468&Submit=Search

[Cristian Iconaru]

Hi Mick,

thanks for the update.

how can you disable it?

Regards

[michaellambert1980]

Check the readme for 8.4(4.5). To disable, its under the arp configg where you define statics, etc. cant remember it exactly, but the readme will tell you. Im going to test on a 5585-x later today.