Cisco ASA IPsec, NAT bypass, PMTU-D issue

[bob.mcchesney]

Hello,

I am having an issue where ICMP "Unreachable, need to fragment" packets sent by the ASA are not delivered, apparently due to asymmetric NAT rules, when communicating over IPsec VPN. The device is a new Cisco ASA 5510 with Security Plus, running version 8.2(5).

The ASA is in a data centre, and I have an IPsec tunnel between it and a router in my office. The ASA has 'inside' and 'outside' interfaces, and dynamic NAT has been enabled for the inside subnet, and a NAT bypass access-list created to exclude traffic from inside to my office subnet. NAT and routing are all working fine (i.e. I can create connections between office subnet and 'inside' subnet, and also 'inside' can connect to the Internet and is correctly NATed).

Problems start when a device on 'inside' tries to send packets too large with DF bit set. For example, if I access a management website on a server, the connection is made correctly but then the server tries to send packets too large. The ASA should send an ICMP message "Unreachable, need to fragment", so that the server knows to resend with smaller packets, but it doesn't. However, if I turn off NAT, it does!

The following warnings are logged:

"PMTU-D packet 1420 bytes greater than effective mtu 1386, dest_addr=192.168.88.151, src_address=192.168.99.52, prot=TCP"

"Asymmetric NAT rules matched for forward and reverse flows; Connection for icmp src outside:192.168.99.1 dst inside:192.168.99.52 (type 3, code 4) denied due to NAT reverse path failure"

(192.168.88.151 is the querying PC, 192.168.99.52 is the server being queried, 192.168.99.1 is the ASA inside address.)

Does anyone know how to resolve this? I think it's a little odd that the warning shows "outside:192.168.99.1" as that's the inside IP address of the ASA.

My NAT configuration:

access-list inside-nat-bypass extended permit ip 192.168.99.0 255.255.255.0 192.168.88.0 255.255.255.0

global (outside) 101 interface

nat (inside) 0 access-list inside-nat-bypass

nat (inside) 101 192.168.99.0 255.255.255.0

Again, the above works (I can access the Internet from 'inside' and I can communicate fine between 'inside' and my office); it is just that the ICMP "need to fragment" packet doesn't get sent, and so PMTU-D doesn't work.

Finally, when I run a packet capture in ASDM, I can see the ASA is creating the packets, but they only appear on "egress: outside", not "ingress: management". However, when NAT is turned off, the packets appear in both. I believe therefore that the packets are failing a rpf-check, which is odd as they're from the ASA itself.

Suggestions very welcome! I can provide configuration if required.

Thanks,

Bob McChesney

[andrew.prince · Accepted solution]

Post the output of "show crypto ipsec df-bit inside"

Also perhaps try configuring "sysopt connection tcpmss 1300"

[bob.mcchesney]

Thanks, Andrew. Your suggestions helped a lot.

By setting "sysopt connection tcpmss 1300" the problem is definitely resolved; all TCP connections are limited to 1300 bytes MSS which is low enough for the IPsec overhead. However, I'm not very keen on that solution as it also reduces the MSS for non-IPsec Internet traffic which will be the majority of communications.

(Is there a way to adjust TCP MSS for specific traffic on an ASA?)

For "show crypto ipsec df-bit", both inside and outside were set to "copy". Setting "crypto ipsec df-bit clear-df outside" also resolves the issue, allowing fragmentation over IPsec. This feels like a better workaround to me as it will only impact the IPSec traffic.

Yet, I am a little disappointed that the ASA is not sending the "unreachable, must fragment" packet, as I believe it should. (And it has been demonstrated that when NAT is off, the packet is sent correctly.)

I can't dwell on this question for too long, but I would like to leave it open for a week in case further suggestions are made. If not, I'll mark your answer as correct.

Thanks,

Bob McChesney

[andrew.prince]

Bob,

The default tcpmss is all ASA's is 1380 - so you are only dropping 80 bytes, so to be honest you will not even notice it in the long run, but it's your choice.

I my experiance I have noticed that when dealing with "unreachable, must fragment" messages being recevied by Windows platforms, Windows actually ignores them.

Thanks for the marking - not important, but glad to have helped.

[bob.mcchesney]

Hello,

I did some more testing on this whenever I had time but I couldn't find an alternative solution that didn't involve changing the MSS of TCP handshakes. I'm marking your solution as the correct answer. Thank you very much for your help.

Regards,

Bob McChesney

[varrao]

Hi Bob,

Can you please share the running configuration from your firewall, i can re-collect a similar issue that I am working on, it might be a known issue.

Thanks,
Varun Rao
Security Team,
Cisco TAC

[bob.mcchesney]

Hello,

I'll gladly provide the configuration. I'll send you a private message with the configuration in a few minutes.

Regards,

Bob

[varrao]

Sure

[varrao · Accepted solution]

Hi Bob,

Its a Bug that you might be running into, the big id is -CSCti20726 you can view the bug details here:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti20726

You have a couple of options in it, either disable incpect icmp eror or upgarde the firewall version to 8.4.2 where it is resolved.

If you take captures on the inside and outside interfaces, you would see that the firewall is report the fragmentation error from the wrong interface, hence the ASA drops the packets which never reaches your server.

Hope this info would be useful to you.

Thanks,
Varun Rao
Security Team,
Cisco TAC

[bob.mcchesney]

Hello,

That is very useful thank you. The bug description you sent seems to match my problem exactly.

I am on version 8.2(5)26 and I do not have a support contract, but the unit is in warranty (purchased at the start of the year). Am I entitled to an updated image with the fix (you mentioned 8.4(2))? The bug description doesn't mention that it is resolved in any version.

Regards,

Bob McChesney

[varrao]

Hi Bob,

I am not sure whether you are entitled to download the software, you can try this link and see if it allows you to login:

http://www.cisco.com/cisco/psn/software/release.html?mdfid=279916854&flowid=4373&softwareid=280775065

It might not be mentioned on the site, maybe its not updated, but it has been resolved on the 8.4.2 version. You can also verify the bug by first disabling the icmp error from the policy and then test the traffic.

Thanks,
Varun Rao
Security Team,
Cisco TAC

[bob.mcchesney]

Hello,

I have confirmed that it is the same bug I was experiencing.

Thanks for sending the link to software images but I do not have access. I would need an active support contract for access to those downloads. However, as the unit is under warranty I believe I am entitled to upgrades that address deficiencies in the software. (For example I received an upgrade from 8.2(5) to 8.2(5)26 because of vulnerabilities.) I will refer to your correspondence here to indicate that 8.4.2 and above resolve the issue and I will attempt to get an update.

Regards,

Bob

[varrao]

Hey that's good news. Your issue was very familiar to one that I am currently been working on one of my TAC cases. Glad it helped you as well.

Cheers.

Thanks,
Varun Rao
Security Team,
Cisco TAC