IPSec L2L VPN between IOS and PIX 6.3 - MTU issue?

[royalle01]

The remote (client) side is behind the PIX 6.3(5). And the head end (server) side is 2911 IOS on 15.0.

The IPSec tunnel comes up just fine and traffic passes. However, there is one server that cannot be fully accessed. Note, this is mainly web traffic.

Client initiates a connection to http://server:8000. They receive a redirect to go to http://server:8000/somepage.jspa. Packet caps show the client acknowledges the redirect with a SYN-ACK reply, but then the connection just hangs. And no other packets are received back. I've noticed that the redirected page is a .jsp and other pages that do work OK are not. I also noticed some MTU and TCP MSS configuations on the head end side that are in place for another GRE VPN tunnel with another site. So I was led down the packet fragmentation path. The PIX side has all the standard IPSec configurations along with the default MTU on the inside/outside interface.

When the MTU is manually set on the client workstation to 1400, access to http://server:8000/somepage.jspa works fine. So I need to tweak the PIX settings. I've tried adjusting MTU size on both inside and outside interface as well as setting 'sysopt connection tcp-mss'. I'm not sure what else to do here.

Here's a summary of the MTU settings on the head end:

Head End:

int tunnel0 (this is the GRE tunnel)

ip mtu 1420

tunnel source G0/0

dest X.X.X.X

tunnel path-mtu-discovery

crypto map vpn 1

description GRE tunnel

blah blah blah

crypto map vpn 2

description IPSec tunnel

blah blah blah

int g0/0 (external interface)

no ip redirects

no ip unreachables

no ip proxy-arp

ip verify unicast reverse

ip nat outside

ip virtual-reassembly

crypto map vpn

int g0/1 (this is the interface facing the server in question)

no ip redirects

no ip unreachables

no ip proxy-arp

  ip nat inside

ip virtual-reassembly

ip tcp adjust-mss 1452

[Marcin Latosiewicz]

If this is indeed a problem with fragmentation the MSS should be set on GRE interface.

I think the default on PIX (double check it) will be 1380.

You can try adjusting it to 1360. Or manaually discover PMTU and set MSS to IP MTU - 40 (assuming no IP options!) :-)

[royalle01]

So let me know if I have this straight.

I'll move the MSS to my tunnel interface on the router instead of the physical interface where the servers reside.
Then manually set the MSS on the PIX to 1360. Should I leave MTU on the PIX default at 1500 for inside/out?

Again, the problem is not over the GRE tunnel. This only happens on the IPSec VPN.

I didn't configure the GRE tunnel, but if the MTU on that interface is 1420, then MSS should be 1380 right?

[Marcin Latosiewicz · Accepted solution]

Ha, sorry my bad. Misread the previous post.

(side note: yes, MSS on tunnel interface should be 40 bytes less than MTU).

Don't tweak the MTU, not for TCP problems (not as the first step) , it's safer to play with MSS. Other things (OSPF for example) might depend on MTU.

Do a ping sweep with DF bit set with increasing size (starting from 1300 bytes for example). By doing this you will check what is the maximum size of packet you can ping through the IPsec tunnel. Once you have that value take it - substract 40 and set this as the MSS value of LAN interface (and adjust PIX's value if you can).

M.

[royalle01]

Hi Marcin and thanks for your answers.

The fix was to enable 'look ahead fragementation' on the head end router. Then I also matched the TCPMSS size on the far end PIX to be 1452, matching the inside interface that faces the servers being accessed.

So on the router:

crypto ipsec df-bit clear

crypto ipsec fragmentation before-encryption

[Marcin Latosiewicz]

well I'm glad you got it working.

However I would say it's workaround. Enabling fragentation on any level is going to decrease overall performance.

You should take this into consideration :-)