MTU on ADSL line through VPN

[gsebk]

Hi, I use PIX501s with os6.3(3). They are connected to the internet through ADSL (pppoe). The firewalls are deployed in hub and spoke network in the center of which there is a VPN router 7140. Among the sites there are VPN site-to-site links. In all local LANs there are Windows domain controllers that must replicate with the central domain controller. It seems that all the spokes can ping the hub server and vica versa but the replication and the remote desktop connection doesn't work. The ADSL ISP told us to use 1454bytes as MTU on outside interfaces. If I set all the MTUs on the PIXs and the central 7140 to 1100 (that should be small enough for the ADSL) it doesn't help. But if I set the MTU size on the central Windows domain controller server (whith which all the spoke servers must replicate and that must be reached via remote desktop) everything works fine but this solution is not a nice one.

How can I tell the PIX and/or the VPN router to communicate the right MTU settings for the Windows servers? Or how can I set the MTUs on the CISCO devices that will work?

The configs are attached

[ehirsel]

I noted in your config that you adjusted the mtu of the pix's outside interface, but not the inside one where the remote domain controllers reside. You need to adjust the mtu on the inside, because that is where the pix will see the tcp mss negotiaion in the tcp syn packet that the domain controller will see.

On the pix's adjust the inside mtu: mtu inside 1454

Do this on all remote pix's. Also, do the same for the outside/adsl interface.

As far as the 7140 goes, go into interface config mode and you should be able to adjust its mtu size too. In fact by adjusting the mtu on the outside/adsl as well as the inside on the pix to 1454, you should not need to do anything on the 7104, because the remote pix will adjust the mss size on any packet generated by the central dc as it comes across the dsl interface.

If you make the mtu adjustment on the 7104, it may not take effect unless you also allow path mtu discovery, something that I do not recommend because you have to allow icmp unreachable messages to flow thru and these messages pertain to things other than path discovery.

Make the adjustments and let me know how it works.

[gsebk]

Hi, thanks for suggestion. I hav tryed to modify both the inside and outside MTU on the PIXs to the same 1454 and after it to 1100 but it doesn't work. The symptoms remained the same: the PINGs can get through but not the microsoft protocols (ie. RPC).

[ehirsel · Accepted solution]

What OS and what service pack level is running on your domain controllers?

If it is win 2000 sp 2 this link:

http://support.microsoft.com/default.aspx?scid=kb;en-us;301337

states that the server will ignore requests to lower the mtu when the request comes from a server (or gateway) on the same subnet as the windows host. SP3 addresses this issue.

I'll do some more searches on Microsoft's site and see what I come up with. It looks like your issue could be this, or something similar. On the other hand, the hub dc seems to be the one controlling the mtu size, so even though a remote client initiates a replication, the hub dc want to use its mtu, and not the one from the client, which should be lower since the pix should see the tcp mss negotiation and lower it according to its interface's mtu.

I'll post what I find.

[gsebk]

Hi,

the solution is that i have found a bug in the PIX OS:

http://www.cisco.com/cgi-bin/Support/Bugtool/onebug.pl?bugid=CSCec13051&cco_product=PIX+Firewall&fset=&swver=&keyw=icmp&target=&train=

In case of pppoe and VPN together the PIX doesn't sends back any ICMP unreachable packet to the sender when the packet is bigger than the MTU of the pppoe link. If you debug with the command: debug icmp trace, you can see that ICMP packets are generated by the PIX when the outgoing packets on the pppoe link are oversized. But when you try to catch these packets with a packet analyser, you simply find that they are not exist at all! Very nice.

The only solution in this case was that a registry entry must have been entered into the Microsoft OS:

EnablePMTUDiscovery=0 (DWORD) into the subfolder

HKEY_LOCAL_MACHINE\system\Currentcontrolset\services\tcpip\parameters

After this the OS won't send the traffic into a remote network with DF bit set but it will send the traffic with DF=0, so the firewalls can fragment them.

In normal case the OS use the DF=1 for packets toward remote networks but if it receives an ICMP unreachable with the maximum MTU from a gateway, the OS will reduce the packet size to the specified maximum MTU size but it uses henceforward the DF=1.