Asymmetric Routing and Maximum Segment Size differences

[Matt Goli]

Hello,

Today I encountered a problem I had never come across before. We had a routing issue which went undetected that was causing asymmetric routing to happen.  Traffic from client to server was going over MPLS WAN, but traffic from server to CLient was going over an IPSec/GRE Tunnel over the internet.  The Client to server traffic had an MSS of 1460, but on the server to client the GRE tunnel had ip tcp adjust-mss 1360 set.

So a TCP connection would be initiated, a SYN packet sent from the client to the server with a MSS of 1460

Then the SYN/ACK would return with a MSS of 1360

the server did not know it would be returning the traffic over the IPsec/GRE tunnel because the asymetric routing was at the network layer.

So now a 1500 byte packet is sent from the server to the client with the Don't Fragment bit set, and the packet never reaches the client.

So is there a solution to prevent this type of thing from happening in the future?  its technically ok for there to be asymetric routing, not ideal in our case, but it shouldn't break an application from working just cause the paths have different MSS sizes.  Is there a technical solution to prevent such a thing from occurring in the future other than setting the MSS on the WAN link towards the MPLS to a lower value?

[Mark Malone]

Just a quick thought on the server side entry into the router use pbr create a route-map with extended access-list any traffic with a source of the server and destination of the client address set the next-hop to be the mpls pe interface to make sure it goes that path

Otherwise you could  manipulate the routing protocol itself depending on whats in place

[xthuijs]

you could also opt for the config of ignoring the DF bit and allow fragmentation to happen, or depending on platform capability, use an MSS adjust feature on the MPLS PE to intercept the packet and rewrite the MSS value (in hardware).

assymetric routing is fact of life, not ideal, but we can make this work depending on what you can "sacrifice".

ignore DF and fragment is one option, but some firewalls dont like fragments. adjusting mss on a platform requires a capable device to be able to do that.

regards

xander

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

My understanding of a typical TCP handshake, both hosts should use the lower of the two hosts MSS.  So, you might want to investigate further if that's not happening.  (Actually, that's the whole point of the adjust-mss spoof, to get hosts to use an optimal MSS from the get go.)

Regardless, the other issue you've described, sending hosts don't appear to be receiving notifications when DF bit is set and fragmentation is needed.  That needs to be investigated and corrected.  (Even if you has symmetrical routing, consider an open flow failing over from MPLS to the tunnel, i.e. the available MTU would decrease.  This shouldn't break anything.)

[Evgeny Taskaev]

Hello,

I would suggest to set MTU on tunnel interface to 1500 ("ip mtu 1500") without "tunnel path-mtu-discovery".
This should prevent dropping large packets with DF-bit, which enters to GRE tunnel.
Fragmentation will be occur of course, but it less evil, than dropping packet at all.

[Peter Koltl]

Have you troubleshooted it, Matt?

[ahmad82pkn]

Matt did you resolve the issue? if yes can you please share how? i am facing same issue. only option i have is to fix by mtu but what values, still not sure. 

please help :)