Possible MTU issue when running in 3G failover on Cisco887 VAG router?

[mitchen]

We have several sites with Cisco887VAG routers connecting to our datacentres over IPSEC VPN.  The routers primarily connect via ADSL but if there is a problem with the ADSL, then they automatically fail over to 3G.  This has all been working nicely…. Or so we thought!  We have had a few seemingly random sites that reported issues when they were running on 3G.  The strange thing was that most of their connectivity seemed fine (internet access, which they break out locally for seemed ok, most internal stuff – our helpdesk could login to their PCs remotely etc – however, the primary application that they use in our datacentres, a web based app behind a loadbalancer, refused to load which was obviously a problem!)

We puzzled over this for a while (usually the ADSL would come back up again and all would be ok before we had got too far with the troubleshooting!)

When it happened again, we discovered that if we changed from using the loadbalanced address for the web app to using the physical address of the web server behind the web app, then things seemed to work again (not sure if the loadbalancer perhaps adds some additional packet overhead?).  We also found that apparently a few external websites would not download completely.  This made me start to wonder if it was an MTU issue so I carried out some further tests.

Our router configuration is configured with ip mtu 1492 on both dialer interfaces (I appreciate we probably only need to have it as 1492 for the ADSL connection due to the additional 8 byte overhead of PPPoE but any how!)   We also have the VLAN interface (for our connected PCs) configured with ip tcp adjust-mss 1452

After testing a few sites, I found that the MTU that we could send on a 3G connection was variable – and that the sites that experienced problems were ones that were only capable of sending a smaller MTU.  E.g. at a site with no issue when running on 3G we can see that the full MTU of 1492 can be sent:

ping 8.8.8.8 size 1492 df-bit

Type escape sequence to abort.

Sending 5, 1492-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:

Packet sent with the DF bit set

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 108/296/1040 ms

!

At a site which has been experiencing the issue, we can see that the MTU is only 1468.

ping 8.8.8.8 size 1468 df-bit

Type escape sequence to abort.

Sending 5, 1468-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:

Packet sent with the DF bit set

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 120/133/148 ms

ping 8.8.8.8 size 1469 df-bit

Type escape sequence to abort.

Sending 5, 1469-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:

Packet sent with the DF bit set

.....

So, I wondered  a) if I’m likely to be on the correct lines by suspecting this is an MTU issue when running on 3G and b) if so, what I can do to try to address it?  I tried lowering the ip tcp adjust-mss (took it down to 1400) but this didn’t seem to make any difference.   Should I also be lowering the MTU on the dialer interface corresponding to the 3G connection? (I had kept it at 1492, despite lowering the MSS)  I think I'm starting to get myself confused over where I should be changing MTU/MSS values!

Can anyone advise on what is the best way of establishing exactly what I should have for my MTU and MSS settings? (although I would obviously much prefer to have a standard which can be applied across all/most sites rather than have to determine the specific MTU on each 3G connection and tailor accordingly?)

Thanks.

[Mohamed Sobair]

Mitchen,

As you done the testing already and experiment on this behaviour, allow me to elaborate on one possible cause you have not stated.

Your Sites are connecting to your DC over an IPSec tunnels using ADSL or 3G connections.

Please note of the following overheads that should be considered over the current links which can negatively result in drops.

The IPSec itself adds additional (53 Bytes) header to the Original packets.

And the PPP adds the (8 Byte) header as well to the Original packets.

The Total overhead for the Original packets would be 61 bytes.

Accordingly, I would recommend you to set the dialer interfaces to some thing like 1500-61 = 1439, apply this to your both dialer interfaces, although the 3G connection doesn't incurred the additional 8 byte PPP header.

On the Other hand, the TCP maximum Segment Size (MSS) should be tailored to 1399. this is because of the TCP header of 40 bytes lower than the modified MTU , and this should be applied on all branch LAN interface.

I believe this should solve your issue, and eliminate the performance degradation you are facing.

HTH

Mohamed

[mitchen]

Hi Mohamed,

Thanks very much for the advice, I think perhaps you are right and I need to allow for the IPSEC overhead so this is probably something that needs to be modified from my original settings anyway!

However, while I understand your reasoning based on an original MTU of 1500, I was wondering, in the cases where for some reason the MTU on certain 3G connections is actually LESS than 1500 to start off with (e.g. in one of the examples I gave the MTU is only 1468) does this mean I should then lower my MTU settings further. i.e:

1468 – 61 (IPSEC of 53 bytes plus 8 bytes PPP) = 1407

1407 – 40 (40 bytes TCP) = 1367

And, in such a case, I should set the MTU on both dialer interfaces to 1407 and the MSS on the LAN segment as 1367?

If that is correct, do you think it would be best practise to simply look at any 3G connections which have a problem due to a lower MTU and adjust settings specific to those branches accordingly?

Or would it be best to find the common settings which will work across the board so that each branch can have a standard configuration template (as they do now)?

I’m not sure the performance impact of lowering the MTU/MSS settings on connections where a higher MTU is possible so not sure if its best to have the same settings on all branch routers or simply tailor them individually for each specific connection if they have an issue?

[Mohamed Sobair]

Hi Mitchen,

If your branches are connected through an IPSec tunnels over ADSL or 3G, then you should modify their MTUs on the dialer to be 1500-61 = 1439. (The 1439 takes into account all overheads. its the correct value).

Like wise, their LAN side should be modified to 1439-40 = 1399. (Also this takes into account the TCP overhead. its the correct value).

If most of your links are connected as above, make it standard config template and apply it to all. Or you can tailor them per branc basis if they have different type of connection or if they are not using IPSec tunnels.

With the above, you shouldn't worry about the performance, as it should enhance it and make sure your connectivity is not impacted by fragmentations/drops on the WAN that could result in the symptoms your experiencing.

Regards,

Mohamed

[mitchen]

Hi Mohamed, ok, thats great - thanks for the advice.