Cell Interface - IP tcp adjust-mss - resetting value after power off

[Stephen Carter]

Hi all, it's me (again) with another wonderful quirk.

So we are running a 927 router at a site, the site connects into our network via an EZVPN connection running over a Cell interface, this is due to the location of the site and the fact that we can't get ADSL/VDSL to the site.

First thing to note - In the UK - you can't get a static IP address on a Cell connection - so we have to run EZVPN.

So what we are observing is that the site is working with no issues, and the Cell interface is set as follows - 

!

interface Cellular0

description Primary WAN Interface

mtu 1400

ip address negotiated

encapsulation slip

ip tcp adjust-mss 1280

load-interval 30

dialer in-band

dialer idle-timeout 0

dialer string lte

dialer-group 1

ipv6 address autoconfig

async mode interactive

crypto ipsec client ezvpn vpn

end

However, when there is any reset of the device, the Cell interface seems to 'reset' the adjust mss value to the default - as shown below - which then causes the site to have poor latency.

interface Cellular0

description Primary WAN Interface

mtu 1400

ip address negotiated

encapsulation slip

ip tcp adjust-mss 1318

load-interval 30

dialer in-band

dialer idle-timeout 0

dialer string lte

dialer-group 1

ipv6 address autoconfig

async mode interactive

crypto ipsec client ezvpn vpn

end

Any one any ideas as to why this is happening ?

Regards,

S.

[Torbjørn]

What version are you running? Could you reset it through EEM?

[Stephen Carter]

We are on -

Cisco IOS Software, C900 Software (C900-UNIVERSALK9-M), Version 15.9(3)M3, RELEASE SOFTWARE (fc1)

and we don't use EEM.

[Joseph W. Doherty]

If the 1280 value is in the startup config, and on router reset is changed to 1318, sort of sounds like a bug, especially as you can reconfigure it back to 1280.  No support contract on device?

Alternatives include searching for known bugs, trying different IOS versions (problematic w/o support contract), or trying something like an EEM script to self reconfigure (as suggested by @Torbjørn ).

[Joseph W. Doherty]

Oh, forgot to ask, anything show in syslog or on console, possibly related to the value being changed during boot?

[Giuseppe Larosa]

Hello @Stephen Carter ,

>> ipv6 address autoconfig

if you don't use IPv6 and you use only IPv4 and you get an IP address is  better to remove this command. I§f your mobile operator uses IPv6 natively you could get an IPv6 address. Minimum MTU for IPv6 is 1280 bytes so minimum TCP MSS for IPv6 should be 1240 bytes.

>> However, when there is any reset of the device, the Cell interface seems to 'reset' the adjust mss value to the default - as shown below - which then causes the site to have poor latency.

Do you  mean that if the cell interface goes down then you see the rewritten ip tcp mss 1318 ?

or if the router is powered off ? what do you mean with "reset" ?

if so you need someone on place to reload the router to get it working again ?

if this is your scenario, I agree that an EEM script that reacts to down events of interface Cell csn solve if the issue is after each reload of the router again you can use an EEM script that is triggered by system uptime 3 minutes to fix the error on cell0

Hope to help

Giuseppe

[Stephen Carter]

Hello @Giuseppe Larosa,

Some updates - >> ipv6 address autoconfig >> we aren't using this - so will get it removed.

Do you mean that if the cell interface goes down then you see the rewritten ip tcp mss 1318 ? >>> Not sure - not seen the interface go down.

>>> However - if the router is powered off then the mss 'reverts / changes' to 1318 as we think some auto negotiation is happening. 

>>> if so you need someone on place to reload the router to get it working again >>> we still have access to the router.

>>> eem isn't an option at this moment in time.

We have seen that the router and the cell provider 'talk' as we are seeing a 'negotiated mtu size' of 1358, if you then take off 40 bytes for headers - then you get your 1318.

So the question is as we are also running a vpn tunnel over this link where to 'place' any adjust-mss values.

I've done a crude diagram to show what our set up is.

We have 3 vlans - so at present to get them to work (one is voice) - then we are applying the mss on the cell 0 interface - however if there is a priority from the provider to auto neg - then this value will change after a reboot etc, so the question then is where do we apply the mss - we have placed it within the vlan settings.

My thought is like rings of an onion, to place the mss on the vlan interface - say at 1250 or less, then leave the cell interface to auto neg as it feel fit - there should then be enough 'packet size space' to enable the voice etc to work.

Thoughts ?

[Joseph W. Doherty]

"So the question is as we are also running a vpn tunnel over this link where to 'place' any adjust-mss values."

Should work on any transit interface that's carrying traffic not in a tunnel.  Normally, I place it on the tunnel interface.

[Stephen Carter]

@Giuseppe Larosa 

here what the router is saying about the MTU size - 

[Stephen Carter]

@Joseph W. Doherty @Giuseppe Larosa @Torbjørn 

After speaking to our resident CCIE - it has been found that there is a bug out there for a different product line having the issue - 

Cisco Bug: CSCwn48714 - MTU is changed from default 1500 to 1358 by itself on cellular interface ir1101

So as a quick fix, we've applied an additional mss setting on the internal interface whilst a TAC case is raised.

[Joseph W. Doherty]

When you contact TAC, it appears you have TWO bugs.  I.e., the MTU bug your resident CCIE found, looks like it may/should apply to your 927 too, but that bug doesn't mention tcp adjust-mss being boot reset, so there's that apparent bug too.

As to your quick fix, actually if the cellular interface only sees the tunnel encapsulated traffic, the mss adjustment is likely being applied to the tunnel's TCP session, but you really want it applied to the TCP flows using the tunnel.

[Stephen Carter]

@Giuseppe Larosa @Joseph W. Doherty @Torbjørn 

OK, apologies for the delay in replying but we've been testing and working out what's what is our set up.

So, as explained, we have a site using a cell connection (poor landline availability) and to gain access to the 'service' we are running a VPN tunnel over this link. The initial issue was that the Cell interface seemed to 'reset' the MSS size to 1318 whenever the device was power cycled.

On this point first - we are thinking that the cell sim 'talks' and 'agrees' a Max MTU size (see previous screenshots), then, as more probable than not, the packet will have to be routed thru the cell providers network there is an 'Auto' reduction for headers. This makes the MSS value be 1318.

Interesting to point out - we never tested with no MSS statement - so what would happen is anyone guess.

Anyhow, as we didn't what any issues with MSS sizes on the cell interface it was decided not to run EEM, so the upshot is that we applied another MSS statement on the internal vlan interface, this was set at 1240, so then with a VPN header being applied - would take us to 1280 - so giving us some 'wiggle' room on variable packet sizes.

These works were completed on Friday, and we have a daily call with the site to confirm state of play, and they haven't reported any issues since the changes have been made.

On the bug front - we are thinking, because of the afore mentioned cell 'handshaking' - that this was / is a red herring, there is no 'bug', as it was that specific user wanting to use a specific MTU size. As we noticed on our firewall that when we were messing with the MTU/MSS settings there was a continuous stream of packets showing retransmits due to fragmentation.

This fragmentation disappeared when the MSS settings were lowered. 

Hope the above makes sense, any updates / queries / corrections please post back but we are all (at this end) happy bunnies to have a working service. So much so that the customer, as this was to be a one off, has decided that it may be rolled out to other sites have similar networking issues - to quote Ren and Stimpy - 'Oh Joy'.

Stephen

[Joseph W. Doherty]

Yes, it make sense, as adjusting MSS on any transit interface, that "sees" the unencapsulated traffic, ingress or egress, ought to work.  One potential gotcha is non-TCP traffic that fills packets (like video), can be problematic.  That can often be mitigated by MTU.  Problem with it, though, logically it only does what we want at a L3 egress interface.

[Giuseppe Larosa]

Hello @Stephen Carter ,

nice to read that you have found a workaround that solves your issues by lowering MSS on the LAN side to 1240 bytes.

Don't worry about delay in feedback. It is a good thing that you have reported it .

Best Regards

Giuseppe