cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
24209
Views
16
Helpful
40
Comments
Aleksandar Vidakovic
Cisco Employee
Cisco Employee

 

Introduction

This document provides a sample configuration for adjusting the TCP Maximim Segmet Size (MSS) on the ASR9000 routers.

TCP MSS adjustment is required when the network operators want to ensure that the TCP packets traversing the network will not exceed a predetermined size, thus preventing any unnecessary fragmentation of TCP packets on links with small MTU. Typically this may happen on links with the MTU of  1500 bytes, when the original IP/TCP packet is encapsulated into PPPoE, GRE or any other encapsulation.

Such modification of TCP MSS size doesn't cause any observable impact on TCP session throughput because the throughput of a TCP session is predominantly dictated by the TCP window size.

TCP basics

When TCP sessions establish, they signal 2 key parameters that dictate the transfer speeds of the session. That is the MSS, the maximum size of a TCP packet and the Window size. In TCP transmission, in its basic operation is to be acknowledged. So packet transferred, receive ACK and send the next packet. This is very slow especially on long end to end delays. Windowing allows for multiple packets to be sent and using a single ACK on that window. If one packet is lost of that window, the whole window is to be retransmitted. The larger the window size is, the better for thruput, but on unreliable links, this can have an adverse affect if large windows have to be retransmitted. Also large windows require larger receive buffers on the receiving side.

So the size of the packet is dictated by the MSS. The number of packets at that size to be sent without direct ACK on each packet is the window size.

 

What does MSS adjust do?

MSS is the value that a TCP client signals to a server indicating what the maximum size of a packet is that it can receive. Generally this value is derived from the client's local MTU size minus 40 bytes. These 40 are composed of 20 bytes for ip header and 20 bytes for the tcp header.

The segment size refers to the actual TCP payload carried.

If there are mtu changes in the network, which can easily be seen for instance with PPPoE, which adds 8 bytes of additional header, it could lead to a client with a local MTU of 1500, signaling a TCP mss of 1460, but actually requiring 1508 bytes on a transmission line (1460 + 20 ip + 20 tcp + 8 pppoe). This will cause some hops to do fragmentation.

The TCP MSS adjust feature intercepts a TCP syn packet and scans for the first option after the tcp header. If that option is the mss size, it can re-adjust the value to the desired value configured and update the checksum along with it.

Example scenario:

 

Prerequisites

Basic understanding of the ASR9000 architecture and IOS-XR operating system.

 

Restrictions

 

  • Feature is supported on Typhoon and Tomahawk line card generations.
  • Feature is supported on BNG subscriber interfaces, on L3 dot1q sub-interfaces and on GRE tunnels. (You can't configure the ipv4 mss enable on main interfaces).
  • Range of supported sizes for TCP MSS adjustment is from 1280 to 1535 bytes.
  • UDP packets are obviously not subject to MSS rewrites, only the TCP SYN packet is intercepted and rewritten.

 

Configure

The implementation of TCP MSS adjust on ASR9000 was dictated by two factors:

  1. minimal performance impact when applied to an interface
  2. performance doesn't change with deployment scale (i.e. number of interfaces where the feature is applied)

To meet these requirements, the chosen implementation requires two configuration steps:

  1. specify the TCP MSS size at the network processor (NP) level.
  2. specify the interfaces on which the configured TCP MSS size should be applied

One consequence of this approach is that all interfaces on a given NP must use the identical value for TCP MSS adjustments.

When enabled on an interface, TCP MSS feature is applies in both directions. i.e. adjustment is applied on ingress and egress packets. Feature is applied to both IPv4 and IPv6 even if only one of them is configured.

To determine which NP controls the interface in question, use the show controllers np ports exec command.

 

BNG specific

For BNG which first supported the mss-adjust capability an different configuration can be used:

subscriber
 pta tcp mss-adjust 1410
!

The value range for the BNG command is 1280-1536. The reason for that is that as you can see we only adjust for the higher order byte to save some space in the ucode variable.

 

Identifying fragmentation

Fragmentation can be identified by looking at the NP counters.

Note that for BNG and non BNG the commands and counters are different.

The reason for that is that Fragmentation is handled by NETIO for regular L3 interfaces. However BNG subscriber interfaces don't register with NETIO. This was done for scalability reasons since BNG subs don't require all the services that NETIO provides. For BNG subs, a "lite-weight" NETIO was designed called SINT (subscriber interface) that provides some basic L3 services such as access to the TCP stack, some basic ICMP support etc.

Verifying BNG subscriber fragmentation:

Looking at fragmentation at the SPP (the interupt switching path) level.

RP/0/RSP0/CPU0:A9K-BNG#show spp node-counters | i frag
Sun Mar  8 11:08:17.501 EST
ipv4_frag
       Drop: Dont-fragment set:            3125 <<<< packets that have DF bit set
                     ipv4-frag:            3854  <<<<< packets fragmented

 

Verifying the NP counters with show controller np counters NP<x> location 0/<y>/CPU0
16  MDF_TX_LC_CPU                11037 107 <<<<<<<<<< 100pps to the LC CPU
17  MDF_TX_WIRE                  17423 201    <<<<< 200 packets to the wire (2 frags)
21  MDF_TX_FABRIC                24798 299 <<<<<<<<<<<200 pps injected from the fab
41  PARSE_INJ_RECEIVE_CNT        17079 201 <<<<<< 200pps injected
45  PARSE_ENET_RECEIVE_CNT       8969  101 <<<<received on wire from the tester
416  RSV_PUNT_IP_MTU_EXCEEDED    9615  99 <<<<<100pps requiring frag

1048  PPPOE_FRAG_NEEDED_PUNT     9615  99  <<<<<on pppoe sessions

 

Verifying regular L3 fragmentation:

with the same command show controller np counters NP<x> loc 0/<y>/cpu0 the following can be observed for fragmentation:

  16  MDF_TX_LC_CPU              718    106 << sent to the LC CPU for frag
  33  PARSE_FAB_RECEIVE_CNT      681    100 << packets received from the fabric
 416  RSV_PUNT_IP_MTU_EXCEEDED   677    100 << packets needing punt because of MTU
 842  IPV4_FRAG_NEEDED_PUNT      677    100 << packets punted for FRAG reasons


Note that in both cases the DF bit is NOT assessed in the hardware, this is handled by the controlling agent, whether it be SPP for BNG subs or NETIO for regular L3 interfaces.

 

Configuration Example

In this example the TCP MSS adjust is applied to interface Bundle-Ether48.10.

Step 1: Determine the NP on which the TCP MSS adjust is to be configured:

RP/0/RSP0/CPU0:av-asr9001#sh bundle bundle-ether 48
<snip>
  Port                  Device           State        Port ID         B/W, kbps
  --------------------  ---------------  -----------  --------------  ----------
  Gi0/0/1/6             Local            Active       0x8000, 0x0003     1000000
      Link is Active
  Gi0/0/1/7             Local            Active       0x8000, 0x0004     1000000
      Link is Active

RP/0/RSP0/CPU0:av-asr9001#show controllers np ports all location 0/0/CPU0

                Node: 0/0/CPU0:
----------------------------------------------------------------

NP Bridge Fia                       Ports
-- ------ --- ---------------------------------------------------
0  0      0
0  0      0   TenGigE0/0/2/0, TenGigE0/0/2/1
1  1      1   GigabitEthernet0/0/1/0 - GigabitEthernet0/0/1/19
1  1      1   TenGigE0/0/2/2, TenGigE0/0/2/3
RP/0/RSP0/CPU0:av-asr9001#

 

Step 2: Configure the desired TCP MSS value and activate the feature on the interface Bundle-Ether48.10.

hw-module location 0/0/CPU0 tcp-mss-adjust np 1 value 1300
!
interface Bundle-Ether48.10
 ipv4 address 4.8.10.4 255.255.255.0
 ipv4 tcp-mss-adjust enable
 encapsulation dot1q 10

 

 

Verify

 

To verify that the feature is enabled on a selected interface, check the microcode interface descriptor block (uidb) setting:

RP/0/RSP0/CPU0:asr9001#sh uidb data location 0/0/CPU0 Bundle-Ether 48.10 ing-extension | i TCP
  TCP MSS ADJ Enable               0x1
RP/0/RSP0/CPU0:asr9001#sh uidb data location 0/0/CPU0 Bundle-Ether 48.10 extension | i TCP
  TCP MSS ADJ Enable               0x1

 

You can verify the rewrite with a simple TCP session on an IOS router connected to the device that is doing the MSS rewrite.

Enable debug tcp transactions and packet.

You'll see:

Feb  6 14:17:26.112: tcp0: I LISTEN 54.1.1.2:1234 81.1.1.2:23 seq 0
        OPTS 4 SYN  WIN 0
Feb  6 14:17:26.112: TCP0: state was LISTEN -> SYNRCVD [23 -> 54.1.1.2(1234)]
Feb  6 14:17:26.112: TCP0: tcb D333A2EC connection to 54.1.1.2:1234, received MSS 1300, MSS is 516

54.1.1.2 is the source and 81.1.1.2 is the destination. The debug is taken from 81.1.1.2

You can see that the decoded MSS value received is 1300 as per configuration on the asr9000 which sits in between client and destination. Our local MSS value is 516 which we advertise. This is merely to illustrate a possible verification of the MSS rewrite rather then focusing on the actual values.

 

Additional Information

It works similar in case of GRE Tunnel:

  • enable under the tunnel-ip the command 'ipv4 tcp-mss-adjust enable'
  • locate the exit interface the tunnel uses ((whether it be before -> traffic going into the tunnel, or after decap -> coming out of the tunnel) and then apply the corresponding 'hw-module location <> tcp-mss-adjust np X value <number>'

Related Information

Comments
xthuijs
Cisco Employee
Cisco Employee

hi sandeeya,

if you only use the interface configuration and you dont define the value for the npu, nothing will happen...

the reason why we have a 2 step approach here is for instruction and memory conservation.

the NPU has only limited instruction memory and data memory available; if we were to support a per interface configuration we would need a table for # of interface on npu times value (which is 2-bytes, 16k). in order to conserve memory and instructions we decided to have a flag for the interface; hey you're enabled or not and then a single (!!!) byte for the higher order byte of the mss value (that is why the range is confined that you can configure).

this way we can ensure pps performance, reduce the mem consumption of structures etc.

the logic is:

if ip and tcp and syn_flag (only) and first 2 bytes after tcp header is 0204 (=mss option) then:

    compare value signaled

        if < then

        rewrite with configured value

        update checksum

        else

            don't touch

xander

chaitugadu
Community Member

Hi

Regards,

CH Chaitanya

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

hi CH,

as the document explains, configuring TCP MSS adjust on asr9k is a two step task. The command "ipv4 tcp-mss-adjust enable" is the first step. In the second step you specify the value:

hw-module location 0/0/CPU0 tcp-mss-adjust np 1 value 1300

Please note that hw-module commands require a reload to make the changes effective.

/Aleksandar

Hi Alexandar,

 

I have a question about a LAC ASR9K in a LAC/LNS scenario.

Assuming that MSS is set in the LNS side, do we need to add the "l2tp tcp-mss-adjust" command to the ASR9K LAC as well?


Thanks and Regards,

Dimitris

mavip
Level 1
Level 1

Hi,

Could you confirm if the TCP MSS adjust on the ASR 9901 is performed just in the SYN packet and not on the SYN-ACK packet?

We ask this question why we are dealing with an AntiDDoS solution that is based on adjustment of the TCP MSS on the SYN ACK packet:

  • Traffic from clients to servers is forwarded via the AntiDDoS provider. This provider adjusts the TCP MSS on the SYN packet and sends it over a GRE tunnel that ends on an ASR9901.
  • Traffic from servers to clients is forwarded directly via the ASR9901, not via the GRE tunnel, so the TCP MSS needs to be modified on the SYN-ACK packet to inform the client about the adjustment. If the TCP MSS is not modified on the SYN ACK, the client and the server will negotiate the TCP MSS based on their own settings not on the TCP MSS needed by the AntiDDoS provider.

Thank you,

Manuel

tomasnohejl
Level 1
Level 1

Hi @Aleksandar Vidakovic and @xthuijs ,

 

We are using ASR9006 with A9K-RSP880-SE, A9K-MOD80-SE, A9K-MPA-4X10GE, SW 6.1.4 and it seems, that after enabling only the first command under IP interface "ipv4 tcp-mss-adjust enable" without specifying the MSS value with the second command "hw-module location x/x/CPUx tcp-mss-adjust np x value xxxx".

Was there any change in code since writing the original post? Is there some default mss value defined?

Is there some command how to show the configured mss value?

 

Regards 

 

Tomas Nohejl

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

@tomasnohejl 

 

Hi Tomas,

your first sentence seems to be incomplete, but I suppose you wanted to say that MSS adjust worked even without specifying the MSS value with the "hw-module ..." command. Default value should be derived from the interface MTU. I suppose the traffic stream in question is transiting the asr9k (i.e. asr9k is not the endpoint of the TCP session). What MSS values did you observe before and after applying the "ipv4 tcp-mss-adjust enable" command? What is the IP MTU of the interface where you have applied the command?

 

/Aleksandar

xthuijs
Cisco Employee
Cisco Employee
hi tomas, it is highly recommended to set a desired value with the np command.
np ucode will pick up some default value, I don’t remember what it is
it is likely derived by the offset of np value 0 (undefined) going to the smallest value it could do (thought it was 1240 or so).
(we’d be only rewriting a set of bits from the mss not the whole 16bits. xander
tomasnohejl
Level 1
Level 1

Hi @Aleksandar Vidakovic and @xthuijs ,

 

@Aleksandar Vidakovic  - sorry for incomplete sentence, you are right.

yes, the traffic stream is transiting the asr9k. I have no chance to make capture to see the size of MSS, the only thing I know is, that before issuing the "ipv4 tcp-mss-adjust enable" command, there was a problem with TCP connection on end VM and after enabling the command on transit IP interface on ASR 9k, the problem disappeared - it was only the first attempt to aim where the problem is... and I was surprised, that ASR9k is manipulating the MSS without defined default MSS value. I had no time more for troubleshooting, hope I will be back on this problem in future..

 

@xthuijs - So the final statement is, that without defining the default MSS value for each NP, probably the value 1240 is used ..

 

thanks both for quick reaction!

 

 

 

xthuijs
Cisco Employee
Cisco Employee
yeah it does sound you were suffering from fragmentation/oversized packets and the mss alleviated that.
considering an unconfigured value renders zero, and that being xlated to the high byte of the mss, you likely ended up
with the smallest configurable value.
there is no negative side effect with that, just that your max packet size is a bit smaller so reduces inefficiency …
xander!
Rui Sam
Level 1
Level 1

Apologies reviving an old thread, but I have a query around the GRE case.

To get it working with GRE we enable the MSS "template" on the NP matching the physical interface and we then "enable" the TCP MSS on the tunnel itself.

In this instance will the MSS adjustment only happen for the traffic going through the GRE tunnel? I.e other traffic going through the physical wont be affected (unless I was to explicitly "enable" TCP MSS adjust on the physical interface itself)?

And also, to verify all is actually working, I'd need to capture the traffic going through the GRE tunnel? ie there is no command that will tell me traffic is being fragmented on a specific interface (as opposed to a NP)?

Thanks in advance.

--Rui

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

@Rui Sam 

hi Rui,
TCP MSS adjust is applied only to those interfaces where you decide to enable it.

/Aleksandar

@Aleksandar Vidakovic  @xthuijs 

Thanks for this page it was very useful in dealing with a cloud DDOS provider, the same one that everyone else is referring to on this thread.  We implemented the changes on a pair of 9001 routers.

Do the same principles apply to the 9900 series and if not can you point me to the page or thread that describes making similar changes on the 9900 series. Interested in the 9904 and 9908.  Thanks in advance!

 

 

Aleksandar Vidakovic
Cisco Employee
Cisco Employee

@Kevin Michael Pratt 

hi Kevin,

thank you for the kind feedback. I'm glad to hear that you've found this feature useful.

TCP MSS adjust is performed by the line card's network processor. Therefore the chassis model is irrelevant, what matters is on which line card generation are you enabling this feature.

TCP MSS adjust is supported by 2nd (aka Typhoon), 3rd (aka Tomahawk) and 5th (Lightspeed Plus) line cards. It may work on 4th gen (aka Lightspeed) as well, but as those cards are only designed for core services, we don't recommend trying this feature on it.

Among fixed chassis, ASR9001 is based upon a Typhoon, ASR9901 is based upon a Tomahawk, while ASR9902 and ASR9903 are based upon a Lightspeed Plus line card.

Hope this answers your question.

/Aleksandar

@Aleksandar Vidakovic  Thanks very much for the Super Quick reply! This helps.

For anyone in the same boat, here is the Cisco matrix for part-number to line-card generation.

https://www.cisco.com/c/en/us/support/docs/routers/asr-9000-series-aggregation-services-routers/116726-qanda-product-00.html

 

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Quick Links