Understanding MTU, MSS, Frame / Packets / TCP and UDP - From a beginner perspective (confused)

[SJ K]

Hi all,

I have read around extensively but i am still not able to grasp the actual relationship of the above.

MTU

1st and foremost, the description of MTU in TCP/IP guide is ->

The data link layer implementation puts the entire IP datagram into the data portion (the payload) of its frame format, just as IP puts transport layer messages, transport headers and all, into its IP Data field. This immediately presents us with a potential issue: matching the size of the IP datagram to the size of the underlying data link layer frame size.
The IP implementation of all devices on an IP internet needs to be aware of the capacity of the technology used by that implementation for its immediate data link layer connection to other devices. This limit is called the maximum transmission unit (MTU) of the network

In another word to my understanding, MTU is the size of the payload in a layer 2 frame -- please let me know if i am wrong.

================================================================

In my Cisco router, I able to set 2 kinds of MTU.

a) System MTU - the mtu of the interface (layer 2)

b) IP MTU - the mtu of IP (layer 3)

So since i am using Ethernet, i went to search for the size of an ethernet frame.

Assuming i am looking at a layer 2 ethernet with 802.1q frame (without the preamble and frame delimiter) -> that would be 1522 bytes. But it is set as 1500 which is the same for both the system mtu and IP mtu.

q1) Hence, why isn't the system mtu set at 1522 instead ? (still set as 1500) <- should this be set as the MTU of the payload in the layer 2 frame or as the MTU of the layer 2 frame ?

q2) Why is it set this way ? or why do we need 2 parameters then for setting the MTU ? Assuming if I have a jumbo frame, i should set the jumbo frame system MTU to 9000 ? but my IP mtu can still stay at 1500 ? but then what's the point of having a jumbo frame then ?

q3) can a router have different interface with different MTU ? if so, how does the system MTU help in configuring these values ?

q4) When deciding whether do fragment an IP packet does the router check its System MTU or Interface's IP MTU ?

However, I do take note that the minimum frame header size is 22 bytes

===============================================================

IP DATAGRAM/PACKET

Then come the payload, assuming it is IPv4. (screenshot from wiki)

It is said that the "Total Length" is 2 bytes, which represent how large in bytes the packet (including its header + payload can get)  which is 65,535 bytes.  Obviously a packet of such size will not be able to fit into the payload of an ethernet frame.

At this point, i am telling myself that it is a limitation of the payload size in a layer 2 frame.

But overhere, I do take note that the minimum IP header size is 20 bytes.

==============================================================

Maximum Segment Size(TCP)

From wiki, it is mentioned that MSS = specifies the largest amount of data, specified in octets, that a computer or communications device can receive in a single TCP segment. It does not count the TCP header or the IP header.

So I am thinking ok.. my payload of my ethernet frame is 1500bytes. So how large can my TCP segment be ?

1500bytes - 20 bytes (IP header)  - 20 bytes (TCP header) = 1460 bytes for my TCP segment.

But i am totally lost when i read this below

The default TCP Maximum Segment Size is 536  - wiki

  THE TCP MAXIMUM SEGMENT SIZE IS THE IP MAXIMUM DATAGRAM SIZE MINUS  -- RFC 879
      FORTY. 

         The default IP Maximum Datagram Size is 576.
         The default TCP Maximum Segment Size is 536.

q3) why is the default IP maximum datagram size  = 576 ? why isn' it 1500 ? or why isn't it 65,353 bytes ?

q4) why is the default TCP maximum segment size = 536 ?  I have got 1460 bytes of free space , isn't it ?

========================================================

UDP datagram

So I move on to UDP

Length = 2 bytes

Hence maximum size of a UDP datagram -> 65535  bytes

Header size of UDP = 8 bytes

Maximum size of UDP datagram without header = 65535 - 8 = 65527 bytes.

Maximum size of UDP datagram without header inside IP datagram  = 65527 - 20 bytes ( ip header) = 65507 bytes

But again, it still will not be send as the MTU is 1500.

=================================================

Fragmentation

q5) Can i say irregardless of TCP or UDP,  if the MTU between 2 gateway does not match, IP fragmentation will occur ?

q6) I read and it says that it depends on the end-devices to rebuild the fragmented packets, so if 1 packet out of the original e.g 5 fragmented packets is lost , the whole message will be discarded ? 
if it is TCP, then the sending device will have to resend the whole message (resulting in another 5 fragmented packets ?)

==================================================

Regards,
Noob

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

First, understand, there are different MTUs, it depends on transmission medium and what layer of the "stack" you're dealing with.

q1: The 1500 is standard Ethernet  IP MTU.  Standard Ethernet L2 frame MTU is 1518 or 1522 if 802.1Q tagged.  The other overhead, that can extend an Ethernet's single frame's utilization on the wire, of up to 1542, isn't, I believe, considered L2 MTU.

q2: Next understand, Jumbo Ethernet is non-standard, so you cannot count on a specific MTU for it.

Yes, you can have jumbo frame support, while still holding IP MTU to 1500.  However, that usually sort of defeats the purpose of jumbos.  It would be similar, in concept, to setting IP MTU to 500 while using standard Ethernet that supports 1500.

q3a: Yes routers normally can, especially if they support different media.  For example, most serial WAN media support larger MTUs than does Ethernet.  (Token ring also supports larger MTU than Ethernet.)

On routers, you're more likely to find MTU can be configured be routed interfaces.  Switches, though, often work with just one media, usually Ethernet, so if they support different L2 MTUs (e.g. jumbos), often all the hardware is enabled or not, globally.

q4a: It should be per interface.

q3b and q4b: I recall Internet guaranteed a minimal MTU of 576 would be supported, so that's often the off local network default.  NB: IP can transmit larger than L2 MTU, but that requires packet fragmentation.

q5: Not unless you define further what you mean.  Also TCP or UDP are L4, IP is L3.

Each L3 link/hop should support same MTU.  Different L3 links/hops can have different MTUs.  If L3 packet is larger than the next L3 link/hop it desires to transit, packet must be fragmented to fit, or it will be dropped.

All L2 link/hops should support same MTU, otherwise too large frames will be dropped.

q6: Yes, whole packet will be lost if fragment is lost.

Yes and no.  TCP can also send segments larger than IP packets.  If it does, and if IP packets are fragmented, and if a fragment is lost, the whole IP packet is lost, which will require retransmission of the whole TCP segment.  (Normally, TCP sets its MSS to fit within a packet's MTU.)

[SJ K]

Hi Joseph,

Thanks for the reply and your time for reading through this lengthy post .

Please allow me to digest the answers against my questions/thoughts and I will get back by tomorrow..

Hopefully you are still around looking at this thread.

Regards,
Noob

[SJ K]

Hi Joseph,

Sorry for the late reply. I have to admit, that the L2 MTU and L3 MTU is indeed confusing.

When we are talking about L2 MTU , are we referring to the size of entire frame itself or the payload of the frame (which is a L3 datagram).

After reading around

http://switchpacket.blogspot.sg/2014/07/understanding-difference-between-mtu.html

It seems like in cisco system MTU = size of L2 frame without its header

and IP-MTU = size of the payload in the L2 frame.

All in all , i would like to think a ethernet MTU is the maximum size of unit that a L2 frame can carry without its L2 header - which is the payload containing a L3 datagram.

Back to our question

q3b and q4b: I recall Internet guaranteed a minimal MTU of 576 would be supported, so that's often the off local network default.  NB: IP can transmit larger than L2 MTU, but that requires packet fragmentation.

q1) So can i say the value provided in the RFC is just a default maximum size, it can be larger actually ?

The default IP Maximum Datagram Size is 576. The default TCP Maximum Segment Size is 536. -- RFC 879

q5) Can i say irregardless of TCP or UDP,  if the MTU between 2 gateway does not match, IP fragmentation will occur ?
q5: Not unless you define further what you mean.  Also TCP or UDP are L4, IP is L3

q2) Actually what i mean is,  if R1 to R2 connection is using MTU of 1400 and if a frame with 1500 bytes payload is required to be send out of R1, it will have to be fragmentated irregardless if it L4 protocol is TCP or UDP right ?

q3) Let's say if I have a frame with 3000bytes of TCP data going out a link with MTU of 1500, assuming it is split into 2 packets (a,b) and send over the link. If packet A is received and packet B is lost, it will not even reach the TCP stack right ?
The received packet A will be discarded at the IP layer and sender TCP not getting an acknowledgement will resend again .

Is my understanding correct ?

Regards,
Noob

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

MTU is the maximum "chunk" of "data" that can be sent.  For standard Ethernet, that would be 1518 bytes.  However, if the hardware supports 802.1Q the frame size increases by 4 bytes.  If the hardware supports Q-in-Q, or MPLS, frames can be even larger, and with MPLS, actual frame size would depend on the number of MPLS tags.

So, some (e.g vendor) to "simplify" things, but which can make for confusion, sometimes Ethernet MTU is considered its unchanging payload size, i.e. 1500 bytes, which is often also the logical IP MTU size.

To further confuse things, IPv4 MTU is 64 KB, but it's normally reduced to match the L2 MTU's payload size.

Your link reference is trying to explain the foregoing.  So when Cisco configures MTU they don't account for L2 overhead.  When Cisco sets the IP MTU, they are setting a logical limit for the maximum datagram size.

q1: It can be larger, and often is today.  However, IP normally assumes 576 unless PMTUD is active.

q2: If R1<>R2 link is L3, then IP will fragment as long as DF is not set; if set it will drop.

q3: Correct (IP reassembles fragments)

[SJ K]

Hi Joseph,

Thanks for replying. I am glad that you reverted. Appreciate that.

MTU is the maximum "chunk" of "data" that can be sent.  For standard Ethernet, that would be 1518 bytes.  However, if the hardware supports 802.1Q the frame size increases by 4 bytes.  If the hardware supports Q-in-Q, or MPLS, frames can be even larger, and with MPLS, actual frame size would depend on the number of MPLS tags.

So, some (e.g vendor) to "simplify" things, but which can make for confusion, sometimes Ethernet MTU is considered its unchanging payload size, i.e. 1500 bytes, which is often also the logical IP MTU size.

q1) In that case, when people talk about ethernet MTU, are they talking about the size of the whole frame, or they are talking about the logical payload (IP-MTU) ? or do i need to clarify with them ?  

q2: If R1<>R2 link is L3, then IP will fragment as long as DF is not set; if set it will drop.

q2) Do you mean if the link is a switch to switch link, then it is L2 and the oversize frame will be drop instead of being fragmentated ?

Are we able to set MTU on L2 interfaces ?

=====================

q3) Lastly, after all these trial and test, i still do not understand the purpose of MTU.  it is say that MTU is use to make sure that the size of the payload can "fit" into the underlying layer 2 frame else what will happen ?

On the sending egress interface, ip fragmentation will occur if the payload is higher then the IP-MTU set

On the sending egress interface, frame not will not be send if the current frame to send is larger then the MTU set on the egress interface

On the receiving ingress interface, frame will be drop and appear as giant if the incoming frame is larger then the MTU set

So either way, just adjusting the proper MTU will do fine.

But is it as simple as that ? I have always thought MTU is something like a physical media limitation, that if it is not adhere to, sending cannot occur but seems to me right now that sending will/can occur, and it is just whether if the receiving device want to accept it or not.

Regards,
Noob

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Reverted?

q1: You need to clarify.  For example, technically Ethernet's MTU is the maximum frame size.  However, as it can be variable, it's easier to refer to its MTU as is maximum payload size, which isn't really MTU, but then you don't need to account for all the Ethernet situations that impact actual MTU.  That said, sometimes you do need to account for such options.

q2: Correct.  Sometimes, although lower end equipment often sets physical MTU globally.

q3: MTU doesn't have a purpose beyond it's just what the hardware can do.  Contrast with, what's the purpose of 10 Mbps Ethernet.  I.e. Why not 9 Mbps or 11 Mbps, or other?

MTU doesn't make payload fit, it's just what it can support.

If an egress packet's size is larger than the interface's IP MTU, packet can either be fragmented or dropped, in will not be forwarded as is (even if it can fit with the actual L2 MTU).  (For example, if you're sending GRE encapsulation traffic, GRE adds 24 bytes to the packet's size.  So, if you start with a 1500 byte IP packets, going out an interface with a 1500 IP MTU, it needs to be fragmented or dropped because the maximum GRE packet is 1524.  To help [GRE] senders optimize their maximum transmission packet sizes, you can set the IP MTU, on the tunnel interface, to 1476.  [Ethernet interface's MTU is as it was.]  Now the physical interface can deal, optimally, with both GRE and non-GRE packets.)

Yes, (some) senders can transmit larger than a receiver can handle, but the receiver will consider the received frame "bad".  To avoid this, we have standards that both sides can agree.  With Ethernet, however, the standards keep evolving, many becoming options, all, unfortunately, not dynamically discovered, MTU being one of the latter.

For example, I believe full-duplex was never really a standard for 10 Mbps Ethernet, but many vendors added it (when it arrived with 100 Mbps FastEthernet).  I've worked on Cisco switches, that if you plugged a 10/full host into the switch port, the switch port, which was only capable of 10/half, would error disable.

[SJ K]

Hi Joseph,

Sorry for my poor english. What i mean is i am glad that you have reverted back in the thread.
 

Thank you for your explanation.

If an egress packet's size is larger than the interface's IP MTU, packet can either be fragmented or dropped

q1) can i check the scenario where it will be dropped ? (if the DF bit is on ?)

If an egress packet's size is larger than the interface's IP MTU, packet can either be fragmented or dropped, in will not be forwarded as is (even if it can fit with the actual L2 MTU).  (For example, if you're sending GRE encapsulation traffic, GRE adds 24 bytes to the packet's size.  So, if you start with a 1500 byte IP packets, going out an interface with a 1500 IP MTU, it needs to be fragmented or dropped because the maximum GRE packet is 1524.  To help [GRE] senders optimize their maximum transmission packet sizes, you can set the IP MTU, on the tunnel interface, to 1476.  [Ethernet interface's MTU is as it was.]  Now the physical interface can deal, optimally, with both GRE and non-GRE packets.)

q2) Can i check when is the IP-MTU value being checked against the actual payload size ? is it before adding the 24bytes or after adding the 24bytes.  If IP-MTU is checked after the adding of the 24bytes to the actual payload, then  how will setting the IP-MTU to 1476 prevent any fragmentation ?

(e.g. 1476 payload is received -> 24 bytes is added  (become 1500) -> check against the IP-MTU parameter of 1476 -> still need to fragment.

q3) For MTU path discovery, is it done on routers or only on end-devices ?   How do we check if MTU path discovery is in use and what is the MTU size in use for a particular route / or to a particular destination  ?

Regards,
Noob

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Ah, not a native English speaker, then my hat off to you.  Any foreign language can be difficult, possible more so when it's technical.

q1: Right, if DF is set, and fragmentation is needed, packet will be dropped.  That's the scenaro, but you can see this in action, pinging with too large packets when DF is set.

q2: Good question.  Normally I would expect IP packet's size to be checked against interface before its transmitted (to see if it "can fit").  Tunnels, though, might be different.  I know the original IP packet size is compared, but the packet is "transmitted" with the additional header (which can exceed the IP MTU).

How it works with GRE tunnels, if original IP packet is limited to 1476, and then the 24 byte GRE header is added, the encapsulated packet is now 1500, which wouldn't normally need to be fragmented.  Compare that to starting with 1500, adding 24, so now 1524 needs to be fragmented.

q3: PMTUD is done on the sender.  Basically, IP packets are sent with DF set and the sender will respond (by reducing packet's size) to ICMP messages of "packet needs fragmentation".  PMTUD also normally resets itself on a timer.  This to allows for changing paths with different MTUs.  I.e. bigger packets might be supported, if the path changes.

[SJ K]

Hi Joseph,

Sorry for the late reply and i thank you for your kind compliment. Yeap.. not being a native english speaker did provide some challenge when i try to put my doubts into words, but i am glad you are able to comprehend what i am asking about ;)

Back to the questions

How it works with GRE tunnels, if original IP packet is limited to 1476, and then the 24 byte GRE header is added, the encapsulated packet is now 1500, which wouldn't normally need to be fragmented.  Compare that to starting with 1500, adding 24, so now 1524 needs to be fragmented.

q1) that is something puzzling me too as well,  in the above scenario, if the IP-MTU is set at 1476, wouldn't it still need to be fragmented ?

PMTUD is done on the sender.  Basically, IP packets are sent with DF set and the sender will respond (by reducing packet's size) to ICMP messages of "packet needs fragmentation".  PMTUD also normally resets itself on a timer.  This to allows for changing paths with different MTUs.  I.e. bigger packets might be supported, if the path changes.

q2) In the above, can i say that transit devices (.e.g router, switches) still have their MTU settings unchanged, but the sender (it can be the transit devices themselves) will send out frames with the payload according to what the PMTUD recommends .

But how can we check if PMTUD is in play ?

Thank you! Hope to hear from you soon.

Regards,
Noob

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

q1:  You'll need to clarify your puzzlement.  I tried to answer that question in my prior post, so you need to further explain what's still puzzling you.

q2: Assuming you can see the packets on the wire, I think that's as simple if you see DF bits being set.  If not, you would need to investigate how the host turns the feature on or off.  For example, on Windows hosts, believe it's a registry setting.

[SJ K]

Hi Joseph,

I am sorry for the confusion.

q2. Duly noted.

For q1,  what i meant is i am quite puzzled about my oirginal question on how IP-MTU works with GRE tunnels.

Assuming if we set IP-MTU at 1476, and the payload of the ethernet frame is 1476 with 24bytes added later for the GRE header which is then = 1500bytes. Wouldn't it exceed the IP MTU of 1476 and result in the fragmentation of the packet ? But perhaps as what you mentioned, it might be different with the GRE tunnels.

Regards,
Noob

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

How IP MTU appears to work on tunnels, it compares to the packet's size without encapsulation overhead.  The physical egress interface, though, compares to the packet's size with the encapsulation overhead.

[SJ K]

Hi Jospeh,

How IP MTU appears to work on tunnels, it compares to the packet's size without encapsulation overhead.  The physical egress interface, though, compares to the packet's size with the encapsulation overhead

Over here, the "physical egress interface compares.."  refer to comparing to the "MTU" of the interface, right ?

Regards,
Noob

[Joseph W. Doherty · Accepted solution]

No, either IP MTU or MTU.