GRE Tunnel Bandwidth and IP MTU

[Mitrixsen]

Hello, everyone!

I am currently learning about GRE Tunnels and I have a question about the bandwidth and the IP MTU settings.

When configuring GRE, is it recommended to change the bandwidth and the IP MTU settings?

The MTU is set to 17916 even though Ethernet has a  maximum MTU of 1500 bytes and the

bandwidth is set to 100 kb/s for some reason. Won't this affect calculations like QoS, etc?

Thank you!

[balaji.bandi · Accepted solution]

yes if the Default MTU 1500 and you setup higher value not going to work - you see some packets dropping

GRE Tunnel most suggested method is 1400

some guidance :

https://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html

bandwidth is set to 100 kb/s for some reason. Won't this affect calculations like QoS, etc?

You mean to say they setup manually  some one ?

[Joseph W. Doherty]

"yes if the Default MTU 1500 and you setup higher value not going to work - you see some packets dropping"

Could you explain further?

"GRE Tunnel most suggested method is 1400"

BTW, that recommendation is when using GRE/IPSec.  You can use it with just GRE but personally when I'm working with just GRE I usually just set IP MTU 24 bytes less.

The 1400 recommendation notes (basically) not taking advantage of the additional 20 of 40 bytes isn't as useful as simpler config, and most of the time I agree but sometimes not and I'm less inclined to not use an additional 76 bytes.  There's more to making this choice that's often (unknowingly) overlooked.

Also BTW @balaji.bandi's reference is an excellent source of information and the IP TCP adjust-mss should not be overlooked.

As good as the reference is, there's some considerations, that it doesn't touch upon that can have a huge impact when working with tunnels that cannot increase frame size.  Unfortunately, don't recall any specific single document that explains.

I'll conclude by mentioning GRE kinds of tunneling can be problematic for modern network traffic like video.

[M02@rt37]

Hello @Mitrixsen,

GRE tunnel adds a 24 byte overhead (4-byte gre header + 20-byte IP).

Therefore, to prevent fragmentation and ensure that packets can traverse the tunnel without issues, you should set the IP MTU on the tunnel interface to be 24 bytes less than the IP MTU of the real outgoing interface.

In the case of an Ethernet outgoing interface with a maximum IP MTU of 1500 bytes, you would set the IP MTU on the GRE tunnel interface to 1476 bytes (1500 - 24). This ensures that the encapsulated packets do not exceed the maximum allowable size and avoids fragmentation, which can impact network performance and reliability.

To ensure proper network operation and accurate QoS calculations, it's generally recommended to configure the IP MTU on a GRE tunnel interface to a value that accounts for the overhead introduced by the GRE encapsulation. For Ethernet networks, this is commonly set to 1476 bytes.

Bandwidth setting should reflect the actual available bandwidth for the tunnel to enable accurate QoS and routing decisions. Incorrectly configured MTU and bandwidth settings can indeed affect various aspects of network performance and calculations like QoS.

[Joseph W. Doherty]

"Therefore, to prevent fragmentation and ensure that packets can traverse the tunnel without issues, you should set the IP MTU on the tunnel interface to be 24 bytes less than the IP MTU of the real outgoing interface."

BTW, understand doing what you suggest does preclude fragmentation on the physical interface because it moves the fragmentation to the tunnel interface.

[M02@rt37]

@Joseph W. Doherty

When you set the IP MTU on the tunnel interface to be 24 bytes less than the IP MTU of the real outgoing interface, it's done to ensure that packets traversing the tunnel do not exceed the IP MTU of the tunnel interface. This practice aims to prevent fragmentation within the tunnel itself.

However, this does not preclude fragmentation on the physical interface. If a packet entering the tunnel exceeds the IP MTU of the physical interface, it will be fragmented at the physical interface before entering the tunnel. The purpose of setting the tunnel interface's IP MTU is to ensure that encapsulated packets within the tunnel do not get fragmented again within the tunnel.

Erratum: setting the IP MTU on the tunnel interface as described helps control fragmentation within the tunnel but does not prevent fragmentation on the physical interface. Fragmentation at the physical interface will occur if packets exceed the IP MTU of that interface.

[Joseph W. Doherty]

I suspect "we're not on the same page".  I have in mind the typical Ethernet GRE tunnel router setup.

For such, set tunnel interface to 1476.

Send 1500 packet to tunnel.

This can be done, correct?

Tunnel will fragment, correct?

Physical egress interface will not fragment, correct?

Assuming above, so how might you get fragmentation on physical interface?

What's the impact if you change tunnel MTU to 1500?  None, correct?

[M02@rt37]

You can get fragmentation on the physical interface if packets entering that interface exceed its IP MTU. This can happen if, for example, large packets (greater than the IP MTU of the physical interface) are directly sent into the router on an incoming physical interface, or if the router receives packets from a source network with larger MTUs. In such cases, the router will fragment those packets before sending them out on the physical interface to ensure they fit within the IP MTU of that interface.

Setting the tunnel MTU to 1500 should be accompanied by ensuring that the physical interface's MTU can also support 1500-byte packets without fragmentation. Otherwise, if the physical interface has a lower MTU (e.g., 1492 bytes for PPPoE), packets will still be fragmented at the physical interface to match its MTU, regardless of the tunnel MTU.

[Joseph W. Doherty]

You're still missing the point.  I fully agree a physical interface will fragment (or drop) packets larger than its MTU, BUT if the packets are first transiting a tunnel interface, on that device, whose MTU is NOT larger than the egress physical interface, how does the egress interface get the too large packets?

Similar too:

4k packet => <9k MTU> R1 <1500 MTU> R2 <1500 MTU>

Will R2 ever need to fragment a packet from R1?

Compare to single router:

4k packet => Jumbo Ethernet interface (9k MTU) : tunnel (1500 MTU) : Ethernet interface (1500 MTU)

Would the right side Ethernet interface, only dealing with tunnel traffic, ever need to fragment (or drop) too large packets (entering the

jumbo Ethernet interface

)?

"Setting the tunnel MTU to 1500 should be accompanied by ensuring that the physical interface's MTU can also support 1500-byte packets without fragmentation. Otherwise, if the physical interface has a lower MTU (e.g., 1492 bytes for PPPoE), packets will still be fragmented at the physical interface to match its MTU, regardless of the tunnel MTU."

Agreed ([addendum - correction] except for ". . . regardless of the tunnel MTU.", because tunnel MTU could also allow for PPoE overhead too), but I never wrote otherwise. What I wrote is, if the tunnel interface's MTU is same or less than the physical interface's MTU, the physical interface would never need to fragment, and any fragmentation would happen just at the tunnel interface.

I suspect you understand this but what you have written might be misunderstood.

For example:

"When you set the IP MTU on the tunnel interface to be 24 bytes less than the IP MTU of the real outgoing interface, it's done to ensure that packets traversing the tunnel do not exceed the IP MTU of the tunnel interface."

The problem with that statement is, I believe, it can be misunderstood.  It's fully correct, packets traversing the tunnel do not exceed the IP MTU of the tunnel interface, but that's true about MTU against any interface's traversing traffic (well, technically, for its egress).  But what about in the broader context of precluding any fragmentation and/or where it might happen?

"This practice aims to prevent fragmentation within the tunnel itself."

How does it prevent, as perhaps might mitigate would be a better word choice.

Again, fragmentation in tunnel is still possible, with egress interface MTU - 24.  If this is unclear, even with a single router like:

Ethernet interface (1500 MTU) : tunnel (1476 MTU) : Ethernet interface (1500 MTU)

The tunnel can receive (outbound) 1500 byte packets.  It won't though transmit them without fragmentation.

"However, this does not preclude fragmentation on the physical interface."

If you do set tunnel IP MTU to physical egress interface MTU less 24, how does the physical interface get the packets, from the GRE tunnel, larger than its MTU?  (In your last reply, you mention "physical interface has a lower MTU (e.g., 1492 bytes for PPPoE)", that's fine, but you wrote "24 bytes less than the IP MTU", i.e.1492 less 24 = 1468, for tunnel, and if set, no fragmentation on physical interface, correct?)

"If a packet entering the tunnel exceeds the IP MTU of the physical interface, it will be fragmented at the physical interface before entering the tunnel."

This statement is unclear.  So you're saying an ingress packet larger than an Interface's MTU will be accepted by the interface?  Logically that's a major error, ingress interface should ignore frame.  Physically some Cisco devices will accept packets larger than MTU, but as far as I know fragmentation is only done based on actual egress interface (because different interfaces might have different MTUs).

Again, I suspect "we're not on the same page", i.e. we have different specifics in mind for how this works, i.e. your mention of PPoE, and/or one or even both of use don't correctly understand how this all works.

I welcome you to take specifically what I wrote, and explain why it's incorrect.

And/or explain how I misunderstood your statements and why I believe some are incorrect, are in fact, correct.

[Joseph W. Doherty]

BTW, here's something I've done in a production network, that one of our heavy weight posters said couldn't be done because he never saw it documented anywhere (Cisco or non-Cisco).  He seemed to not believe me.

I.e.:

Ethernet interface (1500 [default] MTU) : tunnel (17K [default] MTU) : Ethernet (tunnel source) interface (9K [jumbo default] MTU)

Questions or comments?

[M02@rt37]

Thanks for that clarification @Joseph W. Doherty. I'm agree with you we have different specifics in mind for how this work. But, in fine, your clarification is perhaps more accurate for a "general" case. Thanks.

[Joseph W. Doherty]

---

M02@rt37 wrote:

Thanks for that clarification @Joseph W. Doherty. I'm agree with you we have different specifics in mind for how this work. But, in fine, your clarification is perhaps more accurate for a "general" case. Thanks.

---

No doubt we have different specifics in mind, but still unclear to me how some of what you wrote applies in any case.

I take great interest in more than "general" cases, i.e. "corner/specific" cases you're unlikely to bump into, but if they are possible, and if you don't allow for them happening, you might then have unexpected results, which might be very bad.

For example, the sinking of the Titanic.  In "general" it had lifeboats, but "specifically" (as now well known) not enough for everyone.

So, after the Titanic, every ship has to have enough lifeboats for everyone.  (BTW, Titanic, I believe, did have enough lifejackets for everyone, so many of the fatalities weren't due to drowning, but by death by hyperthermia being in near freezing water.)

When the Andrea Doria sank, it had enough lifeboats for everyone, BUT as it listed, it couldn't launch half its lifeboats!  (Fortunately, unlike the Titanic, other ships were close by, and were able to rescue the passengers before the ship sank.)

Interestingly, the Costa Concordia had similar issue launching all its lifeboats due to its list (and in the delay in trying to do so).

So, if you have something specific in mind, that's contrary to my "general" case, I would really like to know.

But if you don't that's okay too.

[M02@rt37]

It's OK @Joseph W. Doherty. 

[Joseph W. Doherty · Accepted solution]

"When configuring GRE, is it recommended to change the bandwidth and the IP MTU settings?

Usually yes, but there are reasons why you might not want to (similar to clearing the DF bit).

"The MTU is set to 17916 even though Ethernet has a maximum MTU of 1500 bytes . . ."

True but remember a tunnel is not limited to just standard Ethernet.  Other media supports larger MTUs, including non-standard jumbo Ethernet.

". . .

the bandwidth is set to 100 kb/s for some reason. Won't this affect calculations like QoS, etc?

"

It may, so often a good idea to set to actual available bandwidth.  (Much like you can do with a physical interface where available bandwidth less than interface's bandwidth.)

[Joseph W. Doherty]

BTW, setting up a GRE tunnel isn't that difficult, but getting the most out of it requires a good amount of information on how and why network "things" work the way they do.  Much more than you likely to need to know now, but keep in mind, if you ever start using such tunneling, "for real", in the future, there's much to understand to get the best out of it, and to understand its pitfalls even when optimally configured.

Unfortunately, all the things you should know aren't likely found in any one single place of information.