BGP path MTU discovery question

dean holroyd · ‎02-01-2008

morning

please tell me why this is? there are two iBGP sessions to the same neighbour in my example below. one session is direct to the interface, the other session is between the loopback. path mtu discovery is on. the directly connected session mss is found to be 1500-40 = 1460 bytes. the loopback sessions default to the 536 byte mss. the mtu of the physical interfaces is 1500. the mtu of the loopbacks is 1514. there is nothing else involved.

why does the session to the loopbacks have an mss of 536?

#sh ip bgp nei | inc max data

Datagrams (max data segment is 1460 bytes):

Datagrams (max data segment is 536 bytes):

pings to the loopback to verify MTU

#ping 1.1.1.1 size 1500 df

Type escape sequence to abort.

Sending 5, 1500-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

Packet sent with the DF bit set

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 24/54/96 ms

#ping 1.1.1.1 size 1501 df

Type escape sequence to abort.

Sending 5, 1501-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

Packet sent with the DF bit set

.....

Success rate is 0 percent (0/5)

TIA

s.arunkumar · ‎02-01-2008

The MSS u saw for loopback (536)is the default value..

MSS in option is send in TCP 3-way handshake time and the default value is used if one or the both end of connection is not using the MSS..

Lets hear form experts why this happening for loopback case only..

arun

Richard Burts · ‎02-01-2008

Dean

I believe that what you see is the different behavior when negotiation is conducted between two neighbors who are both on the same connected subnet (use the MTU of the media) and negotiation between neighbors who are "remote" from each other and traffic will be forwarded over other connecting subnets. If packets will be forwarded over media whose MTU is potentially unknown, the conservative position is to start with a smaller MSS.

HTH

Rick

HTH

Rick

dean holroyd · ‎02-01-2008

thanks arun and rick.

as far as i understand it the PMTUD should work for multiple datalinks to the peer router. i just set up a session over two links to an ethernet int on a non direclty connected peer.... PMTUD works, so it would appear to be locked down to something to do with the loopbacks giving an MSS of 536

#sh ip bgp nei | inc BGP neighbor is|max data

BGP neighbor is 1.0.0.21, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

dean holroyd · ‎02-01-2008

ok, some more testing

#sh ip bgp nei | inc BGP neighbor is|max data

this neighbour is two logical hops away and it uses the facing eth interfaces as source

BGP neighbor is 1.0.0.21, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

this is the same neigbor, but the loopbacks are used.... note MSS is 1460!

BGP neighbor is 3.3.3.2, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

this neighbor is one hop away an it uses the facing eth interfaces as source

BGP neighbor is 1.1.1.2, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

this is the same 'one hop away' neighbor but it uses the loopbacks as source.... MSS is 536 :-(

BGP neighbor is 4.4.4.1, remote AS 1, internal link

Datagrams (max data segment is 536 bytes):

my conclusion is that a neigbour one hop away using loopback as source will report an MSS of 536. this seems to bear out on our 'live' network.

reason?

thanks

dean holroyd · ‎02-20-2008

bump... sorry

can anyone answer the discrepancy caused by using the loopbacks......

mheusing · ‎02-20-2008

Hi,

two ideas: mtu discovery only works on new sessions. Are you sure the session was not established before path MTU discovery was turned on?

Second, path MTU discovery as far as I remember, will depend on ICMP to learn about MTU. Are ACLs in place preventing this?

Regards, Martin

dean holroyd · ‎02-20-2008

Hi Martin

there are no ACLs. I just ran the test again in the lab please see the results below

1.0.0.2 is the FE int of the neighbor

1.1.1.11 is the loopback int of the neighbour

the FE in reports an MSS of 1460 and the loopback reports an MSS of 536

thanks

R0#clear ip bgp *

*Feb 20 10:55:34.939: %BGP-5-ADJCHANGE: neighbor 1.0.0.2 Down User reset

*Feb 20 10:55:34.939: %BGP-5-ADJCHANGE: neighbor 1.1.1.11 Down User reset

*Feb 20 10:55:35.551: %BGP-5-ADJCHANGE: neighbor 1.1.1.11 Up

*Feb 20 10:55:35.879: %BGP-5-ADJCHANGE: neighbor 1.0.0.2 Up

R0#sh ip bgp nei | inc max data|BGP neighbor is

BGP neighbor is 1.0.0.2, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

BGP neighbor is 1.1.1.11, remote AS 1, internal link

Datagrams (max data segment is 536 bytes):

dean holroyd · ‎02-20-2008

ok, I see the error of my ways. please class this as resolved

R0#

*Feb 20 11:37:29.915: %BGP-5-ADJCHANGE: neighbor 1.0.0.2 Down User reset

*Feb 20 11:37:29.915: %BGP-5-ADJCHANGE: neighbor 1.1.1.11 Down User reset

*Feb 20 11:37:29.915: %BGP-5-ADJCHANGE: neighbor 2.0.0.2 Down User reset

*Feb 20 11:37:29.915: %BGP-5-ADJCHANGE: neighbor 2.2.2.22 Down User reset

*Feb 20 11:37:31.079: %BGP-5-ADJCHANGE: neighbor 1.1.1.11 Up

*Feb 20 11:37:31.079: %BGP-5-ADJCHANGE: neighbor 2.2.2.22 Up

*Feb 20 11:37:32.503: %BGP-5-ADJCHANGE: neighbor 1.0.0.2 Up

*Feb 20 11:37:32.503: %BGP-5-ADJCHANGE: neighbor 2.0.0.2 Up

R0#sh ip bgp nei | inc max data|BGP neighbor is

BGP neighbor is 1.0.0.2, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

BGP neighbor is 1.1.1.11, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

BGP neighbor is 2.0.0.2, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

BGP neighbor is 2.2.2.22, remote AS 1, internal link

Datagrams (max data segment is 1460 bytes):

you need mtu discovery enabled on ALL devices.

thanks to anyone who responded though

mohammedmahmoud · ‎02-20-2008

Dean,

To recap your case and have a complete answer for anyone following this thread, Ricks post and your finding complements each other to be the complete answer. PMTUD is disabled by default on Cisco routers (needs the ip tcp path-mtu-discovery command), if kept disabled and the peers are directly connected then the maximum MSS can be used (normally 1460 bytes if connected via Ethernet), if not directly connected then the default MSS (as per RFC 879) which is 536 bytes is used. You need to enable the PMTUD on both routers (you are enabling the PMTUD on the routers as TCP(BGP) hosts now not as routers, i am saying this statement to highlight that PMTUD operates on the TCP hosts not the routers in between) for the BGP to be established with the highest possible MSS, if only one side is configured with PMTUD and the other has it disabled, then the agreed upon MSS will be the lower which will be 536 bytes.

And as Martin has said, if ICMP is blocked in the path in between, PMTUD will fail to work, since it uses the ICMP type 3 code 4 packet (Destination Unreachable; Fragmentation Needed and DF set) to return back the maximum usable MTU.

BR,

Mohammed Mahmoud.