cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
209
Views
0
Helpful
2
Replies
Highlighted

BGP Neighboring flapping

Hello guys.
I have 2 Switches Catalyst 6504-E running BGP. We have seen drops on the neighbors lately. The drop of the session is due to a hold time expire.

Dec 6 08:35:23.947 UTC: %BGP-SW1-3-NOTIFICATION: sent to neighbor 10.180.210.254 4/0 (hold time expired) 0 bytes
Dec 6 08:35:23.947 UTC: %BGP-SW1-5-NBR_RESET: Neighbor 10.180.210.254 reset (BGP Notification sent)
Dec 6 08:35:23.987 UTC: %BGP-SW1-5-ADJCHANGE: neighbor 10.180.210.254 Down BGP Notification sent
Dec 6 08:35:23.987 UTC: %BGP_SESSION-SW1-5-ADJCHANGE: neighbor 10.180.210.254 IPv4 Unicast topology base removed from session BGP Notification sent
Dec 6 08:35:35.844 UTC: %BGP-SW1-5-ADJCHANGE: neighbor 10.180.210.254 Up

 

One of the troubleshooting steps was to get a capture of BGP on both Switches and in the capture we saw there are some packets with a length of 1500. It seems that when it send these packets with this length, the sender doesn´t receive the response and the hold time expire apply dropping the session.
The reason why it doesnt get a response is cause the neighborship is running through a DCI environment with OTV and there was a path without Jumbo Frames enabled on one Nexus 7K. (OTV adds a header of 42 bytes and any packet bigger than 1458 could be drop)

 

53693 Arrival time : 00:20:57.433 UTC Thu Dec 12 2019
Packet Length : 120 , Capture Length : 120
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 106
TCP src=20123, dst=179, seq=844816915, ack=3742816705, win=16145 ACK PSH

53694 Arrival time : 00:20:57.449 UTC Thu Dec 12 2019
Packet Length : 60 , Capture Length : 60
Ethernet II : 0008.e3ff.2222 0008.e311.1111 0800
IP: s=10.180.210.254 , d=10.180.210.253, len 40
TCP src=179, dst=20123, seq=3742821287, ack=844813226, win=15967 ACK

53695 Arrival time : 00:20:57.457 UTC Thu Dec 12 2019
Packet Length : 110 , Capture Length : 110
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 96
TCP src=20123, dst=179, seq=844816981, ack=3742816705, win=16145 ACK PSH

 

53696 Arrival time : 00:20:57.469 UTC Thu Dec 12 2019
Packet Length : 60 , Capture Length : 60
Ethernet II : 0008.e3ff.2222 0008.e311.1111 0800
IP: s=10.180.210.254 , d=10.180.210.253, len 40
TCP src=179, dst=20123, seq=3742821287, ack=844813226, win=15967 ACK

53697 Arrival time : 00:20:57.513 UTC Thu Dec 12 2019
Packet Length : 1514 , Capture Length : 1514
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 1500
TCP src=20123, dst=179, seq=844813226, ack=3742816705, win=16145 ACK

53698 Arrival time : 00:20:58.121 UTC Thu Dec 12 2019
Packet Length : 1514 , Capture Length : 1514
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 1500
TCP src=20123, dst=179, seq=844813226, ack=3742816705, win=16145 ACK

53699 Arrival time : 00:20:59.333 UTC Thu Dec 12 2019
Packet Length : 1514 , Capture Length : 1514
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 1500
TCP src=20123, dst=179, seq=844813226, ack=3742816705, win=16145 ACK

 

53707 Arrival time : 00:22:04.781 UTC Thu Dec 12 2019
Packet Length : 1514 , Capture Length : 1514
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 1500
TCP src=20123, dst=179, seq=844813226, ack=3742816705, win=16145 ACK

53708 Arrival time : 00:22:09.997 UTC Thu Dec 12 2019
Packet Length : 60 , Capture Length : 60
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 40
TCP src=20123, dst=179, seq=844817037, ack=3742816705, win=16145 ACK PSH FIN

 

Once I corrected the MTU path the session was more stable, but I would like to know why is the reason of the ACK PSH. What does an ACK PSH mean on BGP? Additional, I would like to know why BGP is sending packets with 1500 bytes. The BGP hello is not a fixed size? Or are they variable size?

Switch#show ip bgp neighbors | i Datagrams
Datagrams (max data segment is 1460 bytes):

 

If the Max Data segment is 1460 why would it send a bigger packet?

I hope someone could help me to understand little bit more about this behavior.

2 REPLIES 2
Highlighted
VIP Mentor

Re: BGP Neighboring flapping

Hello,

 

to partially answer your question: as I understand it, both BGP speakers agree on a TCP max data segment size. You can check what that size is:

 

6504-E#show ip bgp neighbor 10.180.210.254 | inc segment
Datagrams (max data segment is 1460 bytes)

 

You could also try and enable path MTU discovery:

 

neighbor 10.180.210.254 transport path-mtu-discovery

 

Highlighted
Beginner

Re: BGP Neighboring flapping

Hello,

 

Please follow this article https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/116377-troubleshoot-bgp-mtu.html

 

In here i recommend you test the tcp adjust mss command under the source interface for the bgp session.

 

BR