Hello guys.
I have 2 Switches Catalyst 6504-E running BGP. We have seen drops on the neighbors lately. The drop of the session is due to a hold time expire.
Dec 6 08:35:23.947 UTC: %BGP-SW1-3-NOTIFICATION: sent to neighbor 10.180.210.254 4/0 (hold time expired) 0 bytes
Dec 6 08:35:23.947 UTC: %BGP-SW1-5-NBR_RESET: Neighbor 10.180.210.254 reset (BGP Notification sent)
Dec 6 08:35:23.987 UTC: %BGP-SW1-5-ADJCHANGE: neighbor 10.180.210.254 Down BGP Notification sent
Dec 6 08:35:23.987 UTC: %BGP_SESSION-SW1-5-ADJCHANGE: neighbor 10.180.210.254 IPv4 Unicast topology base removed from session BGP Notification sent
Dec 6 08:35:35.844 UTC: %BGP-SW1-5-ADJCHANGE: neighbor 10.180.210.254 Up
One of the troubleshooting steps was to get a capture of BGP on both Switches and in the capture we saw there are some packets with a length of 1500. It seems that when it send these packets with this length, the sender doesn´t receive the response and the hold time expire apply dropping the session.
The reason why it doesnt get a response is cause the neighborship is running through a DCI environment with OTV and there was a path without Jumbo Frames enabled on one Nexus 7K. (OTV adds a header of 42 bytes and any packet bigger than 1458 could be drop)
53693 Arrival time : 00:20:57.433 UTC Thu Dec 12 2019
Packet Length : 120 , Capture Length : 120
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 106
TCP src=20123, dst=179, seq=844816915, ack=3742816705, win=16145 ACK PSH
53694 Arrival time : 00:20:57.449 UTC Thu Dec 12 2019
Packet Length : 60 , Capture Length : 60
Ethernet II : 0008.e3ff.2222 0008.e311.1111 0800
IP: s=10.180.210.254 , d=10.180.210.253, len 40
TCP src=179, dst=20123, seq=3742821287, ack=844813226, win=15967 ACK
53695 Arrival time : 00:20:57.457 UTC Thu Dec 12 2019
Packet Length : 110 , Capture Length : 110
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 96
TCP src=20123, dst=179, seq=844816981, ack=3742816705, win=16145 ACK PSH
53696 Arrival time : 00:20:57.469 UTC Thu Dec 12 2019
Packet Length : 60 , Capture Length : 60
Ethernet II : 0008.e3ff.2222 0008.e311.1111 0800
IP: s=10.180.210.254 , d=10.180.210.253, len 40
TCP src=179, dst=20123, seq=3742821287, ack=844813226, win=15967 ACK
53697 Arrival time : 00:20:57.513 UTC Thu Dec 12 2019
Packet Length : 1514 , Capture Length : 1514
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 1500
TCP src=20123, dst=179, seq=844813226, ack=3742816705, win=16145 ACK
53698 Arrival time : 00:20:58.121 UTC Thu Dec 12 2019
Packet Length : 1514 , Capture Length : 1514
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 1500
TCP src=20123, dst=179, seq=844813226, ack=3742816705, win=16145 ACK
53699 Arrival time : 00:20:59.333 UTC Thu Dec 12 2019
Packet Length : 1514 , Capture Length : 1514
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 1500
TCP src=20123, dst=179, seq=844813226, ack=3742816705, win=16145 ACK
53707 Arrival time : 00:22:04.781 UTC Thu Dec 12 2019
Packet Length : 1514 , Capture Length : 1514
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 1500
TCP src=20123, dst=179, seq=844813226, ack=3742816705, win=16145 ACK
53708 Arrival time : 00:22:09.997 UTC Thu Dec 12 2019
Packet Length : 60 , Capture Length : 60
Ethernet II : 0008.e311.1111 0008.e3ff.2222 0800
IP: s=10.180.210.253 , d=10.180.210.254, len 40
TCP src=20123, dst=179, seq=844817037, ack=3742816705, win=16145 ACK PSH FIN
Once I corrected the MTU path the session was more stable, but I would like to know why is the reason of the ACK PSH. What does an ACK PSH mean on BGP? Additional, I would like to know why BGP is sending packets with 1500 bytes. The BGP hello is not a fixed size? Or are they variable size?
Switch#show ip bgp neighbors | i Datagrams
Datagrams (max data segment is 1460 bytes):
If the Max Data segment is 1460 why would it send a bigger packet?
I hope someone could help me to understand little bit more about this behavior.
