L2VPN/EVPN E-BGP Peers Flapping

[LewisD1]

Hello, 

I'm having an issue with BGP sessions expiring. This issues seems to resolve itself after hours/days then works solidly for weeks before the issue then re-occurs and i'm drawing up a blank. All the iBGP sessions work without issues just the E-BGP sessions causing me grief. Im using BGP multihop for the Peering with dual 10GB Links and a multiarea OSPF underlay.

This is what i'm seeing in the logs

22 May 24 12:35:51.876119 bgp: [23668] (default) ADJ: 122.122.122.122 keepalive timer fired
2022 May 24 12:35:51.876144 bgp: [23668] (default) ADJ: 122.122.122.122 keepalive timer fired for peer
2022 May 24 12:35:51.876155 bgp: [23668] (default) ADJ: 122.122.122.122 sending KEEPALIVE
2022 May 24 12:35:51.876639 bgp: [23668] (default) ADJ: 122.122.122.122 next keepalive expiry due in 00:00:59

2022 May 24 12:32:22.329490 bgp: [23668] (default) ADJ: 121.121.121.121 keepalive timer fired
2022 May 24 12:32:22.329520 bgp: [23668] (default) ADJ: 121.121.121.121 keepalive timer fired for peer
2022 May 24 12:32:22.329537 bgp: [23668] (default) ADJ: 121.121.121.121 sending KEEPALIVE
2022 May 24 12:32:22.330033 bgp: [23668] (default) ADJ: 121.121.121.121 next keepalive expiry due in 00:00:59

On the other side i see this.

2022 May 24 12:34:18.077881 bgp: [26672] (default) ADJ: 221.221.221.221 keepalive timer fired
2022 May 24 12:34:18.077907 bgp: [26672] (default) ADJ: 221.221.221.221 keepalive timer fired for peer
2022 May 24 12:34:18.077918 bgp: [26672] (default) ADJ: 221.221.221.221 sending KEEPALIVE
2022 May 24 12:34:18.078387 bgp: [26672] (default) ADJ: 221.221.221.221 next keepalive expiry due in 00:00:59
2022 May 24 12:34:18.086460 bgp: [26672] (default) ADJ: Peer 221.221.221.221 has pending data on socket during recv, extending expiry timer
2022 May 24 12:34:18.086940 bgp: [26672] (default) ADJ: 221.221.221.221 KEEPALIVE rcvd

2022 May 24 12:34:03.584558 bgp: [26672] (default) ADJ: 222.222.222.222 keepalive timer fired
2022 May 24 12:34:03.584588 bgp: [26672] (default) ADJ: 222.222.222.222 keepalive timer fired for peer
2022 May 24 12:34:03.584598 bgp: [26672] (default) ADJ: 222.222.222.222 sending KEEPALIVE
2022 May 24 12:34:03.585085 bgp: [26672] (default) ADJ: 222.222.222.222 next keepalive expiry due in 00:00:59
2022 May 24 12:34:03.587219 bgp: [26672] (default) ADJ: Peer 222.222.222.222 has pending data on socket during recv, extending expiry timer
2022 May 24 12:34:03.587696 bgp: [26672] (default) ADJ: 222.222.222.222 KEEPALIVE rcvd

So keep alives are being sent in both directions but only received on one side. Everytime the other side sends one it get the error, has pending data on socket during recv, extending expiry timer. This causing the timers to expire and it to start all over again. 

Site 1 Relationships. 

show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 121.121.121.121, local AS number 65001
BGP table version is 879001, L2VPN EVPN config peers 4, capable peers 4
356 network entries and 505 paths using 104744 bytes of memory
BGP attribute entries [77/13244], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
111.111.111.111 4 65001 80151 67103 879001 0 0 6w5d 118
112.112.112.112 4 65001 69159 64569 879001 0 0 6w5d 89
221.221.221.221 4 65002 130290 71161 879001 0 0 00:00:50 149
222.222.222.222 4 65002 130307 71155 879001 0 0 00:00:05 149

Site 2

show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 221.221.221.221, local AS number 65002
BGP table version is 52220, L2VPN EVPN config peers 4, capable peers 4
149 network entries and 149 paths using 36356 bytes of memory
BGP attribute entries [36/6192], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
121.121.121.121 4 65001 48978 46551 52220 0 0 00:01:34 0
122.122.122.122 4 65001 48986 46548 52220 0 0 00:01:16 0
211.211.211.211 4 65002 48500 40812 52220 0 0 4w1d 92
212.212.212.212 4 65002 42174 39843 52220 0 0 4w1d 57

show run bgp

router bgp 65002
router-id 221.221.221.221
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
template peer INTER-BGP-PEER
remote-as 65001
update-source loopback0
ebgp-multihop 10
address-family ipv4 unicast
send-community
send-community extended
address-family l2vpn evpn
send-community
send-community extended
route-map NH-Unchanged out
template peer INTRA-BGP-PEER
remote-as 65002
update-source loopback0
address-family ipv4 unicast
send-community
send-community extended
route-reflector-client
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 121.121.121.121
inherit peer INTER-BGP-PEER
remote-as 65001
neighbor 122.122.122.122
inherit peer INTER-BGP-PEER
remote-as 65001
neighbor 211.211.211.211
inherit peer INTRA-BGP-PEER
neighbor 212.212.212.212
inherit peer INTRA-BGP-PEER

show run bgp

router bgp 65001
router-id 121.121.121.121
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
template peer INTER-BGP-PEER
remote-as 65002
update-source loopback0
ebgp-multihop 10
address-family ipv4 unicast
send-community
send-community extended
address-family l2vpn evpn
send-community
send-community extended
route-map NH-Unchanged out
template peer INTRA-BGP-PEER
remote-as 65001
update-source loopback0
address-family ipv4 unicast
send-community
send-community extended
route-reflector-client
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 111.111.111.111
inherit peer INTRA-BGP-PEER
neighbor 112.112.112.112
inherit peer INTRA-BGP-PEER
neighbor 221.221.221.221
inherit peer INTER-BGP-PEER
remote-as 65002
neighbor 222.222.222.222
inherit peer INTER-BGP-PEER
remote-as 65002

Software Version 

BIOS: version 07.69

NXOS: version 9.3(8)

Any help would be appreciated. Like i say it will just fix its self without any changes but once its down i cant seem to get the peers to form. 

Thanks

[MHM Cisco World]

show sockets connection tcp foreign x.x.x.x detail <- this commend give the MSS/MTU use for BGP TCP connection 

[MHM Cisco World]

Ok, we check now that along path the mtu is 9000 that perfect,

Now only check that bgp tcp use mss same as mtu.

[LewisD1]

I have ran that. Looks to be using 9164.

show sockets connection tcp foreign 121.121.121.121 detail

Total number of netstack tcp sockets: 8
Active connections (including servers)
[host]: Local host: 222.222.222.222 (639), Foreign host: 121.121.121.121 (59219)
Protocol: tcp, type: stream, ttl: 64, tos: 0xc0, Id: 16
Id: 16
Options: REUSEADR, pcb flags none, state: | NBIO
Receive buffer:
cc: 0, hiwat: 16768, lowat: 1, flags: SEL
Send buffer:
cc: 0, hiwat: 16768, lowat: 2048, flags: none
Sequence number state:
iss: 441971737, snduna: 441985568, sndnxt: 441985568, sndwnd: 16768
irs: 3339910681, rcvnxt: 3339924512, rcvwnd: 16768, sndcwnd: 1048
out-of-order packet parameters:
received: 0, dropped: 0 reassembly queue count: 0
Timing parameters:
srtt: 3900 ms, rtt: 0 ms, rttv: 0 ms, krtt: 1000 ms
rttmin: 1000 ms, mss: 536, duration: 262963900 ms
State: ESTABLISHED
Flags: | RCVD_TSTMP
No MD5 peers Context: default

Kernel Socket Connection:
State Recv-Q Send-Q Local Address:Port Peer Address:Port

ESTAB 0 0 222.222.222.222:179 121.121.121.121:24249 ino:4025825 sk:1 <->

skmem:(r16640,rb262144,t0,tb262144,f3840,w0,o0,bl0) ts sack cubic wscale:2,2 rto:206.666 rtt:6.331/7.821 ato:40 mss:9164 cwnd:10 bytes_acked:8127 bytes_received:105 send 115.8Mb
ps lastsnd:42056 lastrcv:171030 lastack:42056 pacing_rate 231.6Mbps rcv_rtt:223.333 rcv_space:27492

show sockets connection tcp foreign 222.222.222.222 detail

Total number of netstack tcp sockets: 8
Active connections (including servers)
[host]: Local host: 121.121.121.121 (59219), Foreign host: 222.222.222.222 (639)
Protocol: tcp, type: stream, ttl: 64, tos: 0xc0, Id: 21
Id: 21
Options: REUSEADR, pcb flags unknown, state: | NBIO
Receive buffer:
cc: 0, hiwat: 16768, lowat: 1, flags: SEL
Send buffer:
cc: 0, hiwat: 16768, lowat: 2048, flags: none
Sequence number state:
iss: 3339910681, snduna: 3339924509, sndnxt: 3339924509, sndwnd: 16768
irs: 441971737, rcvnxt: 441985565, rcvwnd: 16768, sndcwnd: 1048
out-of-order packet parameters:
received: 0, dropped: 0 reassembly queue count: 0
Timing parameters:
srtt: 3900 ms, rtt: 0 ms, rttv: 0 ms, krtt: 1000 ms
rttmin: 1000 ms, mss: 536, duration: 261761400 ms
State: ESTABLISHED
Flags: | SENDCCNEW
No MD5 peers Context: default

Kernel Socket Connection:
State Recv-Q Send-Q Local Address:Port Peer Address:Port

ESTAB 0 10726 121.121.121.121:24249 222.222.222.222:179 timer:(on,17sec,8) ino:75526319 sk:2e <->

skmem:(r0,rb262144,t0,tb262144,f7424,w49920,o0,bl0) ts sack cubic wscale:2,2 rto:52906.7 backoff:8 rtt:6.549/8.971 ato:40 mss:9164 cwnd:1 ssthresh:7 bytes_acked:106 bytes_receiv
ed:8108 send 11.2Mbps lastsnd:35303 lastrcv:20383 lastack:20383 pacing_rate 44.8Mbps unacked:2 retrans:1/5 lost:1 sacked:1 rcv_rtt:1016.67 rcv_space:27528

This could be the issue. 

ping 222.222.222.222 source 121.121.121.121 packet-size 9163 df-bit
PING 222.222.222.222 (222.222.222.222) from 121.121.121.121: 9163 data bytes
Request 0 timed out
^C
--- 222.222.222.222 ping statistics ---
2 packets transmitted, 0 packets received, 100.00% packet loss
QCLOUD-S12-1# ping 222.222.222.222 source 121.121.121.121 packet-size 9164 df-bit
PING 222.222.222.222 (222.222.222.222) from 121.121.121.121: 9164 data bytes
Request 0 timed out
^C
--- 222.222.222.222 ping statistics ---
2 packets transmitted, 0 packets received, 100.00% packet loss
QCLOUD-S12-1# ping 122.122.122.122 source 121.121.121.121 packet-size 9164 df-bit
PING 122.122.122.122 (122.122.122.122) from 121.121.121.121: 9164 data bytes
9172 bytes from 122.122.122.122: icmp_seq=0 ttl=253 time=1.645 ms
9172 bytes from 122.122.122.122: icmp_seq=1 ttl=253 time=1.243 ms
9172 bytes from 122.122.122.122: icmp_seq=2 ttl=253 time=1.193 ms
9172 bytes from 122.122.122.122: icmp_seq=3 ttl=253 time=1.191 ms
9172 bytes from 122.122.122.122: icmp_seq=4 ttl=253 time=1.213 ms

Can we control what MTU BGP is using? 

Thanks

[MHM Cisco World]

mss: 536<<-- WoW this is so so small.
anyway 
disable PMTU Discovery since it can may be reduce the MSS 
config IP TCP MSS under the interface the peer connect to each other.
hope this solve your issue friend.

[LewisD1]

In this line it states 9164.

Kernel Socket Connection:
State Recv-Q Send-Q Local Address:Port Peer Address:Port

ESTAB 0 10726 121.121.121.121:24249 222.222.222.222:179 timer:(on,17sec,8) ino:75526319 sk:2e <->

skmem:(r0,rb262144,t0,tb262144,f7424,w49920,o0,bl0) ts sack cubic wscale:2,2 rto:52906.7 backoff:8 rtt:6.549/8.971 ato:40 mss:9164 cwnd:1 ssthresh:7 bytes_acked:106 bytes_receiv
ed:8108 send 11.2Mbps lastsnd:35303 lastrcv:20383 lastack:20383 pacing_rate 44.8Mbps unacked:2 retrans:1/5 lost:1 sacked:1 rcv_rtt:1016.67 rcv_space:27528

I am using BGP Multihop for these connections.

SPINE 1 ---> LEAF 1 -----> LEAF 2 ----> SPINE 2
Would i need to run ip tcp mss on the the connection from spine 1 to leaf 1 or from Leaf 1 to leaf 2? 

Thanks

Lewis

[MHM Cisco World · Accepted solution]

both peer mismatch MTU and this make huge CPU utilize and make router stuck,
so please config MSS for tcp
and for spine and leaf, first try config TCP under the LO (which you use as update source), if not success 
then you need to config ip tcp mss in underlaying interface connect leaf to spine (there is no direct connect between leafs).

check again with 

show sockets connection tcp foreign

to see that new mtg value is accept or not.

[LewisD1]

I have ran ip tcp mss on each of the switches. Its not a interface command its a switch command. 

I have set the tcp mss to 9000 on all the switches leaving a little overhead with the maximum MTU size. 

The Peers have now formed and are staying up. The MSS is still saying 536 in the first section but it now showing a lower MSS that previous in the second section when running the show socket connection tcp foreign detail command.

show socket connection tcp foreign 222.222.222.222 detail

Total number of netstack tcp sockets: 8
Active connections (including servers)
[host]: Local host: 121.121.121.121 (59219), Foreign host: 222.222.222.222 (639)
Protocol: tcp, type: stream, ttl: 64, tos: 0xc0, Id: 21
Id: 21
Options: REUSEADR, pcb flags unknown, state: | NBIO
Receive buffer:
cc: 0, hiwat: 16768, lowat: 1, flags: SEL
Send buffer:
cc: 0, hiwat: 16768, lowat: 2048, flags: none
Sequence number state:
iss: 3339910681, snduna: 3339927572, sndnxt: 3339927572, sndwnd: 16768
irs: 441971737, rcvnxt: 441988628, rcvwnd: 16768, sndcwnd: 1048
out-of-order packet parameters:
received: 0, dropped: 0 reassembly queue count: 0
Timing parameters:
srtt: 3900 ms, rtt: 0 ms, rttv: 0 ms, krtt: 1000 ms
rttmin: 1000 ms, mss: 536, duration: 319708300 ms
State: ESTABLISHED
Flags: | SENDCCNEW
No MD5 peers Context: default

Kernel Socket Connection:
State Recv-Q Send-Q Local Address:Port Peer Address:Port

ESTAB 0 10411 121.121.121.121:179 222.222.222.222:35415 timer:(on,13sec,8) ino:76417014 sk:31 <->

skmem:(r0,rb262144,t0,tb262144,f3328,w49920,o0,bl0) ts sack cubic wscale:2,2 rto:52053.3 backoff:8 rtt:2.957/1.264 ato:40 mss:9086 cwnd:1 ssthresh:7 bytes_acked:105 bytes_receiv
ed:7693 send 24.6Mbps lastsnd:38640 lastrcv:30640 lastack:30640 pacing_rate 98.3Mbps unacked:2 retrans:1/9 lost:1 sacked:1 rcv_rtt:973.333 rcv_space:27258

[MHM Cisco World]

ping 222.222.222.222 source 121.121.121.121 packet-size 8500 df-bit <- use IP see in BGP not other IP.

[MHM Cisco World]

Hi friend,
I spent whole last night think about your issue, 
the BGP table revision is increase rapidly...
so If we check the MTU and their is no mismatch then 
route oscillation
route oscillation is happened when the Peer learn one prefix and next-hop is  learn also from BGP not from IGP, here the Router will enter in LOOP of select best path, 
how I detect it ?
please select one prefix from show ip bgp
do
show ip route <prefix>
show ip bgp <prefix>
after some time 
show ip route <prefix>
show ip bgp <prefix>

if there the next-hop is oscillation then 
show ip bgp <next-hop>
see if the next-hop is learn from BGP or it learn from IGP.

this indicate for Best path LOOP.

[LewisD1]

Would route oscillation cause the BGP relationships to come down?

 ping 222.222.222.222 source 121.121.121.121 packet-size 8000 df-bit
PING 222.222.222.222 (222.222.222.222) from 121.121.121.121: 8000 data bytes
8008 bytes from 222.222.222.222: icmp_seq=0 ttl=252 time=3.741 ms
8008 bytes from 222.222.222.222: icmp_seq=1 ttl=252 time=3.327 ms
8008 bytes from 222.222.222.222: icmp_seq=2 ttl=252 time=3.289 ms
8008 bytes from 222.222.222.222: icmp_seq=3 ttl=252 time=3.321 ms
8008 bytes from 222.222.222.222: icmp_seq=4 ttl=252 time=3.324 ms

[MHM Cisco World]

the route oscillation can increase cpu utilize and hence make router not send keepalive in regular interval.