eBGPv6 Flapping every 3 Minutes with BFD Configured on ASR1K and C9500

[bong31]

Having issue with eBGPv6 flapping. A few of the similar eBGPv6 sessions having this same issue of constant flapping. Ping is clean and no packet loss. IBGPv4 is stable. BFD key chain is the same on both ends.

Appreciate from you guys who can share the expertise(or have encountered this before) the soonest. Thank you in advance. Please find config and logs snip below.

A-END:

Configuration:

interface TenGigabitEthernet0/0/1.<vlan>
encapsulation dot1Q <vlan>
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1500
ipv6 address <ipv6>/127
ipv6 enable
cts manual
policy static sgt 500 trusted
bfd template <name>-bfd
end

#show run | s r b
router bgp <as>
bgp router-id <RID>
bgp log-neighbor-changes
bgp graceful-restart
no bgp default ipv4-unicast
.
.
neighbor <ipv6 neighbor> remote-as <as>
neighbor <ipv6 neighbor> fall-over bfd
.
.
address-family ipv6
redistribute connected
maximum-paths 2
neighbor <ipv6 neighbor> activate
neighbor <ipv6 neighbor> soft-reconfiguration inbound

LOGS:
# show log | in <ipv6 neighbor>
30716740: <date> 09:53:34.672 SGP: %BGP-3-NOTIFICATION: sent to neighbor <ipv6 neighbor> 4/0 (hold time expired) 0 bytes
30716741: <date> 09:53:34.672 SGP: %BGP-5-NBR_RESET: Neighbor <ipv6 neighbor> reset (BGP Notification sent)
30716742: <date> 09:53:34.673 SGP: %BGP-5-ADJCHANGE: neighbor <ipv6 neighbor> Down BGP Notification sent
30716743: <date> 09:53:34.674 SGP: %BGP_SESSION-5-ADJCHANGE: neighbor <ipv6 neighbor> IPv6 Unicast topology base removed from session BGP Notification sent
30716758: <date> 09:53:45.979 SGP: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh <ipv6 neighbor> proc:BGP, idb:TenGigabitEthernet0/0/1.3043 handle:53 act
30716759: <date> 09:53:45.979 SGP: %BGP-5-ADJCHANGE: neighbor <ipv6 neighbor> Up

#show log | in bfd
30717675: <date> 10:12:48.338 SGP: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh <ipv6 neighbor> proc:BGP, idb:TenGigabitEthernet0/0/1.3043 handle:59 act

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
<IPV6 neighbor>
4 <AS> 9 12 17541427 0 10 00:02:30 27

=========

B-END:

Configuration:
interface Vlan<vlan>
no ip redirects
carrier-delay msec 50
ipv6 address <IPv6>/127
ipv6 enable
no ipv6 redirects
bfd template <name>-bfd
end

# show run | s r b
router bgp <AS>
bgp router-id interface Loopback0
bgp log-neighbor-changes
bgp graceful-restart
no bgp default ipv4-unicast
.
.
neighbor <IPv6 neighbor> remote-as <AS>
neighbor <IPv6 neighbor> fall-over bfd
.
.
.
neighbor <IPv6 neighbor> activate
neighbor <IPv6 neighbor> advertisement-interval 0
neighbor <IPv6 neighbor> soft-reconfiguration inbound

LOGS:
#show log | in <IPv6 neighbor>
36666447: <date> 14:30:33.798 SGP: %BGP-3-NOTIFICATION: received from neighbor <IPv6 neighbor> 4/0 (hold time expired) 0 bytes
36666448: <date> 14:30:33.798 SGP: %BGP-5-NBR_RESET: Neighbor <IPv6 neighbor> reset (BGP Notification received)
36666449: <date> 14:30:33.798 SGP: %BGP-5-ADJCHANGE: neighbor <IPv6 neighbor> Down BGP Notification received
36666450: <date> 14:30:33.798 SGP: %BGP_SESSION-5-ADJCHANGE: neighbor <IPv6 neighbor> IPv6 Unicast topology base removed from session BGP Notification received
36666458: <date> 14:30:41.034 SGP: %BGP-5-ADJCHANGE: neighbor <IPv6 neighbor> Up
36666568: <date> 14:33:41.194 SGP: %BGP-3-NOTIFICATION: received from neighbor <IPv6 neighbor> 4/0 (hold time expired) 0 bytes
36666569: <date> 14:33:41.195 SGP: %BGP-5-NBR_RESET: Neighbor <IPv6 neighbor> reset (BGP Notification received)
36666570: <date> 14:33:41.195 SGP: %BGP-5-ADJCHANGE: neighbor <IPv6 neighbor> Down BGP Notification received
36666571: <date> 14:33:41.195 SGP: %BGP_SESSION-5-ADJCHANGE: neighbor <IPv6 neighbor> IPv6 Unicast topology base removed from session BGP Notification received
36666577: <date> 14:33:47.411 SGP: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh <IPv6 neighbor> proc:BGP, idb:Vlan<vlan> handle:91 act
36666578: <date> 14:33:47.411 SGP: %BGP-5-ADJCHANGE: neighbor <IPv6 neighbor> Up
#

#show log | in bfd

36666577: <date> 14:33:47.411 SGP: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh <IPv6 neighbor> proc:BGP, idb:Vlan<vlan> handle:91 act

#show bgp ipv6 unicast summary

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
<IPv6 neighbor>
4 <AS> 9 12 17541427 0 10 00:02:30 27

[MHM Cisco World]

What timer you use for bfd?

MHM

[bong31]

Hi MHM,

Thank you for prompt response.

Both ends has 'interval min-tx 100 min-rx 100 multiplier 3' configured

[MHM Cisco World]

Use 300/300/3

If the flapping not stop then it bug 

MHM

[paul driver]

Hello
looks like your carrier delay value is aggressive -  
Have you tried increasing or removing it 
Do you have any issues with the physical link
crc errors etc…

[bong31]

Hi Paul,

Haven't tried removing carrier delay value but may consider augmenting the value. Is there any higher value that you can suggest? Also, is removing/changing value service impacting?

Thank you.

[paul driver]

Hello
Apologies I may have misled you, When I suggested removing the carrier delay i meant to say remove that specific set value and let its default value to take, NOT to remove it entirely.
My thinking is if you wish for fast convergence then let BFD do its job and not tweek the hardware interface settings as well.

Do you see any physical error on the interfaces connecting this ospf adjacency, is the  reachability okay 

[Pavel Tarakanov]

Please check reachability between nodes with high packet size (ping with DF bit and packet size >1472). On which size ping failed?

Other option to troubleshoot it - collect capture of BGP traffic on both sides and compare, what kind of packets doesn't arrive to A-END.

[bong31]

Thank you for the inputs. Appreciated. Did check on some interfaces with BGPv6 and some don't have bfd configured but still flapping. Seems due to mtu mismatch as one end is hard-coded to 1500 while other end is default(jumbo 9000). Unable to replicate though in the lab.