BFD session ld:24 handle:2,is going Down Reason: ECHO FAILURE

BorislavPenchev0962 · ‎11-24-2022

Hi all,

We have two Core N7K running fine and on one of the C9407R we get these error logs few times a day, looks like osfp/bfd flapping but i cant detect the reason:

21:13:21.576: %BFDFSM-6-BFD_SESS_DOWN: BFD-SYSLOG: BFD session ld:24 handle:2,is going Down Reason: ECHO FAILURE
Nov 24 21:13:21.576: %BFD-6-BFD_SESS_DESTROYED: BFD-SYSLOG: bfd_session_destroyed, ld:24 neigh proc:OSPF, handle:2 act
Nov 24 21:13:21.576: %OSPF-5-ADJCHG: Process 243, Nbr 15.2.0.2 on TenGigabitEthernet2/3/0/1.689 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.578: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.2 on TenGigabitEthernet2/3/0/1.497 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.600: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.1 on TenGigabitEthernet1/3/0/1.404 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.638: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.2 on TenGigabitEthernet2/3/0/1.433 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.646: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.2 on TenGigabitEthernet2/3/0/1.405 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.647: %OSPF-5-ADJCHG: Process 242, Nbr 15.6.0.2 on TenGigabitEthernet2/3/0/1.529 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:21.647: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.1 on TenGigabitEthernet1/3/0/1.432 from FULL to DOWN, Neighbor Down: BFD node down
Nov 24 21:13:23.371: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.1 on TenGigabitEthernet1/3/0/1.432 from LOADING to FULL, Loading Done
Nov 24 21:13:23.371: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh 15.7.1.33 proc:OSPF, idb:TenGigabitEthernet1/3/0/1.432 handle:3 act
Nov 24 21:13:23.832: %OSPF-5-ADJCHG: Process 242, Nbr 15.6.0.2 on TenGigabitEthernet2/3/0/1.529 from LOADING to FULL, Loading Done
Nov 24 21:13:24.012: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.2 on TenGigabitEthernet2/3/0/1.497 from LOADING to FULL, Loading Done
Nov 24 21:13:25.269: %SW_MATM-4-MACFLAP_NOTIF: Host f8b4.aaa.aaa in vlan 244 is flapping between port Po81 and port Po142
Nov 24 21:13:26.317: %BFDFSM-6-BFD_SESS_UP: BFD-SYSLOG: BFD session ld:34 handle:3 is going UP
Nov 24 21:13:26.447: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.2 on TenGigabitEthernet2/3/0/1.433 from LOADING to FULL, Loading Done
Nov 24 21:13:26.801: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.1 on TenGigabitEthernet1/3/0/1.404 from LOADING to FULL, Loading Done
Nov 24 21:13:28.004: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.2 on TenGigabitEthernet2/3/0/1.405 from LOADING to FULL, Loading Done
Nov 24 21:13:28.165: %OSPF-5-ADJCHG: Process 243, Nbr 15.2.0.2 on TenGigabitEthernet2/3/0/1.689 from LOADING to FULL, Loading Done



Nov 25 12:44:13.249: %BFDFSM-6-BFD_SESS_DOWN: BFD-SYSLOG: BFD session ld:24 handle:2,is going Down Reason: ECHO FAILURE
Nov 25 12:44:13.249: %BFD-6-BFD_SESS_DESTROYED: BFD-SYSLOG: bfd_session_destroyed, ld:24 neigh proc:OSPF, handle:2 act
Nov 25 12:44:13.249: %OSPF-5-ADJCHG: Process 243, Nbr 15.2.0.2 on TenGigabitEthernet2/3/0/1.689 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.261: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.1 on TenGigabitEthernet1/3/0/1.432 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.264: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.1 on TenGigabitEthernet1/3/0/1.496 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.292: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.2 on TenGigabitEthernet2/3/0/1.497 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.372: %OSPF-5-ADJCHG: Process 242, Nbr 15.6.0.2 on TenGigabitEthernet2/3/0/1.529 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.375: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.2 on TenGigabitEthernet2/3/0/1.405 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.405: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.1 on TenGigabitEthernet1/3/0/1.404 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.410: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.2 on TenGigabitEthernet2/3/0/1.433 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:13.431: %OSPF-5-ADJCHG: Process 254, Nbr 10.254.0.2 on TenGigabitEthernet2/3/0/1.561 from FULL to DOWN, Neighbor Down: BFD node down
Nov 25 12:44:14.086: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.2 on TenGigabitEthernet2/3/0/1.497 from LOADING to FULL, Loading Done
Nov 25 12:44:14.086: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh 15.4.1.49 proc:OSPF, idb:TenGigabitEthernet2/3/0/1.497 handle:7 act
Nov 25 12:44:14.501: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.2 on TenGigabitEthernet2/3/0/1.405 from LOADING to FULL, Loading Done
Nov 25 12:44:15.052: %OSPF-5-ADJCHG: Process 243, Nbr 15.2.0.2 on TenGigabitEthernet2/3/0/1.689 from LOADING to FULL, Loading Done
Nov 25 12:44:15.414: %BFDFSM-6-BFD_SESS_UP: BFD-SYSLOG: BFD session ld:24 handle:2 is going UP
Nov 25 12:44:16.351: %OSPF-5-ADJCHG: Process 238, Nbr 15.5.0.1 on TenGigabitEthernet1/3/0/1.404 from LOADING to FULL, Loading Done
Nov 25 12:44:17.375: %OSPF-5-ADJCHG: Process 242, Nbr 15.6.0.2 on TenGigabitEthernet2/3/0/1.529 from LOADING to FULL, Loading Done
Nov 25 12:44:17.768: %OSPF-5-ADJCHG: Process 241, Nbr 15.4.0.1 on TenGigabitEthernet1/3/0/1.496 from LOADING to FULL, Loading Done
Nov 25 12:44:21.335: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.1 on TenGigabitEthernet1/3/0/1.432 from LOADING to FULL, Loading Done
Nov 25 12:44:21.335: %BFD-6-BFD_SESS_CREATED: BFD-SYSLOG: bfd_session_created, neigh 15.7.1.33 proc:OSPF, idb:TenGigabitEthernet1/3/0/1.432 handle:3 act
Nov 25 12:44:21.743: %OSPF-5-ADJCHG: Process 254, Nbr 10.254.0.2 on TenGigabitEthernet2/3/0/1.561 from LOADING to FULL, Loading Done
Nov 25 12:44:22.639: %BFDFSM-6-BFD_SESS_UP: BFD-SYSLOG: BFD session ld:34 handle:3 is going UP
Nov 25 12:44:22.703: %OSPF-5-ADJCHG: Process 239, Nbr 15.7.0.2 on TenGigabitEthernet2/3/0/1.433 from LOADING to FULL, Loading Done

is this flapping issue? we checked cabling looks fine; CPU is max 30-50%;

Best Regards

MHM Cisco World · ‎11-24-2022

can I see BFD config ?

BorislavPenchev0962 · ‎11-25-2022

let me know what part of it ? here example interface config:

 interface Ethernet8/33.599
bandwidth 10000000
  encapsulation dot1q 333
  vrf member AA-BB
  bfd interval 250 min_rx 250 multiplier 3
  bfd echo-rx-interval 50
  bfd ipv4 interval 50 min_rx 50 multiplier 3
  bfd ipv6 interval 50 min_rx 50 multiplier 3
  bfd ipv4 echo-rx-interval 50
  bfd ipv6 echo-rx-interval 50
  no ip redirects
  ip address a.b.c.d/28
  ip ospf authentication message-digest
  ip ospf message-digest-key 1 md5
  ip ospf network point-to-point
  no ip ospf passive-interface
  ip router ospf 1 area 
  no shutdown

MHM Cisco World · ‎11-25-2022

can you increase the BFD time into 300 300 3 and check again?

  bfd ipv4 interval 50 min_rx 50 multiplier 3

BorislavPenchev0962 · ‎11-27-2022

we will try it, do we need to schedule maintenance in advance as this is production switch ?

we see these errors twice a day and mainly on the above two interfaces while we have many others configured the same way ?

MHM Cisco World · ‎11-28-2022

so it not new config ? if yes ? let me check.
I notice this log message, can you draw topology and more info. about vlan 244 ??
%SW_MATM-4-MACFLAP_NOTIF: Host f8b4.aaa.aaa in vlan 244 is flapping between port Po81 and port Po142 Nov 24 21:13:26.317: %BFDFSM-6-BFD_SESS_UP: BFD-SYSLOG: BFD session ld:34 handle:3 is going UP

paul driver · ‎11-28-2022

Hello
Suggest to disable BFD on the interface and monitor the peering without fast detection,

interface Ethernet8/33.599
ip ospf bfd disable

if your peering is then stable re-enable BFD once again and then set the interface to just have the single tranmsit/receive interval.

interface Ethernet8/33.599
no bfd echo-rx-interval 50
no bfd ipv4 interval 50 min_rx 50 multiplier 3
no bfd ipv6 interval 50 min_rx 50 multiplier 3
no bfd ipv4 echo-rx-interval 50
no bfd ipv6 echo-rx-interval 50
bfd interval 50 min_rx 50 multiplier 3
ip ospf bfd enable

Note: The above changes should not effect the current ospf peering

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Georg Pauwen · ‎11-28-2022

Hello,

is your network actually experiencing outages (at the times indicated in your log files) ? Can you post the output of 'show bfd neighbors detail' ?

It might just be the 'bfd echo-rx-interval 50' being too low. You could try and set it to something higher (such as 250, or 500). Obviously, since this is a production environment, do this after hours.

BorislavPenchev0962 · ‎12-12-2022

we have found that there were messages

TenGigabitEthernet1/3/0/1.404 from FULL to DOWN, Neighbor Down: BFD node down

ECHO FAILURE was the reason hence we entered:

no bfd echo - on all interfaces where we seen this and since then all works fine

MHM Cisco World · ‎12-12-2022

I will run lab I have found some doc. about BFD in NSK.
if I found something I will update you.

MHM Cisco World · ‎12-13-2022

Cisco Nexus 9000 Series NX-OS Interfaces Configuration Guide, Release 6.x - Configuring Bidirectional Forwarding Detection [Cisco Nexus 9000 Series Switches] - Cisco

SVI limitations:

An ASIC reset causes traffic disruption for other ports and it can cause the SVI sessions on the other ports to flap. For example, if the carrier interface is a virtual port channel (vPC), BFD is not supported over the SVI interface and it could cause a trigger for an ASIC reset. When a BFD session is over SVI using virtual port channel (vPC) Peer-Link, the BFD echo function is not supported. You must disable the BFD echo function for all sessions over SVI between vPC peer nodes.
An SVI on the Cisco Nexus series switches should not be configured to establish a BFD neighbor adjacency with a device connected to it via a vPC. This is because the BFD keepalives from the neighbour, if sent over the vPC member link connected to the vPC peer-switch, do not reach this SVI causing the BFD adjacency to fail.
When you change the topology (for example, add or delete a link into a VLAN, delete a member from a Layer 2 port channel, and so on), the SVI session could be affected. It may go down first and then come up after the topology discovery is finished.
BFD over FEX HIF interfaces is not supported.
When a BFD session is over SVI using virtual port-channel (vPC) Peer-Link (either BCM or GEM based ports), the BFD echo function is not supported. You must disable the BFD echo function for all sessions over SVI between vPC peer nodes using the no bfd echo command at the SVI configuration level.

according to Cisco Doc. BFD have limitation in SVI, are you config BFD over SVI ??

rdwived2 · ‎08-27-2024

I believe we need to rectify here little bit, BFD over SVI is supported config but over vPC Peer link (vPC Peer devices and with member ports its not supported.

Limitations of the IETF Bidirectional Forwarding Detection

See the following limitations of the IETF Bidirectional Forwarding Detection:

BFD Limitations
- It cannot co-exist with BFD over logical port-channels or proprietary BFD per-member links. BFD IPv6 logical/proprietary per-link session is also not supported when BFD IETF IPv4 is configured on PC.
- When you configure logical BFD session under any routing protocol, make sure that is not applied to any IETF port-channel. Having both logical and IETF configuration for same port-channel results in undefined behavior during ISSU/reloads.
- IETF BFD IPv6 is not supported.
- Echo functionality is not supported for Micro-BFD sessions.
- Port-channel interfaces should be directly connected between two switches that are running the BFD sessions. No intermediate Layer 2 switches are expected.

EthPCM/LACP Limitations
- If a LACP port-channel has members in hot-standby state, BFD failure in one of the active links may not cause the hot-standby link to come up directly. Once the active link with BFD failure goes down, the hot-standby member becomes active. However, it may not be able to prevent the port-channel from going down before the hot-standby link comes up, in cases where port-channel min-link condition is hit.

General Limitations:
- It is supported only on Layer 3 port-channels.
- It is not supported on the following:
  - vPC
  - Layer 3 sub-interfaces
  - Layer 2 port-channels/Layer 2 Fabric Path
  - FPC/HIF PC
  - Layer 3 sub-interfaces
  - SVI over port-channels