Buy or Renew
Buy or Renew
Duo Security forums now LIVE! Get answers to all your Duo Security questions. Learn more
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
12563
Views
0
Helpful
16
Replies

FTD high availability Standby failed

Vishnu_RR
Level 1
Level 1

‎02-05-2021 09:55 AM

Hi team,

The FMC is generating the alert like below.

SECONDARY (xxxxxxxx) FAILOVER_STATE_STANDBY_FAILED (Check peer event for reason)

Both FTD 9300  are in HA over a port-channel. I have checked both port-channel physical interfaces are in matching the configuration.

The FTD1 is active and FTD2 is standby. when i checked in FTD chassis manager about HA-ROLE are as per expectations.

 

FTD1 Image

active.png

 

FTD2 image

standby.png

 

 

Please help me to fix this issue.

 

 

0 Helpful
16 Replies 16

balaji.bandi
Hall of Fame
Hall of Fame

‎02-05-2021 10:28 AM

Do you see any Service effect, if yes, raise TAC case immediate responce to fix.

 

If not Service effected, please provide version of FTD code running along with below output form both the kits :

 

show failover
show failover history [ details]
show failover state
show failover statistics
show monitor-interface

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

0 Helpful

Vishnu_RR
Level 1
Level 1
In response to balaji.bandi

‎02-05-2021 11:10 AM

Hi,

 

Please find the below config

Active FTD1

> show failover
Failover On
Failover unit Primary
Failover LAN Interface: Failover Port-channel12 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 7 of 1041 maximum
MAC Address Move Notification Interval not set
failover replication http
Version: Ours 9.12(2)33, Mate 9.12(2)33
Serial Number: Ours xxxxxxxx, Mate yyyyyyyy
Last Failover at: 16:34:53 IST Jan 28 2021
This host: Primary - Active
Active time: 718082 (sec)
slot 0: UCSB-B200-M3-U hw/sw rev (0.0/9.12(2)33) status (Up Sys)
Interface servers (10.1.1.1): Normal (Monitored)
Interface ext-mgmt (10.1.1.17): Normal (Monitored)
Interface nonprod (10.1.1.33): Normal (Monitored)
Interface prod (10.1.1.49): Normal (Monitored)
Interface mgmt (10.1.1.65): Normal (Waiting)
Interface SERVICES (10.1.10.1): Normal (Monitored)
Interface diagnostic (0.0.0.0): Normal (Waiting)
slot 1: snort rev (1.0) status (up)
slot 2: diskstatus rev (1.0) status (up)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: UCSB-B200-M3-U hw/sw rev (0.0/9.12(2)33) status (Up Sys)
Interface servers (10.1.1.2): Normal (Monitored)
Interface ext-mgmt (10.1.1.18): Normal (Monitored)
Interface nonprod (10.1.1.34): Normal (Monitored)
Interface prod (10.1.1.50): Normal (Monitored)
Interface mgmt (10.1.1.66): Normal (Waiting)
Interface SERVICES (10.1.10.2): Normal (Monitored)
Interface diagnostic (0.0.0.0): Normal (Waiting)
slot 1: snort rev (1.0) status (up)
slot 2: diskstatus rev (1.0) status (up)
Stateful Failover Logical Update Statistics
Link : Failover Port-channel12 (up)
Stateful Obj xmit xerr rcv rerr
General 12111023 9 73127 0
sys cmd 73127 0 73127 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 2500330 0 0 0
UDP conn 948501 0 0 0
ARP tbl 51685 0 0 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 577 0 0 0
Router ID 0 0 0 0
User-Identity 2 0 0 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0
Umbrella Device-ID 0 0 0 0
Rule DB B-Sync 2 0 0 0
Rule DB P-Sync 206 0 0 0
Rule DB Delete 4 9 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 10 73127
Xmit Q: 0 16 12408156
> show failover history
details Show failover switching history of Current and Peer unit
| Output modifiers
<cr>

> show failover history details
==========================================================================
From State To State Reason
==========================================================================
16:34:05 IST Jan 28 2021
Not Detected Disabled No Error

16:34:07 IST Jan 28 2021
Disabled Negotiation Set by the config command

16:34:53 IST Jan 28 2021
Negotiation Just Active No Active unit found

16:34:53 IST Jan 28 2021
Just Active Active Drain No Active unit found

16:34:53 IST Jan 28 2021
Active Drain Active Applying Config No Active unit found

16:34:53 IST Jan 28 2021
Active Applying Config Active Config Applied No Active unit found

16:34:53 IST Jan 28 2021
Active Config Applied Active No Active unit found

==========================================================================
PEER History Collected at 00:04:17 IST Feb 6 2021
===========================PEER-HISTORY===================================
From State To State Reason
===========================PEER-HISTORY===================================
23:48:49 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:48:54 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

23:51:34 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:51:39 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

23:52:34 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:52:39 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

23:57:34 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:57:39 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

23:59:04 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:59:09 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:00:19 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:00:24 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:01:04 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:01:09 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:01:34 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:01:39 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:02:49 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:02:54 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:03:34 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0
00:03:39 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0
===========================PEER-HISTORY===================================
> show failover state

State Last Failure Reason Date/Time
This host - Primary
Active Ifc Failure 19:44:35 IST Jan 30 2021
Other host - Secondary
Standby Ready Ifc Failure 00:04:19 IST Feb 6 2021
mgmt: Failed
====Configuration State===
Sync Done
====Communication State===
Mac set

> show failover statistics
tx:1664972
rx:1430881
> show monitor-interface
This host: Primary - Active
Interface servers (10.1.1.1): Normal (Monitored)
Interface ext-mgmt (10.1.1.17): Normal (Monitored)
Interface nonprod (10.1.1.33): Normal (Monitored)
Interface prod (10.1.1.49): Normal (Monitored)
Interface mgmt (10.1.1.65): Normal (Waiting)
Interface SERVICES (10.1.10.1): Normal (Monitored)
Interface diagnostic (0.0.0.0): Normal (Waiting)
Other host: Secondary - Standby Ready
Interface servers (10.1.1.2): Normal (Monitored)
Interface ext-mgmt (10.1.1.18): Normal (Monitored)
Interface nonprod (10.1.1.34): Normal (Monitored)
Interface prod (10.1.1.50): Normal (Monitored)
Interface mgmt (10.1.1.66): Normal (Waiting)
Interface SERVICES (10.1.10.2): Normal (Monitored)
Interface diagnostic (0.0.0.0): Normal (Waiting)

 

STANDBY FTD2

show failover
Failover On
Failover unit Secondary
Failover LAN Interface: Failover Port-channel12 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 7 of 1041 maximum
MAC Address Move Notification Interval not set
failover replication http
Version: Ours 9.12(2)33, Mate 9.12(2)33
Serial Number: Ours yyyyyyyy, Mate xxxxxxxx
Last Failover at: 05:35:23 IST Jan 1 2010
This host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: UCSB-B200-M3-U hw/sw rev (0.0/9.12(2)33) status (Up Sys)
Interface servers (10.1.1.2): Normal (Monitored)
Interface ext-mgmt (10.1.1.18): Normal (Monitored)
Interface nonprod (10.1.1.34): Normal (Monitored)
Interface prod (10.1.1.50): Normal (Monitored)
Interface mgmt (10.1.1.66): Normal (Waiting)
Interface SERVICES (10.1.10.2): Normal (Monitored)
Interface diagnostic (0.0.0.0): Normal (Waiting)
slot 1: snort rev (1.0) status (up)
slot 2: diskstatus rev (1.0) status (up)
Other host: Primary - Active
Active time: 718219 (sec)
slot 0: UCSB-B200-M3-U hw/sw rev (0.0/9.12(2)33) status (Up Sys)
Interface servers (10.1.1.1): Normal (Monitored)
Interface ext-mgmt (10.1.1.17): Normal (Monitored)
Interface nonprod (10.1.1.33): Normal (Monitored)
Interface prod (10.1.1.49): Normal (Monitored)
Interface mgmt (10.1.1.65): Normal (Waiting)
Interface SERVICES (10.1.10.1): Normal (Monitored)
Interface diagnostic (0.0.0.0): Normal (Waiting)
slot 1: snort rev (1.0) status (up)
slot 2: diskstatus rev (1.0) status (up)

Stateful Failover Logical Update Statistics
Link : Failover Port-channel12 (up)
Stateful Obj xmit xerr rcv rerr
General 73145 0 12107647 98
sys cmd 73145 0 73145 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 2500146 96
UDP conn 0 0 9481866 0
ARP tbl 0 0 51701 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 0 0 575 2
Router ID 0 0 0 0
User-Identity 0 0 2 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0
Umbrella Device-ID 0 0 0 0
Rule DB B-Sync 0 0 2 0
Rule DB P-Sync 0 0 206 0
Rule DB Delete 0 0 4 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 15 12404853
Xmit Q: 0 1 73145
> show failover history details
==========================================================================
From State To State Reason
==========================================================================
23:51:34 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:51:39 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

23:52:34 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:52:39 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

23:57:34 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:57:39 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

23:59:04 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

23:59:09 IST Feb 5 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:00:19 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:00:24 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:01:04 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:01:09 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:01:34 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:01:39 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:02:49 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:02:54 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:03:34 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:03:39 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

00:04:19 IST Feb 6 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

00:04:24 IST Feb 6 2021
Failed Standby Ready Interface check
This host:0
Other host:0

==========================================================================
PEER History Collected at 00:06:35 IST Feb 6 2021
===========================PEER-HISTORY===================================
From State To State Reason
===========================PEER-HISTORY===================================
16:34:05 IST Jan 28 2021
Not Detected Disabled No Error

16:34:07 IST Jan 28 2021
Disabled Negotiation Set by the config command

16:34:53 IST Jan 28 2021
Negotiation Just Active No Active unit found

16:34:53 IST Jan 28 2021
Just Active Active Drain No Active unit found

16:34:53 IST Jan 28 2021
Active Drain Active Applying Config No Active unit found

16:34:53 IST Jan 28 2021
Active Applying Config Active Config Applied No Active unit found

16:34:53 IST Jan 28 2021
Active Config Applied Active No Active unit found

===========================PEER-HISTORY===================================
> show failover state

State Last Failure Reason Date/Time
This host - Secondary


Standby Ready Ifc Failure 00:06:46 IST Feb 6 2021
Other host - Primary
Active None

====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set

> show failover statistics
tx:1431163
rx:1608772
> show monitor-interface
This host: Secondary - Standby Ready
Interface servers (10.1.1.2): Normal (Monitored)
Interface ext-mgmt (10.1.1.18): Normal (Monitored)
Interface nonprod (10.1.1.34): Normal (Monitored)
Interface prod (10.1.1.50): Normal (Monitored)
Interface mgmt (10.1.1.66): Normal (Waiting)
Interface SHARED_SERVICES (10.1.10.2): Normal (Monitored)
Interface diagnostic (0.0.0.0): Normal (Waiting)
Other host: Primary - Active
Interface servers (10.1.1.1): Normal (Monitored)
Interface ext-mgmt (10.1.1.17): Normal (Monitored)
Interface nonprod (10.1.1.33): Normal (Monitored)
Interface prod (10.1.1.49): Normal (Monitored)
Interface mgmt (10.1.1.65): Normal (Waiting)
Interface SERVICES (10.1.10.1): Normal (Monitored)
Interface diagnostic (0.0.0.0): Normal (Waiting)

0 Helpful

balaji.bandi
Hall of Fame
Hall of Fame
In response to Vishnu_RR

‎02-05-2021 11:22 AM

investigate the below interface. you may get some information on the switch also where this connected both the side.

 

 

23:51:34 IST Feb 5 2021
Standby Ready Failed Interface check
This host:1
single_vf: mgmt
Other host:0

 

Interface mgmt (10.1.1.66): Normal (Waiting)
IInterface mgmt (10.1.1.65): Normal (Waiting)

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

0 Helpful

Marvin Rhoads
Hall of Fame
Hall of Fame

‎02-05-2021 10:31 AM

Start by checking from the FTD cli  "show failover history detail".

0 Helpful

MHM Cisco World
VIP
VIP

‎02-05-2021 06:33 PM

ip verify reverse-path interface

please check if this command is under the monitor interface 

0 Helpful

Vishnu_RR
Level 1
Level 1
In response to MHM Cisco World

‎02-05-2021 09:44 PM

Hi,

If I add this Interface mgmt to HA monitoring, this alert generated by FMC.
If I remove this Interface mgmt from HA monitoring, both device health status is normal.

I have verified IP verify reverse-path interface command not under monitor interface.

 

The mgmt is a sub interface of port-channel 10. the other sub interfaces prod, nonprod, servers, ext-mgmt, prod are part of same port-channel 10. There is no issue for these other sub interfaces.

 

Why the HA issue only for this mgmt interface when add to HA monitor. what should I check at switch side.

0 Helpful

Vishnu_RR
Level 1
Level 1
In response to Vishnu_RR

‎02-07-2021 12:54 AM

Hi team,

 

Please help me on this issue. is this a bug ?

 

0 Helpful

balaji.bandi
Hall of Fame
Hall of Fame
In response to Vishnu_RR

‎02-07-2021 03:22 AM

Not that I am aware of this is bug, since this MGMT interface not required to FTD failover condition as a use case (personally i think)

 

I will investigate the physical path and for the short term not to fall over due to this condition failed. remove mgmt Interface from monitoring until the issue fixed (short term solution)

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

0 Helpful

Vishnu_RR
Level 1
Level 1
In response to balaji.bandi

‎02-07-2021 07:08 PM

Hi,

This is not a OOB interface. It's one of the security zone. I forgot to edit this interface in the config. The security zone name is int-mgmt

Please correct in the configuration.
0 Helpful

balaji.bandi
Hall of Fame
Hall of Fame
In response to Vishnu_RR

‎02-08-2021 02:10 AM

So you need to track them to failover if that fails ? then look in the path where all L2/L3 not stretched.

 

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

0 Helpful

Vishnu_RR
Level 1
Level 1
In response to balaji.bandi

‎02-10-2021 04:09 AM

Hi,

The port channel interfaces are connected to ACI service leaf switch. any way the issue happening only for int-mgmt sub interface only. but other sub interfaces are on same port channel working fine and no issue for those sub interfaces.

0 Helpful

Marvin Rhoads
Hall of Fame
Hall of Fame
In response to Vishnu_RR

‎02-10-2021 04:37 AM

I suspect there may be undocumented behavior with respect to the management interface and failover. We don't typically monitor the physical management interface in HA pairs. You have defined a separate interface for management (a relatively new and uncommon setup) and, while the configuration guide doesn't say you cannot monitor it for failover purposes, your experience seems to indicate that doing so causes problems.

I would suggest opening a TAC case to confirm my suspicion. Otherwise just remove that particular interface from monitoring and everything should work as normal.

0 Helpful

Vishnu_RR
Level 1
Level 1
In response to Marvin Rhoads

‎02-10-2021 05:16 AM

Hi,

its not a OOB. There is separate data interface for other purpose. so the data interface name is int-mgmt. 

likewise there is a ext-mgmt also. ext-mgmt is a data interface.

 

 

0 Helpful

Marvin Rhoads
Hall of Fame
Hall of Fame
In response to Vishnu_RR

‎02-10-2021 05:24 AM

I understand that it is not the out-of-band management interface. However, is it a management interface?

Has the problem occurred since you renamed the interface from mgmt to int-mgmt?

0 Helpful
Customers Also Viewed These Support Documents
Review Cisco Networking products for a $25 gift card