ASA A/S Pair keeps failing over

davebornack · ‎03-28-2013

We keep getting failover syslog messages.

Level: Alert 
Date: 2013-03-22 05:18:48
Host: 10.20.12.250
Message:
%ASA-1-105005: (Secondary) Lost Failover communications with mate on interface inside

Level: Alert 
Date: 2013-03-22 05:18:48
Host: 10.20.12.250
Message:
%ASA-1-105008: (Secondary) Testing Interface inside

ETC....

We never lose connectivity, which is great, as that's how it should work, but things keep failing. Cisco hasn't been able to figure it out, and sent us a replacement unit, which unfortunately didn't do the trick.

Here are some outputs.. which interestingly enough, don't show any fail since Mar 21, but we're getting syslog messages that are indicative of a failover.

Here is syslog from this morning's fialover.. it's odd that it's telling me about VPN and WEBVPN interfaces.....

2013-03-28	08:23:25	Local4	Debug	10.20.12.250	%ASA-7-720042: (VPN-Primary) Receiving Command Link Bulk Sync message (Command 4) from active unit
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-721002: (WebVPN-Primary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Standby Ready.
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-720028: (VPN-Primary) HA status callback: Peer state Standby Ready.
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-720032: (VPN-Primary) HA status callback: id=3,seq=200,grp=0,event=406,op=80,my=Active,peer=Standby Ready.
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-721002: (WebVPN-Primary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Failed.
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-720028: (VPN-Primary) HA status callback: Peer state Failed.
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-720032: (VPN-Primary) HA status callback: id=3,seq=200,grp=0,event=406,op=20,my=Active,peer=Failed.
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-721002: (WebVPN-Primary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Standby Ready.
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-720028: (VPN-Primary) HA status callback: Peer state Standby Ready.
2013-03-28	08:23:25	Local4	Info	10.20.12.250	%ASA-6-720032: (VPN-Primary) HA status callback: id=3,seq=200,grp=0,event=406,op=80,my=Active,peer=Standby Ready.

NY5ASADAT01# sho fail hist

==========================================================================

From State To State Reason

==========================================================================

17:46:07 CDT Mar 20 2013

Active Cold Standby Failover state check

17:46:08 CDT Mar 20 2013

Cold Standby Sync Config Failover state check

17:46:15 CDT Mar 20 2013

Sync Config Sync File System Failover state check

17:46:15 CDT Mar 20 2013

Sync File System Bulk Sync Failover state check

17:46:27 CDT Mar 20 2013

Bulk Sync Standby Ready Failover state check

07:52:39 CDT Mar 21 2013

Standby Ready Just Active Other unit wants me Active

07:52:39 CDT Mar 21 2013

Just Active Active Drain Other unit wants me Active

07:52:39 CDT Mar 21 2013

Active Drain Active Applying Config Other unit wants me Active

07:52:39 CDT Mar 21 2013

Active Applying Config Active Config Applied Other unit wants me Active

07:52:39 CDT Mar 21 2013

Active Config Applied Active Other unit wants me Active

13:18:59 CDT Mar 21 2013

Active Standby Ready Set by the config command

14:22:27 CDT Mar 21 2013

Standby Ready Just Active Other unit wants me Active

14:22:27 CDT Mar 21 2013

Just Active Active Drain Other unit wants me Active

14:22:27 CDT Mar 21 2013

Active Drain Active Applying Config Other unit wants me Active

14:22:27 CDT Mar 21 2013

Active Applying Config Active Config Applied Other unit wants me Active

14:22:27 CDT Mar 21 2013

Active Config Applied Active Other unit wants me Active

NY5ASADAT01# sho fail

Failover On

Failover unit Primary

Failover LAN Interface: failover GigabitEthernet0/7 (up)

Unit Poll frequency 1 seconds, holdtime 15 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 2 of 216 maximum

failover replication http

Version: Ours 8.6(1)2, Mate 8.6(1)2

Last Failover at: 14:22:27 CDT Mar 21 2013

This host: Primary - Active

Active time: 5398172 (sec)

slot 0: ASA5525 hw/sw rev (1.0/8.6(1)2) status (Up Sys)

Interface Bloomberg (192.168.20.3): Unknown (Waiting)

Interface inside (10.20.12.250): Normal (Monitored)

Interface management (0.0.0.0): No Link (Not-Monitored)

slot 1: IPS5525 hw/sw rev (N/A/) status (Unresponsive/Up)

Other host: Secondary - Standby Ready

Active time: 140 (sec)

slot 0: ASA5525 hw/sw rev (1.0/8.6(1)2) status (Up Sys)

Interface Bloomberg (0.0.0.0): Unknown (Waiting)

Interface inside (10.20.12.251): Normal (Monitored)

Interface management (0.0.0.0): Normal (Not-Monitored)

slot 1: IPS5525 hw/sw rev (N/A/) status (Unresponsive/Up)

NY5ASADAT01# sho clo

09:38:55.256 CDT Thu Mar 28 2013

Here is relevant sho run:

NY5ASADAT01# sho run

: Saved

:

ASA Version 8.6(1)2

!

hostname NY5ASADAT01

names

!

interface GigabitEthernet0/0

nameif Bloomberg

security-level 0

ip address 192.168.20.3 255.255.255.0

!

interface GigabitEthernet0/1

nameif inside

security-level 100

ip address 10.20.12.250 255.255.255.0 standby 10.20.12.251

!

interface GigabitEthernet0/7

description LAN/STATE Failover Interface

mtu Bloomberg 1500

mtu inside 1500

mtu management 1500

failover

failover lan unit primary

failover lan interface failover GigabitEthernet0/7

failover replication http

failover link failover GigabitEthernet0/7

failover interface ip failover 192.168.254.1 255.255.255.252 standby 192.168.254.2

no monitor-interface management

arp timeout 14400

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

timeout floating-conn 0:00:00

dynamic-access-policy-record DfltAccessPolicy

threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

ntp server 130.126.24.24 source Bloomberg prefer

ntp server 130.126.24.53 source Bloomberg

ntp server 10.60.12.252 source inside prefer

ntp server 10.50.12.252 source inside

webvpn

!

class-map inspection_default

match default-inspection-traffic

!

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto

message-length maximum 512

policy-map global_policy

class inspection_default

inspect dns preset_dns_map

inspect ftp

inspect h323 h225

inspect h323 ras

inspect rsh

inspect rtsp

inspect esmtp

inspect sqlnet

inspect skinny

inspect sunrpc

inspect xdmcp

inspect sip

inspect netbios

inspect tftp

inspect ip-options

!

service-policy global_policy global

ASAs are connected to a pair of Nexus 7009s. Here is relevant interface config:

interface Ethernet3/16

description To ASA-FW-2

switchport

switchport access vlan 12

no shutdown

interface Ethernet3/16

description To ASA-FW-1

switchport

switchport access vlan 12

no shutdown

That's all I can think of right now.. let me know if there are other outputs needed.

Thanks ahead of time guys, this has been a stumper.

jocamare · ‎04-01-2013

Can you also provide the output of the "show failover state" & "show interface" commands?

We might need to change the failover timers as a possible workaround in case some of the packets are being dropped or arrive late.

davebornack · ‎04-01-2013

Here is sho fail state, along with relevant show int

NY5ASADAT01# sho fail state

State Last Failure Reason Date/Time

This host - Secondary

Active Ifc Failure 19:04:31 CDT Mar 28 2013

inside: Failed

Other host - Primary

Standby Ready Ifc Failure 06:17:45 CDT Apr 1 2013

inside: Failed

====Configuration State===

Sync Done

Sync Done - STANDBY

====Communication State===

Mac set

NY5ASADAT01# sho int

Interface GigabitEthernet0/0 "Bloomberg", is up, line protocol is up

Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

Input flow control is unsupported, output flow control is off

MAC address 0006.f62b.a5b7, MTU 1500

IP address 192.168.20.3, subnet mask 255.255.255.0

4120246861 packets input, 1370969656777 bytes, 0 no buffer

Received 110 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

2399153103 packets output, 207914420418 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 4 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops

input queue (blocks free curr/low): hardware (477/431)

output queue (blocks free curr/low): hardware (511/468)

Traffic Statistics for "Bloomberg":

984151458 packets input, 291215187218 bytes

574218817 packets output, 36547292886 bytes

6801 packets dropped

1 minute input rate 3635 pkts/sec, 1126656 bytes/sec

1 minute output rate 2081 pkts/sec, 128221 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 3791 pkts/sec, 1224242 bytes/sec

5 minute output rate 2171 pkts/sec, 138509 bytes/sec

5 minute drop rate, 0 pkts/sec

Interface GigabitEthernet0/1 "inside", is up, line protocol is up

Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec

Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)

Input flow control is unsupported, output flow control is off

MAC address 0006.f62b.a5b3, MTU 1500

IP address 10.20.12.250, subnet mask 255.255.255.0

2405260026 packets input, 208355843428 bytes, 0 no buffer

Received 321084 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

4130727262 packets output, 1372552267221 bytes, 255 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 7 interface resets

0 late collisions, 0 deferred

0 input reset drops, 768 output reset drops

input queue (blocks free curr/low): hardware (469/448)

output queue (blocks free curr/low): hardware (509/461)

Traffic Statistics for "inside":

575054447 packets input, 36589063815 bytes

986282970 packets output, 291519926771 bytes

695436 packets dropped

1 minute input rate 2082 pkts/sec, 128269 bytes/sec

1 minute output rate 3637 pkts/sec, 1126900 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 2172 pkts/sec, 138556 bytes/sec

5 minute output rate 3793 pkts/sec, 1224581 bytes/sec

5 minute drop rate, 0 pkts/sec

Interface GigabitEthernet0/7 "failover", is up, line protocol is up

Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

Input flow control is unsupported, output flow control is off

Description: LAN/STATE Failover Interface

MAC address 30f7.0d47.a017, MTU 1500

IP address 192.168.254.2, subnet mask 255.255.255.252

10674509 packets input, 2901999368 bytes, 0 no buffer

Received 310 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

55784212 packets output, 54562374044 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 6 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops

input queue (blocks free curr/low): hardware (486/458)

output queue (blocks free curr/low): hardware (455/334)

Traffic Statistics for "failover":

2168497 packets input, 1234830690 bytes

8052935 packets output, 7748660590 bytes

1 packets dropped

1 minute input rate 1 pkts/sec, 118 bytes/sec

1 minute output rate 32 pkts/sec, 35048 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 1 pkts/sec, 117 bytes/sec

5 minute output rate 32 pkts/sec, 35040 bytes/sec

5 minute drop rate, 0 pkts/sec

jocamare · ‎04-01-2013

The problem seems to be only with the "inside" interface.

Let's try to clear the counters and calculate how fast the number of dropped packets increasses.

Issue a "clear interface gi0/1" and then get the output of the "show interface gi0/1" a couple of times with 5 minutes of difference between one output and the other.

davebornack · ‎04-01-2013

Right.. we know the problem exists there, but not sure why.

Switch side is showing flaps every hour or two..

We're going to go through the entire path and replace everything tonight.. (cables, SFPs, patch panels, etc)

Julio Carvajal · ‎04-01-2013

Hello,

debug fo rxip

debug fo txip

to determine if the packets are being exchanged according to the configured polltimes.

It might generate a lot of logs so take it into consideration before enabling it but it ill definetly let us know if the exchange of hello packets is sucessfull,

Is it possible to create a SPAN session on the Nexus?

Also after the RMA you mentioned you are receiving new messages :

it's odd that it's telling me about VPN and WEBVPN interfaces.....

Are you still receiving the Inside interface failures messages

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC