cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2330
Views
0
Helpful
5
Replies

ASA A/S Pair keeps failing over

davebornack
Level 1
Level 1

We keep getting failover syslog messages.

Level: Alert 
Date: 2013-03-22 05:18:48
Host: 10.20.12.250
Message:
%ASA-1-105005: (Secondary) Lost Failover communications with mate on interface inside

Level: Alert 
Date: 2013-03-22 05:18:48
Host: 10.20.12.250
Message:
%ASA-1-105008: (Secondary) Testing Interface inside

ETC.... 

We never lose connectivity, which is great, as that's how it should work, but things keep failing.  Cisco hasn't been able to figure it out, and sent us a replacement unit, which unfortunately didn't do the trick.

Here are some outputs..  which interestingly enough, don't show any fail since Mar 21, but we're getting syslog messages that are indicative of a failover.

Here is syslog from this morning's fialover..   it's odd that it's telling me about VPN and WEBVPN interfaces.....

2013-03-28

08:23:25

Local4

Debug

10.20.12.250

%ASA-7-720042: (VPN-Primary) Receiving Command Link Bulk Sync message (Command 4) from active unit

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-721002: (WebVPN-Primary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Standby Ready.

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-720028: (VPN-Primary) HA status callback: Peer state Standby Ready.

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-720032: (VPN-Primary) HA status callback: id=3,seq=200,grp=0,event=406,op=80,my=Active,peer=Standby Ready.

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-721002: (WebVPN-Primary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Failed.

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-720028: (VPN-Primary) HA status callback: Peer state Failed.

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-720032: (VPN-Primary) HA status callback: id=3,seq=200,grp=0,event=406,op=20,my=Active,peer=Failed.

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-721002: (WebVPN-Primary) HA status change: event HA_STATUS_PEER_STATE, my state Active, peer state Standby Ready.

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-720028: (VPN-Primary) HA status callback: Peer state Standby Ready.

2013-03-28

08:23:25

Local4

Info

10.20.12.250

%ASA-6-720032: (VPN-Primary) HA status callback: id=3,seq=200,grp=0,event=406,op=80,my=Active,peer=Standby Ready.

NY5ASADAT01# sho fail hist

==========================================================================

From State                 To State                   Reason

==========================================================================

17:46:07 CDT Mar 20 2013

Active                     Cold Standby               Failover state check

17:46:08 CDT Mar 20 2013

Cold Standby               Sync Config                Failover state check

17:46:15 CDT Mar 20 2013

Sync Config                Sync File System           Failover state check

17:46:15 CDT Mar 20 2013

Sync File System           Bulk Sync                  Failover state check

17:46:27 CDT Mar 20 2013

Bulk Sync                  Standby Ready              Failover state check

07:52:39 CDT Mar 21 2013

Standby Ready              Just Active                Other unit wants me Active

07:52:39 CDT Mar 21 2013

Just Active                Active Drain               Other unit wants me Active

07:52:39 CDT Mar 21 2013

Active Drain               Active Applying Config     Other unit wants me Active

07:52:39 CDT Mar 21 2013

Active Applying Config     Active Config Applied      Other unit wants me Active

07:52:39 CDT Mar 21 2013

Active Config Applied      Active                     Other unit wants me Active

13:18:59 CDT Mar 21 2013

Active                     Standby Ready              Set by the config command

14:22:27 CDT Mar 21 2013

Standby Ready              Just Active                Other unit wants me Active

14:22:27 CDT Mar 21 2013

Just Active                Active Drain               Other unit wants me Active

14:22:27 CDT Mar 21 2013

Active Drain               Active Applying Config     Other unit wants me Active

14:22:27 CDT Mar 21 2013

Active Applying Config     Active Config Applied      Other unit wants me Active

14:22:27 CDT Mar 21 2013

Active Config Applied      Active                     Other unit wants me Active

NY5ASADAT01#     sho fail

Failover On

Failover unit Primary

Failover LAN Interface: failover GigabitEthernet0/7 (up)

Unit Poll frequency 1 seconds, holdtime 15 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 2 of 216 maximum

failover replication http

Version: Ours 8.6(1)2, Mate 8.6(1)2

Last Failover at: 14:22:27 CDT Mar 21 2013

        This host: Primary - Active

                Active time: 5398172 (sec)

                slot 0: ASA5525 hw/sw rev (1.0/8.6(1)2) status (Up Sys)

                  Interface Bloomberg (192.168.20.3): Unknown (Waiting)

                  Interface inside (10.20.12.250): Normal (Monitored)

                  Interface management (0.0.0.0): No Link (Not-Monitored)

                slot 1: IPS5525 hw/sw rev (N/A/) status (Unresponsive/Up)

        Other host: Secondary - Standby Ready

                Active time: 140 (sec)

                slot 0: ASA5525 hw/sw rev (1.0/8.6(1)2) status (Up Sys)

                  Interface Bloomberg (0.0.0.0): Unknown (Waiting)

                  Interface inside (10.20.12.251): Normal (Monitored)

                  Interface management (0.0.0.0): Normal (Not-Monitored)

                slot 1: IPS5525 hw/sw rev (N/A/) status (Unresponsive/Up)

NY5ASADAT01#  sho clo

09:38:55.256 CDT Thu Mar 28 2013

Here is relevant sho run:

NY5ASADAT01# sho run

: Saved

:

ASA Version 8.6(1)2

!

hostname NY5ASADAT01

names

!

interface GigabitEthernet0/0

nameif Bloomberg

security-level 0

ip address 192.168.20.3 255.255.255.0

!

interface GigabitEthernet0/1

nameif inside

security-level 100

ip address 10.20.12.250 255.255.255.0 standby 10.20.12.251

!

!            

interface GigabitEthernet0/7

description LAN/STATE Failover Interface

mtu Bloomberg 1500

mtu inside 1500

mtu management 1500

failover

failover lan unit primary

failover lan interface failover GigabitEthernet0/7

failover replication http

failover link failover GigabitEthernet0/7

failover interface ip failover 192.168.254.1 255.255.255.252 standby 192.168.254.2

no monitor-interface management

arp timeout 14400

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

timeout floating-conn 0:00:00

dynamic-access-policy-record DfltAccessPolicy

threat-detection basic-threat

threat-detection statistics access-list

no threat-detection statistics tcp-intercept

ntp server 130.126.24.24 source Bloomberg prefer

ntp server 130.126.24.53 source Bloomberg

ntp server 10.60.12.252 source inside prefer

ntp server 10.50.12.252 source inside

webvpn

!

class-map inspection_default

match default-inspection-traffic

!

!

policy-map type inspect dns preset_dns_map

parameters

  message-length maximum client auto

  message-length maximum 512

policy-map global_policy

class inspection_default

  inspect dns preset_dns_map

  inspect ftp

  inspect h323 h225

  inspect h323 ras

  inspect rsh

  inspect rtsp

  inspect esmtp

  inspect sqlnet

  inspect skinny 

  inspect sunrpc

  inspect xdmcp

  inspect sip 

  inspect netbios

  inspect tftp

  inspect ip-options

!

service-policy global_policy global

ASAs are connected to a pair of Nexus 7009s.  Here is relevant interface config:

interface Ethernet3/16

  description To ASA-FW-2

  switchport

  switchport access vlan 12

  no shutdown

interface Ethernet3/16

  description To ASA-FW-1

  switchport

  switchport access vlan 12

  no shutdown

That's all I can think of right now..   let me know if there are other outputs needed.

Thanks ahead of time guys, this has been a stumper.

5 Replies 5

jocamare
Level 4
Level 4

Can you also provide the output of the "show failover state" & "show interface" commands?

We might need to change the failover timers as a possible workaround in case some of the packets are being dropped or arrive late.

Here is sho fail state, along with relevant show int

NY5ASADAT01# sho fail state

               State          Last Failure Reason      Date/Time

This host  -   Secondary

               Active         Ifc Failure              19:04:31 CDT Mar 28 2013

                              inside: Failed

Other host -   Primary

               Standby Ready  Ifc Failure              06:17:45 CDT Apr 1 2013

                              inside: Failed

====Configuration State===

        Sync Done

        Sync Done - STANDBY

====Communication State===

        Mac set

NY5ASADAT01# sho int

Interface GigabitEthernet0/0 "Bloomberg", is up, line protocol is up

  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        MAC address 0006.f62b.a5b7, MTU 1500

        IP address 192.168.20.3, subnet mask 255.255.255.0

        4120246861 packets input, 1370969656777 bytes, 0 no buffer

        Received 110 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        2399153103 packets output, 207914420418 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 4 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops

        input queue (blocks free curr/low): hardware (477/431)

        output queue (blocks free curr/low): hardware (511/468)

  Traffic Statistics for "Bloomberg":

        984151458 packets input, 291215187218 bytes

        574218817 packets output, 36547292886 bytes

        6801 packets dropped

      1 minute input rate 3635 pkts/sec,  1126656 bytes/sec

      1 minute output rate 2081 pkts/sec,  128221 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 3791 pkts/sec,  1224242 bytes/sec

      5 minute output rate 2171 pkts/sec,  138509 bytes/sec

      5 minute drop rate, 0 pkts/sec

Interface GigabitEthernet0/1 "inside", is up, line protocol is up

  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec

        Full-Duplex(Full-duplex), 1000 Mbps(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        MAC address 0006.f62b.a5b3, MTU 1500

        IP address 10.20.12.250, subnet mask 255.255.255.0

        2405260026 packets input, 208355843428 bytes, 0 no buffer

        Received 321084 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        4130727262 packets output, 1372552267221 bytes, 255 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 7 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 768 output reset drops

        input queue (blocks free curr/low): hardware (469/448)

        output queue (blocks free curr/low): hardware (509/461)

  Traffic Statistics for "inside":

        575054447 packets input, 36589063815 bytes

        986282970 packets output, 291519926771 bytes

        695436 packets dropped

      1 minute input rate 2082 pkts/sec,  128269 bytes/sec

      1 minute output rate 3637 pkts/sec,  1126900 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 2172 pkts/sec,  138556 bytes/sec

      5 minute output rate 3793 pkts/sec,  1224581 bytes/sec

      5 minute drop rate, 0 pkts/sec

Interface GigabitEthernet0/7 "failover", is up, line protocol is up

  Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec

        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

        Input flow control is unsupported, output flow control is off

        Description: LAN/STATE Failover Interface

        MAC address 30f7.0d47.a017, MTU 1500

        IP address 192.168.254.2, subnet mask 255.255.255.252

        10674509 packets input, 2901999368 bytes, 0 no buffer

        Received 310 broadcasts, 0 runts, 0 giants

        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

        0 pause input, 0 resume input

        0 L2 decode drops

        55784212 packets output, 54562374044 bytes, 0 underruns

        0 pause output, 0 resume output

        0 output errors, 0 collisions, 6 interface resets

        0 late collisions, 0 deferred

        0 input reset drops, 0 output reset drops

        input queue (blocks free curr/low): hardware (486/458)

        output queue (blocks free curr/low): hardware (455/334)

  Traffic Statistics for "failover":

        2168497 packets input, 1234830690 bytes

        8052935 packets output, 7748660590 bytes

        1 packets dropped

      1 minute input rate 1 pkts/sec,  118 bytes/sec

      1 minute output rate 32 pkts/sec,  35048 bytes/sec

      1 minute drop rate, 0 pkts/sec

      5 minute input rate 1 pkts/sec,  117 bytes/sec

      5 minute output rate 32 pkts/sec,  35040 bytes/sec

      5 minute drop rate, 0 pkts/sec

The problem seems to be only with the "inside" interface.

Let's try to clear the counters and calculate how fast the number of dropped packets increasses.

Issue a "clear interface gi0/1" and then get the output of the "show interface gi0/1" a couple of times with 5 minutes of difference between one output and the other.

Right..  we know the problem exists there, but not sure why.

Switch side is showing flaps every hour or two.. 

We're going to go through the entire path and replace everything tonight..  (cables, SFPs, patch panels, etc)

Hello,

debug fo rxip

debug fo txip

to determine if the packets are being exchanged according to the configured polltimes.

It might generate a lot of logs so take it into consideration before enabling it but it ill definetly let us know if the exchange of hello packets is sucessfull,

Is it possible to create a SPAN session on the Nexus?

Also after the RMA you mentioned you are receiving new messages :

it's odd that it's telling me about VPN and WEBVPN interfaces.....

Are you still receiving the Inside interface failures messages

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: