Unknown 105008 and 105009 logs on non-failover interface

holtchristopher · ‎11-27-2012

Hi,

I have a pair of ASA5510s in a failover configuration where I see these 2 logs repeated every 15 seconds.

105008 1 Nov 27 2012 10:39:27 (Primary) Testing Interface management

105009 1 Nov 27 2012 10:39:28 (Primary) Testing on interface management Passed

I have read other threads where these are accompanied by "105005, Lost Failover communications with mate on interface". But I'm only getting these 2. The other thing that is confusing is that the "management" interface is not the failover interface. So why do I see 105008/9 logs about it?

Output of "sh fail":

5510a# sh fail

Failover On

Failover unit Primary

Failover LAN Interface: failoverlink Ethernet0/3 (up)

Unit Poll frequency 1 seconds, holdtime 15 seconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 3 of 110 maximum

failover replication http

Version: Ours 8.4(4)1, Mate 8.4(4)1

Last Failover at: 21:08:36 CDT Nov 1 2012

This host: Primary - Active

Active time: 2212776 (sec)

slot 0: ASA5510 hw/sw rev (2.0/8.4(4)1) status (Up Sys)

Interface inside (192.168.2.98): Unknown (Waiting)

Interface outside (xxx.yyy.zzz.www): Normal (Waiting)

Interface management (192.168.6.6): Normal (Waiting)

slot 1: empty

Other host: Secondary - Standby Ready

Active time: 0 (sec)

slot 0: ASA5510 hw/sw rev (2.0/8.4(4)1) status (Up Sys)

Interface inside (0.0.0.0): Unknown (Waiting)

Interface outside (0.0.0.0): Unknown (Waiting)

Interface management (192.168.6.7): Normal (Waiting)

slot 1: empty

Stateful Failover Logical Update Statistics

Link : failoverlink Ethernet0/3 (up)

Stateful Obj xmit xerr rcv rerr

General 9774825 0 288600 0

sys cmd 288600 0 288600 0

up time 0 0 0 0

RPC services 0 0 0 0

TCP conn 3641769 0 0 0

UDP conn 1556140 0 0 0

ARP tbl 4277507 0 0 0

Xlate_Timeout 0 0 0 0

IPv6 ND tbl 0 0 0 0

VPN IKEv1 SA 1136 0 0 0

VPN IKEv1 P2 7204 0 0 0

VPN IKEv2 SA 0 0 0 0

VPN IKEv2 P2 0 0 0 0

VPN CTCP upd 574 0 0 0

VPN SDI upd 0 0 0 0

VPN DHCP upd 0 0 0 0

SIP Session 0 0 0 0

Route Session 0 0 0 0

User-Identity 1895 0 0 0

Logical Update Queue Information

Cur Max Total

Recv Q: 0 20 288600

Xmit Q: 0 1425 23464417

Interfaces from "sh run"

!

interface Ethernet0/0

nameif inside

security-level 100

ip address 192.168.2.98 255.255.255.0

!

interface Ethernet0/1

shutdown

no nameif

no security-level

no ip address

!

interface Ethernet0/2

nameif outside

security-level 0

ip address xxx.yyy.zzz.www 255.255.255.248

!

interface Ethernet0/3

description LAN/STATE Failover Interface

!

interface Management0/0

nameif management

security-level 0

ip address 192.168.6.6 255.255.255.0 standby 192.168.6.7

management-only

!

management interface from "sh interface detail"

Interface Management0/0 "management", is up, line protocol is up

Hardware is i82557, BW 100 Mbps, DLY 100 usec

Auto-Duplex(Full-duplex), Auto-Speed(100 Mbps)

Input flow control is unsupported, output flow control is unsupported

MAC address 0023.3353.a9df, MTU 1500

IP address 192.168.6.6, subnet mask 255.255.255.0

4299114 packets input, 268429762 bytes, 0 no buffer

Received 4108937 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 pause input, 0 resume input

0 L2 decode drops

2682572 packets output, 208340210 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 0 interface resets

0 babbles, 0 late collisions, 0 deferred

0 lost carrier, 0 no carrier

0 input reset drops, 0 output reset drops

input queue (curr/max packets): hardware (0/1) software (0/30)

output queue (curr/max packets): hardware (0/5) software (0/1)

Traffic Statistics for "management":

4300070 packets input, 208228356 bytes

2683165 packets output, 166571744 bytes

72339 packets dropped

1 minute input rate 2 pkts/sec, 111 bytes/sec

1 minute output rate 0 pkts/sec, 42 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 2 pkts/sec, 105 bytes/sec

5 minute output rate 0 pkts/sec, 40 bytes/sec

5 minute drop rate, 0 pkts/sec

Management-only interface. Blocked 24 through-the-device packets

0 IPv4 packets originated from management network

24 IPv4 packets destined to management network

0 IPv6 packets originated from management network

0 IPv6 packets destined to management network

Control Point Interface States:

Interface number is 6

Interface config status is active

Interface state is active

Thanks,

Chris

julomban · ‎11-27-2012

Syslog ID 105008 is displayed when the tests a specified interface. This testing is performed only if the ASA fails to receive a message from the standby unit on that interface after the expected interval. Right after 105009 with the test results (either Passed or Failed) of a previous interface test.

No action required if the result is passed, however if you are constantly seeing the messages make sure the cables you are using are in perfect conditions and ports of the switch are operating fine.

Failover is going to check the interfaces not just the failover link thus you can expect to see this message on any interface.

Regards,

Juan Lombana

Please rate helpful posts.

holtchristopher · ‎11-27-2012

So if failover is going to test all the interfaces, why is only the management interface constantly reporting this? What message is the ASA expecting from the failover unit on the management interface when the failover link is on Ethernet0/3?

julomban · ‎11-27-2012

Chris,

Remember this testing is performed only if the primary ASA fails to receive a message from the standby unit on that interface. All the rest interfaces are not performing the testing since they receive the "hello" message from the standby interface, it never fails.

Failover constantly check the status of all interface by sending a "hello" message, if for some reason the primary ASA fails to receive the message from the standby automatically performs the testing on the interface in question (automatically log 105009 will show up) if the issue is constant (bad port) the result will be "Failed" but the test results are Passed there is no action require. 99% of cases is related to a bad cable, transmitting and losing packets while traversing from the primary to secondary.

How the management port is connected between your ASA's? Are you using a switch between these two ports or they’re directly connected?

My suggestion is to check ports and cables (if possible change the cable if they are directly connected). If you are using a switch make sure the switch ports are not flapping or odd color.

I hope it helps,

Juan Lombana

Please rate helpful posts.

pille1234 · ‎11-27-2012

Hi,

your failover pair is not working correctly it seems.

This part here is wrong:

        This host: Primary - Active                Active time: 2212776 (sec)
                slot 0: ASA5510 hw/sw rev (2.0/8.4(4)1) status (Up Sys)
                  Interface inside (192.168.2.98): Unknown (Waiting)
                  Interface outside (xxx.yyy.zzz.www): Normal (Waiting)
                  Interface management (192.168.6.6): Normal (Waiting)
                slot 1: empty
        Other host: Secondary - Standby Ready
                Active time: 0 (sec)
                slot 0: ASA5510 hw/sw rev (2.0/8.4(4)1) status (Up Sys)
                  Interface inside (0.0.0.0): Unknown (Waiting)
                  Interface outside (0.0.0.0): Unknown (Waiting)
                  Interface management (192.168.6.7): Normal (Waiting)
                slot 1: empty

At least the management interface should read "Monitored" instead of "Waiting". It seems the hello packets are not reaching the corresponding interface on secondery ASA or vice versa. Is a ping from active to standby working? How are the ASAs connected to each other? Is there a switch inbetween? Maybe a flapping link or something...

Regards

Pille

holtchristopher · ‎11-27-2012

The two ASA's are only directly connected by the failover link. (Ethernet0/3 on both devices). The Ethernet0/0 on each device is wired to an unmanaged switch (one per ASA). My two servers are connected to each of those switches with bonded interfaces.

Like This:

The two interfaces (em1 and em2) on each of the servers are bonded together into bond0. Originally I had planned on having the 2 servers connected to each of the Eth0/0 and Eth0/1 ports of the 5510s until I realized that the 5510 ports are layer 3 ports and not layer 2 ports. After discovering that I had to have a switch between the 5510 and the servers, I put a small (5 port) switch attached to each 5510 to effectively connect 2 servers to a single port on each 5510. (I put 1 switch per 5510 because I didn't want another single point of failure)

The Mgmt0/0 ports are on a different network (192.168.6.x as opposed to .2.x for everything else). It's reachable from the 2 servers via a sub interface (bond0:0) that's on the .6.x network.

The key point that I'm getting from your comment is that the 2 ASA 5510s can't see each other except on the failover link and that that isn't a normal configuration.

After reading more about bonding and failover and considering what you pointed out, I think what I need to do is connect the 2 unmanged switches, and change the bonding mode of the servers to active-backup instead of broadcast. Then the 2 5510s can see each other on their managment interfaces and there shouldn't be any duplicated packets or loops. Does that sound reasonable, or am I still confused?

Thanks,

Chris

lknutsonuci · ‎11-27-2012

I encountered this exact issue after creating a new subinterface and found that the issue was that the new subinterface hadn't properly replicated to the secondary device. Issue a 'show fail' command on the primary to see the interfaces on the primary, then issue a 'fail exec mate show fail' (again from the primary) to see the interfacs on the secondary and check that all the interfaces exist on both. If there are (sub)interfaces on the primary that aren't on the secondary, try issuing a 'write standby' command from the primary to force the primary config to the secondary device. After doing that, the 105008 and 105009 log entries stopped for me. Note also that I'm discussing an Active/Standby pair; I don't know if this would work as I've described in an Active/Active pair.

Keisuke Katsuda · ‎12-15-2013

Hi,

We're also facing just similar issues. (i'm suspecting bug)

Does someone can show us the reasonable causes ??

ASA5585-X with 8.4.4(1) Active/Active multi mode.

We have around 10 security context, 100 sub-interfaces.

Currently, we only see below matter only one of context, 2 of sub-interface.

Thus, I believe physical connection should be fine, otherwise, all of other sub-interface should face similar issues.

I tried Mr. Lee Knutson suggestion, however, standby mate seems to have the same configuration as active mate.

#previously, i fixed this issues trying the above way, but this times seems not the case.

Below all of interfaces are under same phyisical interface.

Failover On

Last Failover at: 11:04:31 SGT May 29 2013

This context: Active

Active time: 17362860 (sec)

****

Interface TEST (10.255.200.253): Normal (Monitored)

Interface TEST2 (10.255.200.164): Normal (Monitored)

Interface TEST3 (10.81.251.201): Normal (Waiting)

Peer context: Standby Ready

Active time: 0 (sec)

***

Interface TEST (10.255.200.254): Normal (Monitored)

Interface TEST2 (10.255.200.165): Normal (Waiting)

Interface TEST3 (10.81.251.202): Normal (Waiting)

Stateful Failover Logical Update Statistics

Status: Configured.

Stateful Obj xmit xerr rcv rerr

RPC services 0 0 0 0

TCP conn 714143069 0 405 0

UDP conn 184574526 0 826 0

ARP tbl 82808148 0 188 0

Xlate_Timeout 0 0 0 0

IPv6 ND tbl 0 0 0 0

SIP Session 0 0 0 0

Route Session 0 0 0 0

User-Identity 1 0 1 0