topic ASA Secondary FW Sub-Interface Failure in Network Security

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Tue, 12 Mar 2019 01:09:18 GMT

I have two ASA 5520s in Active/Standby. I try and test this quartely to ensure it is working correctly. Everything works fine, except I have an issue with one interface. When doing a show failover, it shows the interface as failed on the secondary unit, and I am not sure why. It shows it as normal on the primary. Has anyone else had experience this, or can guide me in the right direction to resolve it.

This host: Primary - Active

Active time: 9277305 (sec)

slot 0: ASA5520 hw/sw rev (2.0/8.2(4)) status (Up Sys)

Interface WaterworksCanopy (192.x.x.x): Normal

slot 1: ASA-SSM-4GE hw/sw rev (1.0/1.0(0)10) status (Up)

Other host: Secondary - Failed

Active time: 0 (sec)

slot 0: ASA5520 hw/sw rev (2.0/8.2(4)) status (Up Sys)

Interface WaterworksCanopy (192.x.x.x): Failed (Waiting)

ASA Secondary FW Sub-Interface Failure

jocamare — Mon, 04 Mar 2013 18:17:49 GMT

What it means is that the active unit is not getting replies to the "hello" messages that are exchanged in order to monitor the status of the interfaces.

One reason for this, the interface is down or unreachable.

Have you checked the output of the "show interface ip brief" command from the standby unit?

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Mon, 04 Mar 2013 18:33:56 GMT

Yes, I have, and it shows the interface as up.

GigabitEthernet1/0.201 192.x.x.x YES CONFIG up up

Alos, I have another sub-if on this GigabitEthernet1/0 interface and it works, fine and shows as fine. As far as I can tell they are both set up the same.

ASA Secondary FW Sub-Interface Failure

jocamare — Mon, 04 Mar 2013 19:10:11 GMT

The problem might be related to slow responses from the standby unit that result in an unit marked as failed.

Can you ping the stanby IP address? Is yes, what are the time stats of those replies?

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Mon, 04 Mar 2013 19:21:07 GMT

I can not ping the standby interface from the primary face. Not sure why at this time, the I can ping the standby of the other sub-if on this interface. Nto sure why, they are configured the same, and the pyhiscal networking is the same.

See below, the sub-if not working is the top, and the one working is the bottom. Only difference is the securtiy level.

interface GigabitEthernet1/0.201

description Canopy Interface to Waterworks

vlan 201

nameif WaterworksCanopy

security-level 60

ip address 192.x.x.x 255.255.255.0 standby 192.x.x.x

WCPSNASA#

WCPSNASA# sh running-config interface gi1/0.202

interface GigabitEthernet1/0.202

vlan 202

nameif WCWS_Security

security-level 65

ip address 192.x.x.x 255.255.255.0 standby 192.x.x.x

ASA Secondary FW Sub-Interface Failure

jocamare — Mon, 04 Mar 2013 20:02:22 GMT

Can you ping anything else apart from the stanby address?

On the standby ASA, can you ping anything at all?

What doesn the "show run icmp" display?

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Mon, 04 Mar 2013 20:19:38 GMT

On the sub-if I am having problems with I can not ping the standby address from the primary, nor the primary from the standy.

However, or all other interfaces and sub-ifs I can do this.

With the show run icmp, I get

icmp unreachable rate-limit 1 burst-size 1

icmp permit any Outside

ASA Secondary FW Sub-Interface Failure

jocamare — Mon, 04 Mar 2013 22:02:19 GMT

Can you ping other devices in the same network from both firewalls?

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Tue, 05 Mar 2013 12:51:39 GMT

I can other resources on that network from the primary, however not from the secondary.

ASA Secondary FW Sub-Interface Failure

jocamare — Tue, 05 Mar 2013 18:21:15 GMT

Ok, the issue is now isolated.

Can you provide the output of the "show interface" command from that interface and try to shut it down and then turn it back on?

Also, a wide open capture on that interface will be useful.

capture tac in

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Wed, 06 Mar 2013 13:27:05 GMT

Below is the information.

There is the show interface of the overall interface gi1/0, and of the sub-ifs, gi1/0.201 and gi1/0.202.

Gi1/0.201 is the sub-if not working correctly on failover, while gi1/0.202 works fine.

Alos, the packet capture is 42 packets. If you need one larger, please let me know.

Interface GigabitEthernet1/0 "Canopy", is up, line protocol is up
Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec
        Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
        Input flow control is unsupported, output flow control is unsupported
        Media-type configured as RJ45 connector
        Description: Canopy Master Interface
        MAC address 001d.a298.c7bb, MTU 1500
        IP address unassigned
        15689423 packets input, 1262652964 bytes, 0 no buffer
        Received 485671 broadcasts, 0 runts, 0 giants
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
        0 pause input, 0 resume input
        1 L2 decode drops
        64215 packets output, 5772926 bytes, 0 underruns
        0 pause output, 0 resume output
        0 output errors, 0 collisions, 0 interface resets
        0 late collisions, 0 deferred
        0 rate limit drops
        0 input reset drops, 0 output reset drops
        input queue (blocks free curr/low): hardware (0/0)
        output queue (blocks free curr/low): hardware (0/0)
Traffic Statistics for "Canopy":
        429936 packets input, 28515990 bytes
        7 packets output, 196 bytes
        251866 packets dropped
      1 minute input rate 2 pkts/sec, 160 bytes/sec
      1 minute output rate 0 pkts/sec, 0 bytes/sec
      1 minute drop rate, 1 pkts/sec
      5 minute input rate 2 pkts/sec, 186 bytes/sec
      5 minute output rate 0 pkts/sec, 0 bytes/sec
      5 minute drop rate, 1 pkts/sec

Interface GigabitEthernet1/0.201 "WaterworksCanopy", is up, line protocol is up
Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec
<--- More --->

VLAN identifier 201
Media-type configured as RJ45 connector
Description: Canopy Interface to Waterworks
MAC address 001d.a298.c7bb, MTU 1500
IP address 192.168.200.253, subnet mask 255.255.255.0
Traffic Statistics for "WaterworksCanopy":
0 packets input, 0 bytes
33776 packets output, 2287568 bytes
0 packets dropped

Interface GigabitEthernet1/0.202 "WCWS_Security", is up, line protocol is up
Hardware is VCS7380 rev01, BW 1000 Mbps, DLY 10 usec
VLAN identifier 202
Media-type configured as RJ45 connector
MAC address 001d.a298.c7bb, MTU 1500
IP address 192.168.x.x, subnet mask 255.255.255.0
Traffic Statistics for "WCWS_Security":
119477 packets input, 7005389 bytes
30173 packets output, 2051484 bytes
22071 packets dropped

42 packets captured

   1: 08:09:27.676554 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
   2: 08:09:32.675959 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
   3: 08:09:37.675349 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
   4: 08:09:42.674739 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
   5: 08:09:47.674128 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
   6: 08:09:52.673533 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
   7: 08:09:57.672938 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
   8: 08:10:02.672374 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
   9: 08:10:07.671718 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
10: 08:10:12.671153 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
11: 08:10:16.161093 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
12: 08:10:17.670512 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
13: 08:10:18.160346 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
14: 08:10:19.160224 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
15: 08:10:22.669917 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
16: 08:10:23.159751 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
17: 08:10:27.669322 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
18: 08:10:28.159140 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
19: 08:10:32.668712 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
20: 08:10:33.158530 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
21: 08:10:37.668117 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
22: 08:10:38.157935 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
23: 08:10:42.667522 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
24: 08:10:43.157325 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
25: 08:10:47.666957 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
26: 08:10:48.156730 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
27: 08:10:52.666332 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
28: 08:10:53.156150 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
29: 08:10:57.665706 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
30: 08:10:58.155524 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
31: 08:11:02.665111 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
32: 08:11:03.154929 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
33: 08:11:07.664501 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
34: 08:11:08.154334 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
35: 08:11:12.663906 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
36: 08:11:13.153724 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253
37: 08:11:17.663295 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
38: 08:11:18.153144 802.1Q vlan#201 P0 arp who-has 192.168.200.102 tell 192.168.200.253

39: 08:11:22.662700 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
40: 08:11:27.662090 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
41: 08:11:32.661571 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
42: 08:11:37.660900 802.1Q vlan#201 P0 192.168.200.253 > 192.168.200.254: ip-proto-105, length 48
42 packets shown

ASA Secondary FW Sub-Interface Failure

Jouni Forss — Wed, 06 Mar 2013 13:34:38 GMT

Hi,

Are you sure there hasnt been any changes in the switching for Vlan201 that would break the link between ASAs and/or Standby ASA and the rest of the devices on Vlan201?

Can you check the connected switch if it seems the MAC address of the ASA in the interface where its connected? If yes, is the Vlan201 configured all the way to the Primary ASA? Can you see the Primary ASA Vlan201 interface MAC address on that same switch?

The capture above seems to only contain ASA Failover Hello messages sent by the Standby ASA itself. Its also doing ARP requests

And as the interfaces counter says its not receiving ANY traffic on that interface.

- Jouni

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Wed, 06 Mar 2013 13:42:18 GMT

No there has not been any changes for the switching. Below is the config for the connected switchport for that interface, along with the mac table for that vlan (201) and it shows the ASA mac address in it.

interface GigabitEthernet1/47

description ASA-B Canopy IF

switchport trunk encapsulation dot1q

switchport trunk allowed vlan 201,202

switchport mode trunk

media-type rj45

end

LSW8#show mac address-table vlan 201
Unicast Entries
vlan   mac address     type        protocols               port
-------+---------------+--------+---------------------+--------------------
201    0000.0c07.acc9   dynamic ip                    Port-channel5
201    0000.bc06.ef67   dynamic ip                    Port-channel5
201    0000.bc06.ef87   dynamic ip                    Port-channel5
201    0000.bc21.7750   dynamic ip                    Port-channel5
201    0000.bc21.782e   dynamic ip                    Port-channel5
201    0000.bc21.7c71   dynamic ip                    Port-channel5
201    0000.bc21.7e9b   dynamic ip                    Port-channel5
201    0000.bc21.81ad   dynamic ip                    Port-channel5
201    0000.bc32.f44a   dynamic ip                    Port-channel5
201    0000.bc32.f451   dynamic ip                    Port-channel5
201    0000.bc32.f466   dynamic ip                    Port-channel5
201    0000.bc32.f49c   dynamic ip                    Port-channel5
201    0000.bc36.d00b   dynamic ip                    Port-channel5
201    0000.bc36.d122   dynamic ip                    Port-channel5
201    0000.bc36.d124   dynamic ip                    Port-channel5
201    0000.bc36.d1fb   dynamic ip                    Port-channel5
201    0000.bc36.d227   dynamic ip                    Port-channel5
201    0000.bc36.d325   dynamic ip                    Port-channel5
201    0000.bc39.e4d2   dynamic ip                    Port-channel5
201    0000.bc3a.a8b7   dynamic ip                    Port-channel5
201    0000.bc3b.34eb   dynamic ip                    Port-channel5
201    0000.bc3b.3bf1   dynamic ip                    Port-channel5
201    0000.bc3b.a994   dynamic ip                    Port-channel5
201    0000.bc3b.c9c4   dynamic ip                    Port-channel5
201    0000.bc3b.ef54   dynamic ip                    Port-channel5
201    0000.bc3c.02a8   dynamic ip                    Port-channel5
201    0000.bc3c.417c   dynamic ip                    Port-channel5
201    0000.bc3d.5774   dynamic ip                    Port-channel5
201    0000.bc3e.2aa6   dynamic ip                    Port-channel5
201    0000.bc3e.2ac2   dynamic ip                    Port-channel5
201    0000.bc3e.3ee8   dynamic ip                    Port-channel5
201    0000.bc3e.8b6f   dynamic ip                    Port-channel5
201    0000.bc3f.8d8e   dynamic ip                    Port-channel5
201    0000.bc3f.8e7f   dynamic ip                    Port-channel5
201    0000.bc3f.9057   dynamic ip                    Port-channel5
201    0000.bc3f.92ba   dynamic ip                    Port-channel5
201    0000.bc3f.92c2   dynamic ip                    Port-channel5
201    0000.bc3f.cb2a   dynamic ip                    Port-channel5
201    0000.bc3f.cb3f   dynamic ip                    Port-channel5
201    0000.bc3f.d39f   dynamic ip                    Port-channel5
201    0000.bc3f.d3c4   dynamic ip                    Port-channel5
201    0000.bc3f.d462   dynamic ip                    Port-channel5
201    0000.bc40.2bf4   dynamic ip                    Port-channel5
201    0000.bc57.9d0a   dynamic ip                    Port-channel5
201    0000.bc57.a41f   dynamic ip                    Port-channel5
201    0000.bc63.de62   dynamic ip                    Port-channel5
201    0000.bc63.deb1   dynamic ip                    Port-channel5
201    0000.bc63.dee1   dynamic ip                    Port-channel5
201    0000.bc63.dee2   dynamic ip                    Port-channel5
201    0000.bc64.7d8f   dynamic ip                    Port-channel5
201    0000.bc64.a2a2   dynamic ip                    Port-channel5
201    0000.bc64.adcc   dynamic ip                    Port-channel5
201    0000.bc64.ade5   dynamic ip                    Port-channel5
201    0000.bc64.af5f   dynamic ip                    Port-channel5
201    0000.bc64.d33e   dynamic ip                    Port-channel5
201    0000.bc65.ba65   dynamic ip                    Port-channel5
201    0000.bc65.bd96   dynamic ip                    Port-channel5
201    0000.bc66.6a83   dynamic ip                    Port-channel5
201    0000.bcce.867a   dynamic ip                    Port-channel5
201    0000.bcce.86de   dynamic ip                    Port-channel5
201    0000.bcd0.cb50   dynamic ip                    Port-channel5
201    000f.7300.c84d   dynamic ip                    Port-channel5
201    000f.7300.d143   dynamic ip                    Port-channel5
201    000f.7300.d15c   dynamic ip                    Port-channel5
201    000f.7300.dfa4   dynamic ip                    Port-channel5
201    000f.7300.dfad   dynamic ip                    Port-channel5
201    000f.7300.dfb2   dynamic ip                    Port-channel5
201    000f.7300.f244   dynamic ip                    Port-channel5
201    0018.8b78.aaf6   dynamic ip                    Port-channel5
201    0018.8b78.ac4b   dynamic ip                    Port-channel5
201    001a.a0b7.93b5   dynamic ip                    Port-channel5
201    001a.a0b7.cc48   dynamic ip                    Port-channel5
201    001b.d4e4.ec3f   dynamic ip                    Port-channel5
201    001b.d554.5800   dynamic ip                    GigabitEthernet1/47
201    001c.c0f8.1594   dynamic ip                    Port-channel5
201    001d.0932.698a   dynamic ip,other              Port-channel5
201    001d.a298.c7bb   dynamic ip                    Port-channel5
201    001e.4fa5.5317   dynamic ip                    Port-channel5
201    001e.4fbe.edd3   dynamic ip                    Port-channel5
201    001e.4fbf.adff   dynamic ip                    Port-channel5
201    0021.9900.2ddd   dynamic ip                    Port-channel5
201    0021.9900.2df1   dynamic ip                    Port-channel5
201    0050.5689.0035   dynamic ip                    Port-channel5
201    00a0.4557.123f   dynamic ip                    Port-channel5
201    0a00.3e04.69b2   dynamic ip                    Port-channel5
201    0a00.3e10.0b2b   dynamic ip                    Port-channel5
201    20cf.30c9.66a8   dynamic ip,other              Port-channel5
201    20cf.30c9.66ad   dynamic ip,other              Port-channel5
201    20cf.30c9.66b6   dynamic ip,other              Port-channel5
201    20cf.30c9.66bc   dynamic ip,other              Port-channel5
201    20cf.30c9.66bd   dynamic ip,other              Port-channel5
201    bcae.c518.c091   dynamic ip,other              Port-channel5
201    bcae.c518.c0fe   dynamic ip,other              Port-channel5
201    d8d3.858e.7bb9   dynamic ip                    Port-channel5

Multicast Entries
vlan mac address type ports
-------+---------------+-------+--------------------------------------------
201 ffff.ffff.ffff system Gi1/21,Gi1/22,Gi1/23,Gi1/24,Gi1/47,Po5

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Wed, 06 Mar 2013 13:57:47 GMT

Actually I found it. I was looking at the wrong switchport. The 2 switch ports for the primary and secondary were labeled wrong, and therfore Iwas looking at them backwords. The standby ASA switchport had switchport trunk native vlan 201 and no switchport trunk natvie vlan tag commands on it. Not sure why. Once I cleared this the failover status is showing as normal. Beat by the basics. Below is the ouput of the show failover from the primary ASA.

Last Failover at: 07:14:24 EST Mar 4 2013

This host: Primary - Active

Active time: 9442599 (sec)

slot 0: ASA5520 hw/sw rev (2.0/8.2(4)) status (Up Sys)

Interface Canopy (0.0.0.0): Normal (Not-Monitored)

Interface WaterworksCanopy (192.168.x.x): Normal

Interface WCWS_Security (192.168.x.x): Normal

slot 1: ASA-SSM-4GE hw/sw rev (1.0/1.0(0)10) status (Up)

Other host: Secondary - Standby Ready

Active time: 0 (sec)

slot 0: ASA5520 hw/sw rev (2.0/8.2(4)) status (Up Sys)

Interface Canopy (0.0.0.0): Normal (Not-Monitored)

Interface WaterworksCanopy (192.168.x.x.): Normal

Interface WCWS_Security (192.168.x.x): Normal

slot 1: ASA-SSM-4GE hw/sw rev (1.0/1.0(0)10) status (Up)

ASA Secondary FW Sub-Interface Failure

Jouni Forss — Wed, 06 Mar 2013 13:58:03 GMT

Hi,

Is this from the switch connected directly to the Standby ASA?

Would seem that this is possibly the Active ASAs switch since its showing the MAC from a Port-channel5 ?

Which would probably mean that the following MAC address is from the Active ASA

201 001b.d554.5800 dynamic ip GigabitEthernet1/47

I imagine the Standby ASA doesnt show anything for the interface holding Vlan201 with "show arp" command?

- Jouni

ASA Secondary FW Sub-Interface Failure

Dustin Flint — Wed, 06 Mar 2013 17:36:44 GMT

Jouni,

Please see my previous post. I believe I found the isue.