Wierd Issue - one server cannot reach primary gateway IP - ASA Active / Standby Failover

[leogxn]

Description the issue:

I have two ASAs which were configured Active / Standby Fail-over.  The issue is one of three servers that reside under this 10.71.0.0/24 subnet cannot reach the primary gateway 10.71.0.1. However, it is able to reach the standby IP 10.71.0.2 which is weird. Other two servers, meanwhile, are able to reach 10.71.0.1 normally and not able to reach 10.71.0.2 which is correct.

I have rebooted the issue server as well as the both ASAs but no lucky. If anyone has clue about this situation? 

Below are the configuration:

Primary ASA:

PCCFW1-2/pri/act# show run interface po1.10

interface Port-channel1.10
vlan 10
nameif PCCNet
security-level 100
ip address 10.71.0.1 255.255.255.0 standby 10.71.0.2

Fail over state:

PCCFW1-2/pri/act# show failover state

                            State                                   Last Failure                      Reason Date/Time
This host -            Primary
                            Active                                  None
Other host -         Secondary
                            Standby Ready                    None

====Configuration State===
Sync Done
====Communication State===
Mac set

Active ASA has  ip 10.71.0.1 and up up status:

PCCFW1-2/pri/act# show interface ip brief | i 1.10
Port-channel1.10 10.71.0.1 YES CONFIG up up

Standby ASA:

PCCFW1-2/sec/stby# show failover state

                               State                         Last Failure                   Reason Date/Time
This host -              Secondary
                              Standby Ready           None
Other host -            Primary
                              Active                         None

====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set

PCCFW1-2/sec/stby# show interface ip brief | i 1.10

Port-channel1.10           10.71.0.2       YES CONFIG up                    up

Below I tried to ping the gateway from three servers (they are connected to the ports under same VLAN 10 of stacked 9300 switches - switch mode access)

Server ONE: ip address 10.71.0.12 (CANNOT reach the gateway)

ipconfig:

Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 10.71.0.12
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.71.0.1

C:\Users\Administrator>ping 10.71.0.1

Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.

Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

C:\Users\Administrator>ping 10.71.0.2

Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255

Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

Server TWO: ip address 10.71.0.10

C:\Users\Administrator>ping 10.71.0.1

Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255

Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Users\Administrator>ping 10.71.0.2

Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.

Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Server THREE: ip address 10.71.0.13

C:\Users\Administrator>ping 10.71.0.1

Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255

Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Users\Administrator>ping 10.71.0.2

Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.

Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

[Sheraz.Salim]

hello leogxn,

in order to get this sorted could you please capture the traffic on your firewall from the problematic server to gateway.

example is below

capture capin interface inside match ip 192.168.10.10 255.255.255.255
 203.0.113.3 255.255.255.255

just curious, could you confirm the all server are connected on stack1 switch or they are on stack1 =server1 and stack2=server2 and stack3=server3.

firewall config looks alright to me.

[leogxn]

Hi Radio_City,

Three servers are connecting their NICs to both stacks and they are exactly same.

Each servers have 4 NICs and they are teamed together.

Two NICs were connected to stack1 and another two were connected to stack2.

One port of Stack1 and one port of Stack2 were configured as a port-channel2 to the Active ASA. Each Stack has one more port configured as a port-channel3 to the standby ASA

Here is the SW ports configuration - they are all same:

PCCSW1-2#show run | b 1/0/1
interface GigabitEthernet1/0/1
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/2
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/3
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/4
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/5
switchport access vlan 10
switchport mode access

PCCSW1-2#show run | b 2/0/1
interface GigabitEthernet2/0/1
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/2
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/3
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/4
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/5
switchport access vlan 10
switchport mode access

PCCSW1-2#

I tried to capture from primary ASA but nothing was captured from 10.71.0.12 to 10.71.0.1 (I noticed that they are not reachable by each other. While I reload the secondary ASA, 10.71.0.1 becomes reachable from the server, and it will again become unreachable when the secondary ASA come back - 10.71.0.2 becomes reachable from the server)

I cannot ping the problematic server from the primary ASA:

PCCFW1-2/pri/act# ping 10.71.0.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.71.0.12, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)

It is only reachable from secondary ASA:
PCCFW1-2/sec/stby# ping 10.71.0.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.71.0.12, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Here is the capture:

PCCFW1-2/pri/act# show capture

capture CAPIN type raw-data interface PCCNet [Capturing - 0 bytes]

  match icmp host 10.71.0.12 host 10.71.0.1

PCCFW1-2/pri/act# show capture CAPIN detail

0 packet captured

0 packet shown

PCCFW1-2/sec/stby# show capture

capture CAPIN type raw-data interface PCCNet [Capturing - 752 bytes]

 match icmp host 10.71.0.12 host 10.71.0.2

PCCFW1-2/sec/stby# show capture CAPIN detail

8 packets captured

1: 17:32:30.565354 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9898)
2: 17:32:30.565476 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 3815)
3: 17:32:31.574845 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9938)
4: 17:32:31.574936 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 10654)
5: 17:32:32.590408 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9984)
6: 17:32:32.590484 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 1072)
7: 17:32:33.605925 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 10005)
8: 17:32:33.606001 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 14345)
8 packets shown

[Sheraz.Salim]

I assume might be it could be an issue with firewall configuration. as your config on the switch look fine as so the ASA interface config too looks good.

could you share the following output from the both boxes

show run failover

show failover (i know you did post the output of this command)

also give the config of the switch where this failover is configured.

[leogxn]

PCCFW1-2/pri/act# sh run failover
failover
failover lan unit primary
failover lan interface FO GigabitEthernet0/6
failover link STATE GigabitEthernet0/7
failover interface ip FO 10.10.11.1 255.255.255.252 standby 10.10.11.2
failover interface ip STATE 10.10.11.5 255.255.255.252 standby 10.10.11.6

PCCFW1-2/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FO GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown
Last Failover at: 10:36:50 EST Dec 17 2018
This host: Primary - Active
Active time: 4046 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.1): Normal (Monitored)
Interface PCCNet (10.71.0.1): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.2): Normal (Monitored)
Interface PCCNet (10.71.0.2): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : STATE GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 7774 0 540 0
sys cmd 540 0 540 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 127 0 0 0
UDP conn 109 0 0 0
ARP tbl 6993 0 0 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 4 0 0 0
Router ID 0 0 0 0
User-Identity 1 0 0 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 10 4595
Xmit Q: 0 30 9999

PCCSW1-2#show run | b 1/0/47
interface GigabitEthernet1/0/47
switchport mode trunk
speed 1000
channel-group 3 mode on
!
interface GigabitEthernet1/0/48
description L2 PCCFW Secondary G0/1
switchport mode trunk
speed 1000
channel-group 2 mode on

Here is the switch configuration for those port channels:

PCCSW1-2#show run | b 2/0/47
interface GigabitEthernet2/0/47
description L2 PCCFW Secondary G0/2
switchport mode trunk
speed 1000
channel-group 3 mode on
!
interface GigabitEthernet2/0/48
switchport mode trunk
speed 1000
channel-group 2 mode on

[Sheraz.Salim]

yes, we got a problem

MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown 

Interface PCCNet (10.71.0.1): Normal (Waiting)

give me the output of passive firewall , show run failover 

and aslo run the command on both active and passive boxes

show run Interface PCCNet

[leogxn]

So you mean I have set the notification interval? Is the serial number "Mate Unknown" will cause the issue?

[Sheraz.Salim]

from active firewall when you issue command show failover should show its mate, but here we see unknown means somehow active ASA do not see the passive firewall.

[leogxn]

Hi,

I just got a chance to grab the output from the devices.

I noticed that both ASA cannot ping each other using these port-channel subinterfaces. I have another backup site which has identical topology and connections between the ASA and switches. They works fine and can ping each other.

Here is the outputs of two physical interfaces of problematic ASAs.  I am not sure if 1 interface resets is a clue of this issue.

PCCFW1-2/pri/act# show int g0/1
Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 0027.e322.54f3, MTU not set
IP address unassigned
648234 packets input, 56053686 bytes, 0 no buffer
Received 13853 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
452002 packets output, 38161543 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (478/457)
output queue (blocks free curr/low): hardware (493/441)
PCCFW1-2/pri/act# show int g0/2
Interface GigabitEthernet0/2 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 0027.e322.54f8, MTU not set
IP address unassigned
427978 packets input, 140813400 bytes, 0 no buffer
Received 61953 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
380876 packets output, 46451354 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (499/456)
output queue (blocks free curr/low): hardware (469/443)

PCCFW1-2/sec/stby# show int g0/1
Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 700f.6ac0.aa94, MTU not set
IP address unassigned
457843 packets input, 41893242 bytes, 0 no buffer
Received 15215 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
403996 packets output, 34045120 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (490/457)
output queue (blocks free curr/low): hardware (511/504)

Here are the required show results:

PCCFW1-2/sec/stby# show run int po1.10

interface Port-channel1.10
vlan 10
nameif PCCNet
security-level 100
ip address 10.71.0.1 255.255.255.0 standby 10.71.0.2

PCCFW1-2/sec/stby# show failov
Failover On
Failover unit Secondary
Failover LAN Interface: FO GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown
Last Failover at: 10:36:45 EST Dec 17 2018
This host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.2): Normal (Monitored)
Interface PCCNet (10.71.0.2): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Other host: Primary - Active
Active time: 122915 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.1): Normal (Monitored)
Interface PCCNet (10.71.0.1): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : STATE GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 16387 0 74426 8
sys cmd 16387 0 16387 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 1328 0
UDP conn 0 0 1621 0
ARP tbl 0 0 55079 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 0 0 10 8
Router ID 0 0 0 0
User-Identity 0 0 1 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 13 263992
Xmit Q: 0 1 16387

[Sheraz.Salim]

Apologies, too much information flicked my eyes.
yes sorry you have acive passive configured properly
Version: Ours 9.8(2), Mate 9.8(2)
however, you have issue with Interface PCCNet (10.71.0.1): Normal (Waiting)

this suppose to be a monitor but in our case its waiting. that could explain why you can not ping from different server to this address.

could you please confirm if you have this command on your firewall monitor-interface PCCNet  i assume you have configured this that is why its showing waiting.

1.can you ping from active firewall to address 10.71.0.1 and can you also ping 10.71.0.2 from the active firewall

2.can you ping form passive firewall to address 10.10.0.1 and can you also ping 10.71.0.2 for the passive firewall.

I assume you have issue in between interface PCCNET.

action plan

1. Make sure to check the ports are up on the etherchannel on switch side and on the firewall both Acitve/Passive

    at switch side issue command show etherchannel summary this will show you all port ups or any port down.

2. as mention above ping from the firewall from active and passive the ip address of PCCNET.

Regards,

Radio_City

[Sheraz.Salim]

Interface PCCNet (10.71.0.2): Normal (Waiting) is noted on both ASA.
this above output shows we have issue on this interface PCCNet.

ASA config are all good. could you please get this information from the both ASAs port-channel1 is connected to switch port-channel. could you show the output on that switch "show etherchannel summary" the reason we see the interface reset is could be some or one of port channel interface is down.

the other strange think i noted is your firewall see the mate version but it down not show the serial no
"Serial Number: Ours xxx, Mate Unknown"
does is show to you on another side same output? i have checked my firewall 5555-X they showed the serial no though.

[leogxn]

Hi,

I checked both ASA 5525 pair, none of them can see mate SN #. I do not see the port-channel issue, they are all up in use.

PCCFW1-2/pri/act# show port-channel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
U - in use N - not in use, no aggregation/nameif
M - not in use, no aggregation due to minimum links not met
w - waiting to be aggregated
Number of channel-groups in use: 1
Group Port-channel Protocol Span-cluster Ports
------+-------------+---------+------------+------------------------------------
1 Po1(U) - No Gi0/1(P) Gi0/2(P)

PCCFW1-2/sec/stby# show port-channel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
U - in use N - not in use, no aggregation/nameif
M - not in use, no aggregation due to minimum links not met
w - waiting to be aggregated
Number of channel-groups in use: 1
Group Port-channel Protocol Span-cluster Ports
------+-------------+---------+------------+------------------------------------
1 Po1(U) - No Gi0/1(P) Gi0/2(P)

PCCSW1-2#show etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator

M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

A - formed by Auto LAG

Number of channel-groups in use: 2
Number of aggregators: 2

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
2 Po2(SU) - Gi1/0/48(P) Gi2/0/48(P)
3 Po3(SU) - Gi1/0/47(P) Gi2/0/47(P)

One more thing I noticed was that the packet-tracer showed the packet was dropped by implicit rule but I had access-lists allow any any in / out on that interface:

PCCFW1-2/pri/act# packet-tracer input PCCNet tcp 10.71.0.1 1234 10.71.0.2 http detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 10.71.0.2 using egress ifc PCCNet

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaac8d30e80, priority=501, domain=permit, deny=true
hits=3, user_data=0x7, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=10.71.0.1, mask=255.255.255.255, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=PCCNet, output_ifc=any

Result:
input-interface: PCCNet
input-status: up
input-line-status: up
output-interface: PCCNet
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

PCCFW1-2/pri/act# show run | i access
access-list PCCNET_ACCESS_IN extended permit ip any any
access-list PCCNET_ACCESS_OUT extended permit ip any any

access-group PCCNET_ACCESS_IN in interface PCCNet
access-group PCCNET_ACCESS_OUT out interface PCCNet

[Sheraz.Salim]

Hi Leogxn,

had spent a lot of time in your output given configuration from beginning  to end. to me its seems like you have an issue with switch port channel.

I shall try to simple and easy.

if you draw the topology of your switch network

=================================================================

 switch 1/0/48 Port-channel 2 --------->g0/1 Port-channel1 standby firewall

 switch 2/0/47 Port-channel 3 --------->g0/2 Port-channel1 standby firewall

switch 1/0/47  Port-channel 3--------->gig0/1 Port-channel1 active firewall

switch 2/0/48  Port-channel 2--------->gig0/2 Port-channel1 active firewall

=================================================================

so you pay attention on the above configuration why

     port 1/0/48, 2/0/47 are in different port-channel 2 and 3.

they should be in one port-channel X

same for the other config interfaces

   port 1/0/47,2/0/48 are in different port-channel 3 and 2

having said that, the correct sample config suppose to be like this.

!
interface range gig 1/0/48,gig2/0/47
 channel-group 2 mode active
 switchport mode trunk
 switchport trunk allowed vlan add xxxxxx
 no shut
!
interface channel-group 2
 switchport mode trunk
 switchport trunk allowed vlan add xxxxxx
!
interface range gig 1/0/47,gig2/0/48
channel-group 3 mode active
 switchport mode trunk
 switchport trunk allowed vlan add xxxxxx
 no shut
!
interface channel-group 3
 switchport mode trunk
 switchport trunk allowed vlan add xxxxxx

in regards to packet-tracer command why you doing a trace from active firewall ip to passive firewall. your access list is fine. nothing to worry about.

please dont forget to rate if i was helpful.

[leogxn]

Hi,

Here is the diagram:

Each Stack has a port to form one channel group so whichever the ASA device is active, the port-channel will link to both stacks. Let me double check if I actually configured the correct channel groups.

[leogxn]

Hi Radio_City, I think you are right. All configure was actually correct. The description of port-channel was added by me when I did port tracking remotely but I did not pay attention to the port-channel number. When they remotely reconnected the network cables, they accidentally swapped the port 47 and 48 on stack2 which caused this issue.

Thanks for your time and point out the issue.