12-16-2018 10:50 AM - edited 02-21-2020 08:34 AM
Description the issue:
I have two ASAs which were configured Active / Standby Fail-over. The issue is one of three servers that reside under this 10.71.0.0/24 subnet cannot reach the primary gateway 10.71.0.1. However, it is able to reach the standby IP 10.71.0.2 which is weird. Other two servers, meanwhile, are able to reach 10.71.0.1 normally and not able to reach 10.71.0.2 which is correct.
I have rebooted the issue server as well as the both ASAs but no lucky. If anyone has clue about this situation?
Below are the configuration:
Primary ASA:
PCCFW1-2/pri/act# show run interface po1.10
interface Port-channel1.10
vlan 10
nameif PCCNet
security-level 100
ip address 10.71.0.1 255.255.255.0 standby 10.71.0.2
Fail over state:
PCCFW1-2/pri/act# show failover state
State Last Failure Reason Date/Time
This host - Primary
Active None
Other host - Secondary
Standby Ready None
====Configuration State===
Sync Done
====Communication State===
Mac set
Active ASA has ip 10.71.0.1 and up up status:
PCCFW1-2/pri/act# show interface ip brief | i 1.10
Port-channel1.10 10.71.0.1 YES CONFIG up up
Standby ASA:
PCCFW1-2/sec/stby# show failover state
State Last Failure Reason Date/Time
This host - Secondary
Standby Ready None
Other host - Primary
Active None
====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set
PCCFW1-2/sec/stby# show interface ip brief | i 1.10
Port-channel1.10 10.71.0.2 YES CONFIG up up
Below I tried to ping the gateway from three servers (they are connected to the ports under same VLAN 10 of stacked 9300 switches - switch mode access)
Server ONE: ip address 10.71.0.12 (CANNOT reach the gateway)
ipconfig:
Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 10.71.0.12
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.71.0.1
C:\Users\Administrator>ping 10.71.0.1
Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.
Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
C:\Users\Administrator>ping 10.71.0.2
Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Server TWO: ip address 10.71.0.10
C:\Users\Administrator>ping 10.71.0.1
Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
C:\Users\Administrator>ping 10.71.0.2
Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.
Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Server THREE: ip address 10.71.0.13
C:\Users\Administrator>ping 10.71.0.1
Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
C:\Users\Administrator>ping 10.71.0.2
Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.
Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
12-16-2018 12:09 PM - edited 12-16-2018 12:21 PM
hello leogxn,
in order to get this sorted could you please capture the traffic on your firewall from the problematic server to gateway.
example is below
capture capin interface inside match ip 192.168.10.10 255.255.255.255
203.0.113.3 255.255.255.255
just curious, could you confirm the all server are connected on stack1 switch or they are on stack1 =server1 and stack2=server2 and stack3=server3.
firewall config looks alright to me.
12-16-2018 03:04 PM
Hi Radio_City,
Three servers are connecting their NICs to both stacks and they are exactly same.
Each servers have 4 NICs and they are teamed together.
Two NICs were connected to stack1 and another two were connected to stack2.
One port of Stack1 and one port of Stack2 were configured as a port-channel2 to the Active ASA. Each Stack has one more port configured as a port-channel3 to the standby ASA
Here is the SW ports configuration - they are all same:
PCCSW1-2#show run | b 1/0/1
interface GigabitEthernet1/0/1
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/2
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/3
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/4
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/5
switchport access vlan 10
switchport mode access
PCCSW1-2#show run | b 2/0/1
interface GigabitEthernet2/0/1
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/2
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/3
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/4
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/5
switchport access vlan 10
switchport mode access
PCCSW1-2#
I tried to capture from primary ASA but nothing was captured from 10.71.0.12 to 10.71.0.1 (I noticed that they are not reachable by each other. While I reload the secondary ASA, 10.71.0.1 becomes reachable from the server, and it will again become unreachable when the secondary ASA come back - 10.71.0.2 becomes reachable from the server)
I cannot ping the problematic server from the primary ASA:
PCCFW1-2/pri/act# ping 10.71.0.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.71.0.12, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)
It is only reachable from secondary ASA:
PCCFW1-2/sec/stby# ping 10.71.0.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.71.0.12, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Here is the capture:
PCCFW1-2/pri/act# show capture
capture CAPIN type raw-data interface PCCNet [Capturing - 0 bytes]
match icmp host 10.71.0.12 host 10.71.0.1
PCCFW1-2/pri/act# show capture CAPIN detail
0 packet captured
0 packet shown
PCCFW1-2/sec/stby# show capture
capture CAPIN type raw-data interface PCCNet [Capturing - 752 bytes]
match icmp host 10.71.0.12 host 10.71.0.2
PCCFW1-2/sec/stby# show capture CAPIN detail
8 packets captured
1: 17:32:30.565354 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9898)
2: 17:32:30.565476 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 3815)
3: 17:32:31.574845 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9938)
4: 17:32:31.574936 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 10654)
5: 17:32:32.590408 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9984)
6: 17:32:32.590484 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 1072)
7: 17:32:33.605925 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 10005)
8: 17:32:33.606001 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 14345)
8 packets shown
12-17-2018 03:44 AM
I assume might be it could be an issue with firewall configuration. as your config on the switch look fine as so the ASA interface config too looks good.
could you share the following output from the both boxes
show run failover
show failover (i know you did post the output of this command)
also give the config of the switch where this failover is configured.
12-17-2018 09:35 AM
PCCFW1-2/pri/act# sh run failover
failover
failover lan unit primary
failover lan interface FO GigabitEthernet0/6
failover link STATE GigabitEthernet0/7
failover interface ip FO 10.10.11.1 255.255.255.252 standby 10.10.11.2
failover interface ip STATE 10.10.11.5 255.255.255.252 standby 10.10.11.6
PCCFW1-2/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FO GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown
Last Failover at: 10:36:50 EST Dec 17 2018
This host: Primary - Active
Active time: 4046 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.1): Normal (Monitored)
Interface PCCNet (10.71.0.1): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.2): Normal (Monitored)
Interface PCCNet (10.71.0.2): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Stateful Failover Logical Update Statistics
Link : STATE GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 7774 0 540 0
sys cmd 540 0 540 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 127 0 0 0
UDP conn 109 0 0 0
ARP tbl 6993 0 0 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 4 0 0 0
Router ID 0 0 0 0
User-Identity 1 0 0 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 10 4595
Xmit Q: 0 30 9999
PCCSW1-2#show run | b 1/0/47
interface GigabitEthernet1/0/47
switchport mode trunk
speed 1000
channel-group 3 mode on
!
interface GigabitEthernet1/0/48
description L2 PCCFW Secondary G0/1
switchport mode trunk
speed 1000
channel-group 2 mode on
Here is the switch configuration for those port channels:
PCCSW1-2#show run | b 2/0/47
interface GigabitEthernet2/0/47
description L2 PCCFW Secondary G0/2
switchport mode trunk
speed 1000
channel-group 3 mode on
!
interface GigabitEthernet2/0/48
switchport mode trunk
speed 1000
channel-group 2 mode on
12-17-2018 09:41 AM - edited 12-17-2018 09:49 AM
yes, we got a problem
MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown
Interface PCCNet (10.71.0.1): Normal (Waiting)
give me the output of passive firewall , show run failover
and aslo run the command on both active and passive boxes
show run Interface PCCNet
12-17-2018 09:46 AM
So you mean I have set the notification interval? Is the serial number "Mate Unknown" will cause the issue?
12-17-2018 09:51 AM
from active firewall when you issue command show failover should show its mate, but here we see unknown means somehow active ASA do not see the passive firewall.
12-18-2018 07:16 PM
Hi,
I just got a chance to grab the output from the devices.
I noticed that both ASA cannot ping each other using these port-channel subinterfaces. I have another backup site which has identical topology and connections between the ASA and switches. They works fine and can ping each other.
Here is the outputs of two physical interfaces of problematic ASAs. I am not sure if 1 interface resets is a clue of this issue.
PCCFW1-2/pri/act# show int g0/1
Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 0027.e322.54f3, MTU not set
IP address unassigned
648234 packets input, 56053686 bytes, 0 no buffer
Received 13853 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
452002 packets output, 38161543 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (478/457)
output queue (blocks free curr/low): hardware (493/441)
PCCFW1-2/pri/act# show int g0/2
Interface GigabitEthernet0/2 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 0027.e322.54f8, MTU not set
IP address unassigned
427978 packets input, 140813400 bytes, 0 no buffer
Received 61953 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
380876 packets output, 46451354 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (499/456)
output queue (blocks free curr/low): hardware (469/443)
PCCFW1-2/sec/stby# show int g0/1
Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 700f.6ac0.aa94, MTU not set
IP address unassigned
457843 packets input, 41893242 bytes, 0 no buffer
Received 15215 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
403996 packets output, 34045120 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (490/457)
output queue (blocks free curr/low): hardware (511/504)
Here are the required show results:
PCCFW1-2/sec/stby# show run int po1.10
interface Port-channel1.10
vlan 10
nameif PCCNet
security-level 100
ip address 10.71.0.1 255.255.255.0 standby 10.71.0.2
PCCFW1-2/sec/stby# show failov
Failover On
Failover unit Secondary
Failover LAN Interface: FO GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown
Last Failover at: 10:36:45 EST Dec 17 2018
This host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.2): Normal (Monitored)
Interface PCCNet (10.71.0.2): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Other host: Primary - Active
Active time: 122915 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.1): Normal (Monitored)
Interface PCCNet (10.71.0.1): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Stateful Failover Logical Update Statistics
Link : STATE GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 16387 0 74426 8
sys cmd 16387 0 16387 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 1328 0
UDP conn 0 0 1621 0
ARP tbl 0 0 55079 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 0 0 10 8
Router ID 0 0 0 0
User-Identity 0 0 1 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0
Logical Update Queue Information
Cur Max Total
Recv Q: 0 13 263992
Xmit Q: 0 1 16387
12-17-2018 10:01 AM - edited 12-17-2018 01:43 PM
Apologies, too much information flicked my eyes.
yes sorry you have acive passive configured properly
Version: Ours 9.8(2), Mate 9.8(2)
however, you have issue with Interface PCCNet (10.71.0.1): Normal (Waiting)
this suppose to be a monitor but in our case its waiting. that could explain why you can not ping from different server to this address.
could you please confirm if you have this command on your firewall monitor-interface PCCNet i assume you have configured this that is why its showing waiting.
1.can you ping from active firewall to address 10.71.0.1 and can you also ping 10.71.0.2 from the active firewall
2.can you ping form passive firewall to address 10.10.0.1 and can you also ping 10.71.0.2 for the passive firewall.
I assume you have issue in between interface PCCNET.
action plan
1. Make sure to check the ports are up on the etherchannel on switch side and on the firewall both Acitve/Passive
at switch side issue command show etherchannel summary this will show you all port ups or any port down.
2. as mention above ping from the firewall from active and passive the ip address of PCCNET.
Regards,
Radio_City
12-19-2018 01:12 AM