cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3874
Views
5
Helpful
15
Replies

Wierd Issue - one server cannot reach primary gateway IP - ASA Active / Standby Failover

leogxn
Level 1
Level 1

Description the issue:

I have two ASAs which were configured Active / Standby Fail-over.  The issue is one of three servers that reside under this 10.71.0.0/24 subnet cannot reach the primary gateway 10.71.0.1. However, it is able to reach the standby IP 10.71.0.2 which is weird. Other two servers, meanwhile, are able to reach 10.71.0.1 normally and not able to reach 10.71.0.2 which is correct.

 

I have rebooted the issue server as well as the both ASAs but no lucky. If anyone has clue about this situation? 

 

Below are the configuration:

 

Primary ASA:

 

PCCFW1-2/pri/act# show run interface po1.10

interface Port-channel1.10
vlan 10
nameif PCCNet
security-level 100
ip address 10.71.0.1 255.255.255.0 standby 10.71.0.2

 

Fail over state:


PCCFW1-2/pri/act# show failover state

                            State                                   Last Failure                      Reason Date/Time
This host -            Primary
                            Active                                  None
Other host -         Secondary
                            Standby Ready                    None

====Configuration State===
Sync Done
====Communication State===
Mac set

 

Active ASA has  ip 10.71.0.1 and up up status:

 

PCCFW1-2/pri/act# show interface ip brief | i 1.10
Port-channel1.10 10.71.0.1 YES CONFIG up up

 

 

Standby ASA:

 

PCCFW1-2/sec/stby# show failover state

                               State                         Last Failure                   Reason Date/Time
This host -              Secondary
                              Standby Ready           None
Other host -            Primary
                              Active                         None

====Configuration State===
Sync Done - STANDBY
====Communication State===
Mac set

PCCFW1-2/sec/stby# show interface ip brief | i 1.10

Port-channel1.10           10.71.0.2       YES CONFIG up                    up

 

 

 

Below I tried to ping the gateway from three servers (they are connected to the ports under same VLAN 10 of stacked 9300 switches - switch mode access)

 

Server ONE: ip address 10.71.0.12 (CANNOT reach the gateway)

ipconfig:

Connection-specific DNS Suffix . :
IPv4 Address. . . . . . . . . . . : 10.71.0.12
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.71.0.1

 

C:\Users\Administrator>ping 10.71.0.1

Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.
Reply from 10.71.0.12: Destination host unreachable.

Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

 

C:\Users\Administrator>ping 10.71.0.2

Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255
Reply from 10.71.0.2: bytes=32 time<1ms TTL=255

Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

 

Server TWO: ip address 10.71.0.10

C:\Users\Administrator>ping 10.71.0.1

Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255

Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Users\Administrator>ping 10.71.0.2

Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.
Reply from 10.71.0.10: Destination host unreachable.

Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

 

Server THREE: ip address 10.71.0.13

C:\Users\Administrator>ping 10.71.0.1

Pinging 10.71.0.1 with 32 bytes of data:
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255
Reply from 10.71.0.1: bytes=32 time<1ms TTL=255

Ping statistics for 10.71.0.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms

 

C:\Users\Administrator>ping 10.71.0.2

Pinging 10.71.0.2 with 32 bytes of data:
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.
Reply from 10.71.0.13: Destination host unreachable.

Ping statistics for 10.71.0.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

15 Replies 15

hello leogxn,

 

in order to get this sorted could you please capture the traffic on your firewall from the problematic server to gateway.

 

example is below

 

capture capin interface inside match ip 192.168.10.10 255.255.255.255
203.0.113.3 255.255.255.255

 

just curious, could you confirm the all server are connected on stack1 switch or they are on stack1 =server1 and stack2=server2 and stack3=server3.

 

firewall config looks alright to me.

please do not forget to rate.

Hi Radio_City,

 

Three servers are connecting their NICs to both stacks and they are exactly same.

Each servers have 4 NICs and they are teamed together.

Two NICs were connected to stack1 and another two were connected to stack2.

One port of Stack1 and one port of Stack2 were configured as a port-channel2 to the Active ASA. Each Stack has one more port configured as a port-channel3 to the standby ASA

 

Here is the SW ports configuration - they are all same:

PCCSW1-2#show run | b 1/0/1
interface GigabitEthernet1/0/1
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/2
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/3
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/4
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/5
switchport access vlan 10
switchport mode access

 

PCCSW1-2#show run | b 2/0/1
interface GigabitEthernet2/0/1
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/2
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/3
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/4
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet2/0/5
switchport access vlan 10
switchport mode access

PCCSW1-2#

 

 

I tried to capture from primary ASA but nothing was captured from 10.71.0.12 to 10.71.0.1 (I noticed that they are not reachable by each other. While I reload the secondary ASA, 10.71.0.1 becomes reachable from the server, and it will again become unreachable when the secondary ASA come back - 10.71.0.2 becomes reachable from the server)

 

I cannot ping the problematic server from the primary ASA:

PCCFW1-2/pri/act# ping 10.71.0.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.71.0.12, timeout is 2 seconds:
?????
Success rate is 0 percent (0/5)

 

It is only reachable from secondary ASA:
PCCFW1-2/sec/stby# ping 10.71.0.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.71.0.12, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

 

Here is the capture:

 

PCCFW1-2/pri/act# show capture

capture CAPIN type raw-data interface PCCNet [Capturing - 0 bytes]

  match icmp host 10.71.0.12 host 10.71.0.1

 

PCCFW1-2/pri/act# show capture CAPIN detail

0 packet captured

0 packet shown

 

 

 

PCCFW1-2/sec/stby# show capture

capture CAPIN type raw-data interface PCCNet [Capturing - 752 bytes]

 match icmp host 10.71.0.12 host 10.71.0.2

PCCFW1-2/sec/stby# show capture CAPIN detail

8 packets captured

1: 17:32:30.565354 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9898)
2: 17:32:30.565476 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 3815)
3: 17:32:31.574845 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9938)
4: 17:32:31.574936 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 10654)
5: 17:32:32.590408 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 9984)
6: 17:32:32.590484 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 1072)
7: 17:32:33.605925 00b7.71ff.3525 700f.6ac0.aa94 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.12 > 10.71.0.2: icmp: echo request (ttl 128, id 10005)
8: 17:32:33.606001 700f.6ac0.aa94 00b7.71ff.3525 0x8100 Length: 78
802.1Q vlan#10 P0 10.71.0.2 > 10.71.0.12: icmp: echo reply (ttl 255, id 14345)
8 packets shown

I assume might be it could be an issue with firewall configuration. as your config on the switch look fine as so the ASA interface config too looks good.

 

 

could you share the following output from the both boxes

 

show run failover

show failover (i know you did post the output of this command)

 

also give the config of the switch where this failover is configured.

 

 

 

please do not forget to rate.

PCCFW1-2/pri/act# sh run failover
failover
failover lan unit primary
failover lan interface FO GigabitEthernet0/6
failover link STATE GigabitEthernet0/7
failover interface ip FO 10.10.11.1 255.255.255.252 standby 10.10.11.2
failover interface ip STATE 10.10.11.5 255.255.255.252 standby 10.10.11.6

 

PCCFW1-2/pri/act# show failover
Failover On
Failover unit Primary
Failover LAN Interface: FO GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown
Last Failover at: 10:36:50 EST Dec 17 2018
This host: Primary - Active
Active time: 4046 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.1): Normal (Monitored)
Interface PCCNet (10.71.0.1): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Other host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.2): Normal (Monitored)
Interface PCCNet (10.71.0.2): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : STATE GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 7774 0 540 0
sys cmd 540 0 540 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 127 0 0 0
UDP conn 109 0 0 0
ARP tbl 6993 0 0 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 4 0 0 0
Router ID 0 0 0 0
User-Identity 1 0 0 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 10 4595
Xmit Q: 0 30 9999

 

PCCSW1-2#show run | b 1/0/47
interface GigabitEthernet1/0/47
switchport mode trunk
speed 1000
channel-group 3 mode on
!
interface GigabitEthernet1/0/48
description L2 PCCFW Secondary G0/1
switchport mode trunk
speed 1000
channel-group 2 mode on

 

Here is the switch configuration for those port channels:

 

PCCSW1-2#show run | b 2/0/47
interface GigabitEthernet2/0/47
description L2 PCCFW Secondary G0/2
switchport mode trunk
speed 1000
channel-group 3 mode on
!
interface GigabitEthernet2/0/48
switchport mode trunk
speed 1000
channel-group 2 mode on

yes, we got a problem

 

MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown 

 

 

Interface PCCNet (10.71.0.1): Normal (Waiting)

 

 

give me the output of passive firewall , show run failover 

 

and aslo run the command on both active and passive boxes

show run Interface PCCNet

please do not forget to rate.

So you mean I have set the notification interval? Is the serial number "Mate Unknown" will cause the issue?

from active firewall when you issue command show failover should show its mate, but here we see unknown means somehow active ASA do not see the passive firewall.

please do not forget to rate.

Hi,

 

I just got a chance to grab the output from the devices.

 

I noticed that both ASA cannot ping each other using these port-channel subinterfaces. I have another backup site which has identical topology and connections between the ASA and switches. They works fine and can ping each other.

 

Here is the outputs of two physical interfaces of problematic ASAs.  I am not sure if 1 interface resets is a clue of this issue.

PCCFW1-2/pri/act# show int g0/1
Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 0027.e322.54f3, MTU not set
IP address unassigned
648234 packets input, 56053686 bytes, 0 no buffer
Received 13853 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
452002 packets output, 38161543 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (478/457)
output queue (blocks free curr/low): hardware (493/441)
PCCFW1-2/pri/act# show int g0/2
Interface GigabitEthernet0/2 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 0027.e322.54f8, MTU not set
IP address unassigned
427978 packets input, 140813400 bytes, 0 no buffer
Received 61953 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
380876 packets output, 46451354 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (499/456)
output queue (blocks free curr/low): hardware (469/443)

 

PCCFW1-2/sec/stby# show int g0/1
Interface GigabitEthernet0/1 "", is up, line protocol is up
Hardware is i82574L rev00, BW 1000 Mbps, DLY 10 usec
Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
Input flow control is unsupported, output flow control is off
Active member of Port-channel1
MAC address 700f.6ac0.aa94, MTU not set
IP address unassigned
457843 packets input, 41893242 bytes, 0 no buffer
Received 15215 broadcasts, 0 runts, 0 giants
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 pause input, 0 resume input
0 L2 decode drops
403996 packets output, 34045120 bytes, 0 underruns
0 pause output, 0 resume output
0 output errors, 0 collisions, 1 interface resets
0 late collisions, 0 deferred
0 input reset drops, 0 output reset drops
input queue (blocks free curr/low): hardware (490/457)
output queue (blocks free curr/low): hardware (511/504)

 

Here are the required show results:

PCCFW1-2/sec/stby# show run int po1.10

interface Port-channel1.10
vlan 10
nameif PCCNet
security-level 100
ip address 10.71.0.1 255.255.255.0 standby 10.71.0.2

 

PCCFW1-2/sec/stby# show failov
Failover On
Failover unit Secondary
Failover LAN Interface: FO GigabitEthernet0/6 (up)
Reconnect timeout 0:00:00
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 216 maximum
MAC Address Move Notification Interval not set
Version: Ours 9.8(2), Mate 9.8(2)
Serial Number: Ours xxx, Mate Unknown
Last Failover at: 10:36:45 EST Dec 17 2018
This host: Secondary - Standby Ready
Active time: 0 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.2): Normal (Monitored)
Interface PCCNet (10.71.0.2): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
Other host: Primary - Active
Active time: 122915 (sec)
slot 0: ASA5525 hw/sw rev (3.1/9.8(2)) status (Up Sys)
Interface INTER-FW (11.11.11.1): Normal (Monitored)
Interface PCCNet (10.71.0.1): Normal (Waiting)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)
slot 1: SFR5525 hw/sw rev (N/A/6.2.2-81) status (Up/Up)
ASA FirePOWER, 6.2.2-81, Up, (Monitored)

Stateful Failover Logical Update Statistics
Link : STATE GigabitEthernet0/7 (up)
Stateful Obj xmit xerr rcv rerr
General 16387 0 74426 8
sys cmd 16387 0 16387 0
up time 0 0 0 0
RPC services 0 0 0 0
TCP conn 0 0 1328 0
UDP conn 0 0 1621 0
ARP tbl 0 0 55079 0
Xlate_Timeout 0 0 0 0
IPv6 ND tbl 0 0 0 0
VPN IKEv1 SA 0 0 0 0
VPN IKEv1 P2 0 0 0 0
VPN IKEv2 SA 0 0 0 0
VPN IKEv2 P2 0 0 0 0
VPN CTCP upd 0 0 0 0
VPN SDI upd 0 0 0 0
VPN DHCP upd 0 0 0 0
SIP Session 0 0 0 0
SIP Tx 0 0 0 0
SIP Pinhole 0 0 0 0
Route Session 0 0 10 8
Router ID 0 0 0 0
User-Identity 0 0 1 0
CTS SGTNAME 0 0 0 0
CTS PAC 0 0 0 0
TrustSec-SXP 0 0 0 0
IPv6 Route 0 0 0 0
STS Table 0 0 0 0

Logical Update Queue Information
Cur Max Total
Recv Q: 0 13 263992
Xmit Q: 0 1 16387

Apologies, too much information flicked my eyes.
yes sorry you have acive passive configured properly
Version: Ours 9.8(2), Mate 9.8(2)
however, you have issue with Interface PCCNet (10.71.0.1): Normal (Waiting)

this suppose to be a monitor but in our case its waiting. that could explain why you can not ping from different server to this address.

 

could you please confirm if you have this command on your firewall monitor-interface PCCNet  i assume you have configured this that is why its showing waiting.

 

1.can you ping from active firewall to address 10.71.0.1 and can you also ping 10.71.0.2 from the active firewall

2.can you ping form passive firewall to address 10.10.0.1 and can you also ping 10.71.0.2 for the passive firewall.

 

I assume you have issue in between interface PCCNET.

 

 

action plan

1. Make sure to check the ports are up on the etherchannel on switch side and on the firewall both Acitve/Passive

    at switch side issue command show etherchannel summary this will show you all port ups or any port down.

2. as mention above ping from the firewall from active and passive the ip address of PCCNET.

 

 

 

 

Regards,

Radio_City

please do not forget to rate.

Interface PCCNet (10.71.0.2): Normal (Waiting) is noted on both ASA.
this above output shows we have issue on this interface PCCNet.

ASA config are all good. could you please get this information from the both ASAs port-channel1 is connected to switch port-channel. could you show the output on that switch "show etherchannel summary" the reason we see the interface reset is could be some or one of port channel interface is down.

the other strange think i noted is your firewall see the mate version but it down not show the serial no
"Serial Number: Ours xxx, Mate Unknown"
does is show to you on another side same output? i have checked my firewall 5555-X they showed the serial no though.
please do not forget to rate.