07-12-2013 01:16 PM - edited 03-11-2019 07:11 PM
To whoever can help:
I've setup a pair of ASA 5515X firewalls in an active/standby failover group. When I issue the command ip address 172.16.1.2 255.255.255.0 standby 172.16.1.3 from the management interface configuration mode, I'm able to ping and connect via ssh or asdm for period of time. If I let the interface go idle for any extended period of time and then attempt to ping it again, I get one ping off and then "Request timed out." The only way to get the interface to begin accepting traffic again is to change the IP address again, or reload each device in sequence.
I've tested this effect from 2 different workstations on the same network and both end up with the same results: a single ping and then "request timed out." I've disbled the firewalls on each of the workstations (not that it would matter) just to make sure I had an open line of communication with the ASA, yet the problem persists. This occurs even if I set an IP with no standby on each unit.
The weird thing is, the switches do not experience this behavior when attempting to ping or ssh to the ASAs. My workstation will not be able to ping it, but the switch that is connected to the management is able to ping and SSH to the ASAs with no problems. My current setup is like this:
C3750 Switch
Gi1/0/18 (VLAN 999) = Management Interface on ASA1 172.16.1.2
Gi2/0/18 (VLAN 999) = Management Interface on ASA2 172.16.1.3
Gi1/0/10 (VLAN 999) = Workstation 172.16.1.100
SVI 999 = 172.16.1.1
I've set console logging to debug on the ASAs and watched as the first ping came in...and then nothing. So it almost seems as though as if the problem is on the C3750 because the packets just aren't making it to the management interface. Just writing this out has sorta given me a logical perspective to maybe analyze the switch a bit more in depth. I'd still like to see if anyone has any ideas as to what might be causing this behavior.
Thanks for any insight anyone might be able to provide.
John H
.
Solved! Go to Solution.
07-12-2013 05:05 PM
Hi,
The information you have given does seem to point to a wierd situation.
Essentially when in Failover pair the ASAs should to my understanding have a MAC address that is alway son the Active unit and a MAC address that is always on the Standby unit. Same naturally goes for the IP address therefore even though the switch of the active unit happens the ARP should stay the same.
I am not aware of any other IP/MAC like there might be with Cisco Routers HSRP implementations.
To my understanding the Management interface has had some changes when comparing the original ASA5500 series and the new ASA5500-X series.
One big change is ofcourse the fact that the interface has been disabled as a Data interface and can only be used for management purposes. And there have been mentions on Cisco documents that it couldnt be used in any high availability setups.
Here is a quote from a Cisco document explaining migration from ASA5500 series to ASA5500-X series
Management Port Configuration Changes
The ASA 5500-X Series introduced a shared management port for firewall and IPS services.,There are certain caveats to follow during migration from the ASA 5500 Series.• The shared management port cannot be used as a data port. All through-the-box traffic arriving at the management port will be dropped implicitly. This cannot be disabled.
• The shared management port cannot be used as a part of a high availability configuration.
If the ASA management port (M0/0) on the ASA 5500 Series appliance was being used as a data port, the configuration associated with that port should be moved to one of the gigabit data ports numbered above G0/3.
Source:
http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5708/ps5709/ps6120/guide_c07-727453.html
The ASA 8.6 configuration guide also states the following
Management 0/0 Interface on the ASA 5512-X through ASA 5555-X
The Management 0/0 interface on the ASA 5512-X through ASA 5555-X has the following characteristics:
•No through traffic support
•No subinterface support
•No priority queue support
•No multicast MAC support
•The IPS SSP software module shares the Management 0/0 interface. Separate MAC addresses and IP addresses are supported for the ASA and IPS module. You must perform configuration of the IPS IP address within the IPS operating system. However, physical characteristics (such as enabling the interface) are configured on the ASA.
What I am wondering here is if this could have anything to do with the fact that the ASA5500-X series ASA CX or IPS management is also handled through this management port. I am not sure if there is even a chance that the management IP address of the CX or IPS could be configured to overlap with the actual IP address of the Management interface.
I was personally quite confused at first when I got my ASA5515-X with ASA CX. The Management interface had its IP address and also ASA CX had a management IP address on the same interface but this didnt show on the interface configuration. It could be reset on the ASDM side through the Startup Wizard atleast. You could skip all the other phases and just apply the CX settings if you wanted to change that management IP address.
You have not mention if you have either IPS or CX. To be honest I am not 100% sure but I think if the ASA comes with the SSD HD then atleast in CX case there should be an evaluation license so you could actually use it and therefore there would be a management IP address also.
One thing you could naturally do on the ASAs is to "debug arp" and monitor that traffic and compare it to what you are seeing on the switch and the hosts.
I dont think you should be seeing 2 different MAC addresses on a single switch port unless its maybe somehow related to the CX/IPS setup
On my ASA5515-X for example I could use these commands to view the IP addresses configured for CX or IPS management (I have CX)
show module ips details
show module cxsc details
For example in my case
some output omitted
ASA-CX# show module cxsc details
Getting details from the Service Module, please wait...
Card Type: ASA CX5515 Security Appliance
Model: ASA CX5515
Software version: 9.1.1
MAC Address Range: f872.ea24.ed03 to f872.ea24.ed03
App. name: ASA CX
App. Status: Up
App. Status Desc: Normal Operation
App. version: 9.1.1
Data Plane Status: Up
Status: Up
Mgmt IP addr: 10.0.250.251
Mgmt Network mask: 255.255.255.0
Mgmt Gateway: 10.0.250.2
Mgmt web ports: 443
Mgmt TLS enabled: true
While my Management interface configuration is
interface Management0/0
management-only
nameif MGMT
security-level 100
ip address 10.0.250.250 255.255.255.0
As for your question about the Management interface and its IP addresses. To my understanding you are unable to separate any interface from the actual Failover process. You can only specify that some interfaces are not monited with regards to Failover but their configuration will still change depending on the Failover state.
So I would presume that even though you manually set the interface IP address separately on the units that some Failover event might trigger that the other units interface configuration would be overwritten by the other unit.
I would imagine that the Management interface should work in a Failover setup because they dont give an option to use the Management interface in the way you are attempting. Though again the documents are pretty vague with regards to this information.
- Jouni
07-12-2013 02:15 PM
I have some new information. I started to look through the arp entries on my PC and the 3750 and I'm not at all certain how to interpret what's going on other than to say, the switch is simply giving the wrong MAC address.
C:>arp -d *
C:>ping 172.16.1.2 -t
Pinging 172.16.1.2 with 32 bytes of data:
Reply from 172.16.1.2: bytes=32 time<1ms TTL=255
Request timed out.
Ping statistics for 172.16.1.2:
Packets: Sent = 2, Received = 1, Lost = 1 (50% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C
M:\>arp -a
Interface: 172.16.1.10 --- 0x11
Internet Address Physical Address Type
172.16.1.2 6c-20-56-bd-f8-33 dynamic (This also shows up as 6c-20-56-bd-f6-23 occasionally)
224.0.0.22 01-00-5e-00-00-16 static
DistSW01# show ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.16.1.10 0 0010.1884.69d2 ARPA Vlan100
Internet 172.16.1.1 - e4d3.f155.6042 ARPA Vlan100
Internet 172.16.1.3 46 6c20.56bd.f625 ARPA Vlan100
Internet 172.16.1.2 0 6c20.56bd.f835 ARPA Vlan100
I would expect this kind of behavior from a high availability routing protocol, but I honestly know little to nothing about how the ASAs hand out MAC information in a high availability cluster. Upon further review of the mac address table on the switch I see the following:
DistSW01# show mac address-table
.
.
.
100 6c20.56bd.f623 DYNAMIC Gi1/0/18 (No clue what this is because the real MAC of ASA2 is 6c20.56bd.f625)
100 6c20.56bd.f833 DYNAMIC Gi2/0/18 (virtual MAC?)
100 6c20.56bd.f835 DYNAMIC Gi2/0/18 (real MAC of ASA1)
There are two MAC addresses tied to one port, which suggests that the high availability protocol does indeed use a virtual MAC address. I'm not sure what the formula is for that, but I'm sure I'll figure that out later. What bugs me is, why does the switch's arp table not show the same MAC address that my work station's arp table shows? The fact that the switch has the real MAC address of the ASA explains why it is able to ping it and why my workstation (which does not have the same MAC in its arp table) cannot.
I've taken this investigation probably as far as I (that is my level of knowledge) can go at this point. Can anyone fill in the blanks as to why, if this ASA is supposed to answer for that virtual MAC, does it not answer? I've read in several locations that the management interface cannot be included in the failover process. I'm not entirely sure if that relates since that same documentation outlines how to add an IP to the management interface on the standby unit (ip address
Any help is much appreciated.
John H
Message was edited by: John Holmes. Edited for accuracy of MAC address information.
07-12-2013 04:14 PM
So at this point, I'm pretty sure that the whole standby management interface is just a wrong idea on my part. So I would like to assign unique IP addresses to each management interface on the ASAs. However when I attempt to assign a unique IP to the standby unit I get the following:
ASA01# conf t
**** WARNING ****
Configuration Replication is NOT performed from Standby unit to Active unit.
Configurations are no longer synchronized.
ASA01(config)# int management 0/0
**** WARNING ****
Configuration Replication is NOT performed from Standby unit to Active unit.
Configurations are no longer synchronized.
And even if I do assign an IP address to this standby unit, the next time one of the devices reboots, it's going to delete its configuration (with exception of the failover configuration) and download the config from the active node. This means it's going to get the management IP of the active node. Is there a way to exclude the management interface config from the deletion? I thought this would have been a default behavior given that management interface can't be included in HA.
John H
07-12-2013 05:05 PM
Hi,
The information you have given does seem to point to a wierd situation.
Essentially when in Failover pair the ASAs should to my understanding have a MAC address that is alway son the Active unit and a MAC address that is always on the Standby unit. Same naturally goes for the IP address therefore even though the switch of the active unit happens the ARP should stay the same.
I am not aware of any other IP/MAC like there might be with Cisco Routers HSRP implementations.
To my understanding the Management interface has had some changes when comparing the original ASA5500 series and the new ASA5500-X series.
One big change is ofcourse the fact that the interface has been disabled as a Data interface and can only be used for management purposes. And there have been mentions on Cisco documents that it couldnt be used in any high availability setups.
Here is a quote from a Cisco document explaining migration from ASA5500 series to ASA5500-X series
Management Port Configuration Changes
The ASA 5500-X Series introduced a shared management port for firewall and IPS services.,There are certain caveats to follow during migration from the ASA 5500 Series.• The shared management port cannot be used as a data port. All through-the-box traffic arriving at the management port will be dropped implicitly. This cannot be disabled.
• The shared management port cannot be used as a part of a high availability configuration.
If the ASA management port (M0/0) on the ASA 5500 Series appliance was being used as a data port, the configuration associated with that port should be moved to one of the gigabit data ports numbered above G0/3.
Source:
http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5708/ps5709/ps6120/guide_c07-727453.html
The ASA 8.6 configuration guide also states the following
Management 0/0 Interface on the ASA 5512-X through ASA 5555-X
The Management 0/0 interface on the ASA 5512-X through ASA 5555-X has the following characteristics:
•No through traffic support
•No subinterface support
•No priority queue support
•No multicast MAC support
•The IPS SSP software module shares the Management 0/0 interface. Separate MAC addresses and IP addresses are supported for the ASA and IPS module. You must perform configuration of the IPS IP address within the IPS operating system. However, physical characteristics (such as enabling the interface) are configured on the ASA.
What I am wondering here is if this could have anything to do with the fact that the ASA5500-X series ASA CX or IPS management is also handled through this management port. I am not sure if there is even a chance that the management IP address of the CX or IPS could be configured to overlap with the actual IP address of the Management interface.
I was personally quite confused at first when I got my ASA5515-X with ASA CX. The Management interface had its IP address and also ASA CX had a management IP address on the same interface but this didnt show on the interface configuration. It could be reset on the ASDM side through the Startup Wizard atleast. You could skip all the other phases and just apply the CX settings if you wanted to change that management IP address.
You have not mention if you have either IPS or CX. To be honest I am not 100% sure but I think if the ASA comes with the SSD HD then atleast in CX case there should be an evaluation license so you could actually use it and therefore there would be a management IP address also.
One thing you could naturally do on the ASAs is to "debug arp" and monitor that traffic and compare it to what you are seeing on the switch and the hosts.
I dont think you should be seeing 2 different MAC addresses on a single switch port unless its maybe somehow related to the CX/IPS setup
On my ASA5515-X for example I could use these commands to view the IP addresses configured for CX or IPS management (I have CX)
show module ips details
show module cxsc details
For example in my case
some output omitted
ASA-CX# show module cxsc details
Getting details from the Service Module, please wait...
Card Type: ASA CX5515 Security Appliance
Model: ASA CX5515
Software version: 9.1.1
MAC Address Range: f872.ea24.ed03 to f872.ea24.ed03
App. name: ASA CX
App. Status: Up
App. Status Desc: Normal Operation
App. version: 9.1.1
Data Plane Status: Up
Status: Up
Mgmt IP addr: 10.0.250.251
Mgmt Network mask: 255.255.255.0
Mgmt Gateway: 10.0.250.2
Mgmt web ports: 443
Mgmt TLS enabled: true
While my Management interface configuration is
interface Management0/0
management-only
nameif MGMT
security-level 100
ip address 10.0.250.250 255.255.255.0
As for your question about the Management interface and its IP addresses. To my understanding you are unable to separate any interface from the actual Failover process. You can only specify that some interfaces are not monited with regards to Failover but their configuration will still change depending on the Failover state.
So I would presume that even though you manually set the interface IP address separately on the units that some Failover event might trigger that the other units interface configuration would be overwritten by the other unit.
I would imagine that the Management interface should work in a Failover setup because they dont give an option to use the Management interface in the way you are attempting. Though again the documents are pretty vague with regards to this information.
- Jouni
07-13-2013 01:46 PM
Jouni,
This is fantastic information. We're upgrading a pair of ASA 5510s that I never really touched the IPS on. So it didn't occur to me to check the IPS. To be perfectly honest, I a little new to IPS so it didn't occur to me to check this. I had read all of the resources you were talking about, and I must have seen that statement about the IPS and the ASA sharing the same management port at least 5 times. It never occurred to me to check the IPS (and tbh, I didn't realize we had one). Thanks a ton for your information. I won't be back in the office until Monday to check the solution on the active unit (console is currently plugged into the standby unit), but as soon as I can confirm that changing the IPS's IP address will resolve this issue, I'll mark your answer as correct.
THANK YOU so much for your insight.
Cheers,
John H
07-15-2013 09:45 AM
Jouni,
It was indeed the IPS. Thank's a million for your insight! Now I guess the only thing left to determine is, why would Cisco assign the same IP address 2 to different devices with distict MAC addresses using the same port. Perhaps there's a valid reason, but for the time being I will just be content to know that I can now use the management interface. I can't say for certain that it's going to work, and I'll monitor it in the coming weeks, but I believe this resolves the failover functionality of the mgmt interfaces as well. I'm now able to use the command ip addr (ip) (mask) standby (standby ip). I will perform some reboots and make sure the configurations stick across both devices and then test failover access.
Thanks again for your excellent explanation.
Cheers,
John H
07-15-2013 09:57 AM
Hi,
Glad to hear you found the problem.
To be honest I was almost confident that it could NOT be the IPS management IP address but I guess it was.
I cant remember what the management interface and the ASA-CX management IP address was when I first unboxed my ASA unit. I guess I could always return it to factory default and check at some point but it does seem somehow unbelievable that the IP address would be set to overlap.
Hopefully there are no more surprises for your failover setup
Thank you for marking the correct reply
- Jouni
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide