2110 FTD HA setup w/ Virtual Mac Addresses

[mumbles202]

I setup a pair of 2110s (6.5.0.4) in HA that are managed by an FMC w/ a port-channel facing the LAN and a single outside interface for now.  I set the port-channel and the outside interface to use a virtual mac address for the active and standby units.  I don't have any standby ip addresses defined for either.  I've noticed that while the interfaces are up/up on the core, I'm not seeing the standby mac address on the switch it's connected to.  if I failover the active mac is populated where I'd expect it to be, but curious as to why the standby isn't there.  On a similar setup w/ a pair of ASAs i do see it so I'd expect the same. 

[Sheraz.Salim]

For Active/Standby High Availability, see the following for IP address and MAC address usage during a failover event:

1. The active unit always uses the primary unit's IP addresses and MAC addresses.

2. When the active unit fails over, the standby unit assumes the IP addresses and MAC addresses of the failed unit and begins passing traffic.

3. When the failed unit comes back online, it is now in a standby state and takes over the standby IP addresses and MAC addresses.

However, if the secondary unit boots without detecting the primary unit, then the secondary unit becomes the active unit and uses its own MAC addresses, because it does not know the primary unit MAC addresses. When the primary unit becomes available, the secondary (active) unit changes the MAC addresses to those of the primary unit, which can cause an interruption in your network traffic. Similarly, if you swap out the primary unit with new hardware, a new MAC address is used.

Virtual MAC addresses guard against this disruption, because the active MAC addresses are known to the secondary unit at startup, and remain the same in the case of new primary unit hardware. If you do not configure virtual MAC addresses, you might need to clear the ARP tables on connected routers to restore traffic flow. The Firepower Threat Defense device does not send gratuitous ARPs for static NAT addresses when the MAC address changes, so connected routers do not learn of the MAC address change for these addresses.

Virtual MAC Addresses

The Firepower Threat Defense device has multiple methods to configure virtual MAC addresses. The recommended using only one method. If you set the MAC address using multiple methods, the MAC address used depends on many variables, and might not be predictable.

For multi-instance capability, the FXOS chassis autogenerates only primary MAC addresses for all interfaces. You can overwrite the generated MAC address with a virtual MAC address with both the primary and secondary MAC addresses, but predefining the secondary MAC address is not essential; setting the secondary MAC address does ensure that to-the-box management traffic is not interrupted in the case of new secondary unit hardware.

[Anchi.Zhang]

@Sheraz.Salim I have been struggling with issues related to this for a year--one time in trying to replace a failed HA unit, I disabled our Internet browsing for 43 minutes before "clear arp" on the edge routers--with multiple TAC cases to no avail until I read your post today. I thank you greatly and so should our Cisco account team and various TAC engineers.

[robert.humphreys1]

Thank you for this explanation as has helped identify the root cause of an issue that occurred with our set-up and with possible future mitigations when we had to rebuild a HA active/passive multi-instance pair running on 4110 chassis.

This was due to an upgrade that worked to FTD 7.1 from 6.4 but then stopped us from deploying policies to the HA pair when faulty node was active. 

We needed to break HA to rebuild the primary node, the secondary was active with the Primary's MAC for the interfaces, as per design.

When we broke HA the secondary remained active and traffic flow was observed but the Primary MAC addresses we replaced by the secondary's. All static NATs stopped responding to external requests but weren't aware immediately. 

We needed to clear the ARP table on the ISP routers to allow ARP table to populate with the updated MAC addresses for the IP addresses though until reading this were not sure why.

Very much appreciated. Thanks.

[Francesco Molino]

Hi,

You should see the mac address on your switch. Instead of looking at the mac address on a dedicated interface on the lan side, can you take the mac address of your FTD interface and do a sh mac add add xxxx.xxxx.xxxx and see if you see it, just to make sure

[mumbles202]

Thanks for the reply. So as noted, the virtual mac does follow the active unit. VMacs are aaaa.aaaa.1111 and aaaa.bbbb.2222. If I do a sh Mac add | in aaaa only the Mac address of the active unit is shown. This happens for both the outside interface as well as the plan port channel. Outside goes to a Aruba switch and inside to a 3850 stack. 

[Francesco Molino]

Are you able to ping the secondary ip ?

[mumbles202]

I don't have secondary ips defined for the interfaces.  While I expect there won't be an arp entry for the standby firewall since it has no address, i would expect the virtual mac address to appear in the switches mac table.  If i need to add a standby ip to accomplish this I can but wanted to confirm that's necessary.

[Sheraz.Salim]

as I have mentioned earlier The active unit always uses the primary unit's IP addresses and MAC addresses.

[mumbles202]

Correct.  I understand this.  What I'm saying is Eth1/3 is configured as 192.168.100.1/28 w/ no secondary ip address defined.  The active unit should have a virtual mac of aaaa.aaaa.1111 and the standby unit should have a virtual mac of aaaa.aaaa.2222.  The primary unit Eth1/3 is connected to Switch A port 3 and the standby unit is connected to Switch A port 4.  if the primary unit is active, I see the aaaa.aaaa.1111 mac address on Switch A port 3 as expected.  But if I do a "sh mac add | in aaaa" only port 3 is returned.  And if I do a show mac address int gi 1/0/4 nothing is returned.  If i flip the active/standby roles I get the reverse results on the switch ports.  The aaaa mac is returned only on Gi 1/0/4 and nothing is returned for Gi 1/0/3.  While I expect to only have 1 arp entry should the mac address still remain in the switch tables?

[Francesco Molino]

This is not mandatory.

Here my asa config:

failover
failover lan unit primary
failover lan interface fover GigabitEthernet0/4
failover mac address GigabitEthernet0/0 aaaa.bbbb.cc01 aaaa.bbbb.cc02
failover link fover GigabitEthernet0/4
failover interface ip fover 169.200.200.1 255.255.255.252 standby 169.200.200.2

show ip on ASA:

ASA1# sh ip 
System IP Addresses:
Interface                Name                   IP address      Subnet mask     Method 
GigabitEthernet0/0       inside                 192.168.10.1    255.255.255.0   CONFIG

and finally, mac address info from active and standby: (validation my mac addresses are set):

ASA1# sh int g0/0
Interface GigabitEthernet0/0 "inside", is up, line protocol is up
  Hardware is i82540EM rev03, BW 1000 Mbps, DLY 10 usec
	Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
	Input flow control is unsupported, output flow control is off
	MAC address aaaa.bbbb.cc01, MTU 1500
	IP address 192.168.10.1, subnet mask 255.255.255.0
	5 packets input, 320 bytes, 0 no buffer
	Received 4 broadcasts, 0 runts, 0 giants
	0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
	0 pause input, 0 resume input
	0 L2 decode drops
	9 packets output, 540 bytes, 0 underruns
	0 pause output, 0 resume output
	0 output errors, 0 collisions, 1 interface resets
	0 late collisions, 0 deferred
	0 input reset drops, 0 output reset drops
	input queue (blocks free curr/low): hardware (507/507)
	output queue (blocks free curr/low): hardware (511/509)
  Traffic Statistics for "inside":
	4 packets input, 184 bytes
	6 packets output, 168 bytes
	0 packets dropped
      1 minute input rate 0 pkts/sec,  0 bytes/sec
      1 minute output rate 0 pkts/sec,  0 bytes/sec
ASA1#         
ASA1# failo exec ma sh int g0/0 
Interface GigabitEthernet0/0 "inside", is up, line protocol is up
  Hardware is i82540EM rev03, BW 1000 Mbps, DLY 10 usec
	Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)
	Input flow control is unsupported, output flow control is off
	MAC address aaaa.bbbb.cc02, MTU 1500
	IP address unassigned
	6 packets input, 384 bytes, 0 no buffer
	Received 5 broadcasts, 0 runts, 0 giants
	0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

Now on my switch, I do see these 2 mac addresses:

CORE#sh mac address-table vlan 2
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
   2    aaaa.bbbb.cc01    DYNAMIC     Et0/0
   2    aaaa.bbbb.cc02    DYNAMIC     Et0/1
Total Mac Addresses for this criterion: 2

The topology from the virtual lab is:

[mumbles202]

Exactly my point.  I'm only showing the 1 mac of the active unit at any given time.  

[Francesco Molino]

The mac you're not seeing is it connected to the aruba switch? Can you try connecting it to the Cisco switch?

[Sheraz.Salim]

@Francesco Molino From reading above notes I think both FTD are connected to same switch.

@mumbles202 could you share the show tech file of Aruba switch and FTD with us?

[mumbles202]

I'll get the show tech for the switches but I've seen the same behavior with the Aruba switch that sites on the outside as well as the Cisco switch that sits on the inside.