FTD HA Pair Using Several Thousand MAC Addresses

[Mike Wagner]

Hi Everyone,

Posting here in a last ditch effort.  Any help is greatly appreciated.  Cisco TAC, despite the hefty fee we pay, is very unresponsive.

We have two Firepower 4110 units in an HA Active/Standby cluster.  The "outside" interfaces of these two FTD boxes are connected to a 2960-X switch stack on an L2 VLAN, as are our two edge ASR 1002-X router "inside" interfaces.  This is due to not having enough 10Gb ports on the ASR's to connect both FTD's to each ASR.  The ASR's are in HSRP mode and iBGP is configured for our backup connection to kick in whenever the main goes down.  The scenario is fully redundant, and has worked for over 6 months.  

Recently the outside interface of the FTD HA pair has become unresponsive randomly.  The only way I could make the FTD pair outside interface to become responsive is by rebooting the 2960-X switch stack, or flipping the active/standby units on the HA pair.  Since it's Saturday and there aren't many people in, I was able to delve into this much further since it happened again this morning...

What I found is that the 2960-X stack is showing over 8k MAC addresses on the port that the active FTD unit is plugged into.  It should only be one (that of the active outside interface on the HA pair), as it's an L3 interface on the FTD side.  

Like I said, this issue comes and goes, so I rebooted the switch stack, and we're back down to the normal amount of MAC addresses.

This is a wild theory, but I never did setup active/standby MACs on the HA Pair when I created it.  Everything I read indicates this should not be a problem.  However, I decided to go ahead and set them up now as a last ditch effort.  As soon as I created them and hit deploy, the deployment hung up at 20% for a very long time.  I then used OmniQuery to remove the deploy.  I rebooted the FTD's and FMC.  Unfortunately, even though OmniQuery shows no more status 7 tasks, the task is still showing in FMC as In Progress... (%)  - No Number for the percent!

Any ideas are greatly appreciated!  I'm at my wits end.!

Thanks!

[MHM Cisco World]

Can you draw topology 
ASR is connect to FTD I confuse on this point.

[Mike Wagner]

Sorry for the quick and dirty redacting and screen capture of the attached network drawing

There is two of everything, and each ASR and each FTD are connected to 10Gb ports in a 2960 stack (cheapest way I could get a switch with 4 10gb ports at the time)... to create a big L2 domain between edge ASRs and edge FTDs.

[MHM Cisco World]

one more Q. are you config BD in FTD ?

[Mike Wagner]

Sorry for my ignorance.  What do you mean by BD?

[MHM Cisco World]

bridge domain

[Mike Wagner]

Interesting... That definitely would explain some things.

I checked in FMC, and did not find any bridge groups.  I did, however, check the the FXOS configurations themselves, and found an administratively down cluster etherchannel interface.  It's down, so unless there's some strange bug it shouldn't be causing the issue.  However, I've deleted it just in case.  Maybe that is the culprit?

[MHM Cisco World]

one more Q are ASR is run BGP with ISP?

[Mike Wagner]

The ASR’s are indeed BGP peering with the ISP equipment.  

[Sheraz.Salim]

How many instane are running on FTD 4100? what is the version you on FXOS and what version is the FTD on the chassis and what version you running on FMC?

What I found is that the 2960-X stack is showing over 8k MAC addresses on the port that the active FTD unit is plugged into. It should only be one (that of the active outside interface on the HA pair), as it's an L3 interface on the FTD side.

could you issue command show logging on the swtich and see what logs it display. Is your L2 is consistant? I mean on STP issues anywhere? you can check "show spanning-tree detail | in ieee|from|occur|is exec". 

once you failed over the HA pair everything start working fine. it could be a issue some where in Layer2 (My guess). whats change happen in this network since last six months?

[Mike Wagner]

Nothing has changed on the L2 side, and nothing in logging

only thing is, about a month ago we had to cut power to the datacenter.  Maybe something did not come back up right.  

not sure on FXOS version, will let you know.  FTD version 7.0.1-82.  Previously on 6.6 and it was doing it then.  

one instance on each 4110

[MHM Cisco World]

@Sheraz.Salim and Me suspect that this issue relate some how to L2,
but 8K MAC<<<< this to huge mac address, 
you mention that there is DC, and I see iBGP
SO here the Q,
are you config any L2VPN (L2 over MPLS)??
I think that L2VPN and there is iBGP interconnect both Edge router ASR through SW is cause this huge Number of Mac address.

check the BGP L2VPN

[Mike Wagner]

Hello,

The iBGP link is done through dedicated east-west interfaces on the ASRs, so that traffic never makes it to the L2 2960 stack.  What’s odd is, all of the “extra” MAC addresses are showing up on the interface facing the active Firepower, not the ASRs.  Also, this doesn’t happen all of the time.  For instance, it’s been 24 hours since I deleted the “administratively down” etherchannel in FXOS, and we have not had any issues.  But I could happen again tomorrow.  Sometimes it will be hours or a day before it happens again. My fingers are crossed that it’s some strange bug in FXOS and deleting that etherchannel fixed it.  

[MHM Cisco World]

but you don't answer me are you use any L2VPN ?
if yes then stare the output of show bgp, 

8K MAC address for your network is too high number, but if there is L2VPN then this DC have MAC address for all user in all site, 
keep in mind that the BGP can carry MAC address and use it to exchange L2 traffic between the DC and other site. 
for some reason the FTD is in way and show this huge number of MAC address.

[Mike Wagner]

I understand what you’re saying, however there is no L2VPN in use at all.  All of this, the dual devices and everything, is contained within one single datacenter.  Our outside BGP peers are with our carrier on a government network.