topic Re: FTD HA Pair Using Several Thousand MAC Addresses in Network Security

FTD HA Pair Using Several Thousand MAC Addresses

Mike Wagner — Sat, 28 May 2022 19:54:26 GMT

Hi Everyone,

Posting here in a last ditch effort. Any help is greatly appreciated. Cisco TAC, despite the hefty fee we pay, is very unresponsive.

We have two Firepower 4110 units in an HA Active/Standby cluster. The "outside" interfaces of these two FTD boxes are connected to a 2960-X switch stack on an L2 VLAN, as are our two edge ASR 1002-X router "inside" interfaces. This is due to not having enough 10Gb ports on the ASR's to connect both FTD's to each ASR. The ASR's are in HSRP mode and iBGP is configured for our backup connection to kick in whenever the main goes down. The scenario is fully redundant, and has worked for over 6 months.

Recently the outside interface of the FTD HA pair has become unresponsive randomly. The only way I could make the FTD pair outside interface to become responsive is by rebooting the 2960-X switch stack, or flipping the active/standby units on the HA pair. Since it's Saturday and there aren't many people in, I was able to delve into this much further since it happened again this morning...

What I found is that the 2960-X stack is showing over 8k MAC addresses on the port that the active FTD unit is plugged into. It should only be one (that of the active outside interface on the HA pair), as it's an L3 interface on the FTD side.

Like I said, this issue comes and goes, so I rebooted the switch stack, and we're back down to the normal amount of MAC addresses.

This is a wild theory, but I never did setup active/standby MACs on the HA Pair when I created it. Everything I read indicates this should not be a problem. However, I decided to go ahead and set them up now as a last ditch effort. As soon as I created them and hit deploy, the deployment hung up at 20% for a very long time. I then used OmniQuery to remove the deploy. I rebooted the FTD's and FMC. Unfortunately, even though OmniQuery shows no more status 7 tasks, the task is still showing in FMC as In Progress... (%) - No Number for the percent!

Any ideas are greatly appreciated! I'm at my wits end.!

Thanks!

Re: FTD HA Pair Using Several Thousand MAC Addresses

MHM Cisco World — Sat, 28 May 2022 21:05:19 GMT

Can you draw topology
ASR is connect to FTD I confuse on this point.

Re: FTD HA Pair Using Several Thousand MAC Addresses

Mike Wagner — Sat, 28 May 2022 21:16:31 GMT

Sorry for the quick and dirty redacting and screen capture of the attached network drawing

There is two of everything, and each ASR and each FTD are connected to 10Gb ports in a 2960 stack (cheapest way I could get a switch with 4 10gb ports at the time)... to create a big L2 domain between edge ASRs and edge FTDs.

Re: FTD HA Pair Using Several Thousand MAC Addresses

MHM Cisco World — Sat, 28 May 2022 23:25:25 GMT

one more Q. are you config BD in FTD ?

Re: FTD HA Pair Using Several Thousand MAC Addresses

Mike Wagner — Sun, 29 May 2022 00:43:57 GMT

Sorry for my ignorance. What do you mean by BD?

Re: FTD HA Pair Using Several Thousand MAC Addresses

MHM Cisco World — Sun, 29 May 2022 00:50:11 GMT

bridge domain

Re: FTD HA Pair Using Several Thousand MAC Addresses

Mike Wagner — Sun, 29 May 2022 01:29:35 GMT

Interesting... That definitely would explain some things.

I checked in FMC, and did not find any bridge groups. I did, however, check the the FXOS configurations themselves, and found an administratively down cluster etherchannel interface. It's down, so unless there's some strange bug it shouldn't be causing the issue. However, I've deleted it just in case. Maybe that is the culprit?

Re: FTD HA Pair Using Several Thousand MAC Addresses

MHM Cisco World — Sun, 29 May 2022 02:32:18 GMT

one more Q are ASR is run BGP with ISP?

Re: FTD HA Pair Using Several Thousand MAC Addresses

Mike Wagner — Sun, 29 May 2022 03:00:04 GMT

The ASR’s are indeed BGP peering with the ISP equipment.

Re: FTD HA Pair Using Several Thousand MAC Addresses

Sheraz.Salim — Sun, 29 May 2022 08:45:03 GMT

How many instane are running on FTD 4100? what is the version you on FXOS and what version is the FTD on the chassis and what version you running on FMC?

could you issue command show logging on the swtich and see what logs it display. Is your L2 is consistant? I mean on STP issues anywhere? you can check "show spanning-tree detail | in ieee|from|occur|is exec".

once you failed over the HA pair everything start working fine. it could be a issue some where in Layer2 (My guess). whats change happen in this network since last six months?

Re: FTD HA Pair Using Several Thousand MAC Addresses

Mike Wagner — Sun, 29 May 2022 23:38:00 GMT

Nothing has changed on the L2 side, and nothing in logging 😞

only thing is, about a month ago we had to cut power to the datacenter. Maybe something did not come back up right.

not sure on FXOS version, will let you know. FTD version 7.0.1-82. Previously on 6.6 and it was doing it then.

one instance on each 4110

Re: FTD HA Pair Using Several Thousand MAC Addresses

MHM Cisco World — Sun, 29 May 2022 23:50:25 GMT

@Sheraz.Salim and Me suspect that this issue relate some how to L2,
but 8K MAC<<<< this to huge mac address,
you mention that there is DC, and I see iBGP
SO here the Q,
are you config any L2VPN (L2 over MPLS)??
I think that L2VPN and there is iBGP interconnect both Edge router ASR through SW is cause this huge Number of Mac address.

check the BGP L2VPN

Re: FTD HA Pair Using Several Thousand MAC Addresses

Mike Wagner — Mon, 30 May 2022 01:53:13 GMT

Hello,

The iBGP link is done through dedicated east-west interfaces on the ASRs, so that traffic never makes it to the L2 2960 stack. What’s odd is, all of the “extra” MAC addresses are showing up on the interface facing the active Firepower, not the ASRs. Also, this doesn’t happen all of the time. For instance, it’s been 24 hours since I deleted the “administratively down” etherchannel in FXOS, and we have not had any issues. But I could happen again tomorrow. Sometimes it will be hours or a day before it happens again. My fingers are crossed that it’s some strange bug in FXOS and deleting that etherchannel fixed it.

Re: FTD HA Pair Using Several Thousand MAC Addresses

MHM Cisco World — Mon, 30 May 2022 15:25:04 GMT

but you don't answer me are you use any L2VPN ?
if yes then stare the output of show bgp,

8K MAC address for your network is too high number, but if there is L2VPN then this DC have MAC address for all user in all site,
keep in mind that the BGP can carry MAC address and use it to exchange L2 traffic between the DC and other site.
for some reason the FTD is in way and show this huge number of MAC address.

Re: FTD HA Pair Using Several Thousand MAC Addresses

Mike Wagner — Mon, 30 May 2022 16:23:29 GMT

I understand what you’re saying, however there is no L2VPN in use at all. All of this, the dual devices and everything, is contained within one single datacenter. Our outside BGP peers are with our carrier on a government network.

Re: FTD HA Pair Using Several Thousand MAC Addresses

Sheraz.Salim — Mon, 30 May 2022 18:33:18 GMT

The best course of action would be work with Cisco TAC as you already mentioned you opened case with Cisco. If you not happy with your TAC Engineer you can always request for change of Engineer or even you could request a Senior TAC Engineer. Once you make this request Cisco TAC can not deny you to put you though to senior Netowrk Engineer.

as we do not have much information on few bits of detial. personally, I would recommand to work with TAC and get this sorted the issue.