FMC failure - FTD change scenario.

[darrenj]

Simple scenario, one FMC managing two pairs of firewalls in HA. To repeat there is one FMC and no plan to deploy another for redundancy.

If FMC was to completely fail, I'm guessing the firewalls keep working with the last pushed FMC config. If I need to make a change to the firewalls, is there a way to apply a firewall rule directly on the firewall via CLI or GUI (without FMC involvement)?

I'm more used to working on "another firewall vendor" and if the management server dies with the "other firewall vendor" you can still apply configurations locally on the firewall (a great fail-safe option). Not sure if Cisco works like this though (I hope it does!).

Thanks

Darren

[Rob Ingram]

@darrenj probably not as much as you would hope. From the latest version 7.7 if you lose the management connection to your device, you can make select configuration changes directly at the device CLI to:

• Restore the management connection if you are using a data interface for manager access.

• Make select policy changes that can't wait until the connection is restored.

After the management connection is restored, the management center will detect the configuration changes on the device.

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/release-notes/threat-defense/770/threat-defense-release-notes-77.html

You can configure the following feature areas at the diagnostic CLI in recovery-config mode: Interfaces, Static Routes, Dynamic Routing: BGP and OSPF, Prefilters and Site-to-site VPN.

https://www.cisco.com/c/en/us/td/docs/security/firepower/command_ref/b_Command_Reference_for_Firepower_Threat_Defense/c_3.html#wp4205799262

[balaji.bandi]

If FMC were to completely fail, I'm guessing the firewalls would keep working with the last FMC config that was pushed. 

yes FTD works as expected - no service outage here. you loose management capabilities.

is there a way to apply a firewall rule directly on the firewall via CLI or GUI (without FMC involvement)?

I will not take the chance of that. Also, it's very difficult—chances may go wrong, and it is  a complete disaster.

Fail-safe when the firewalls completely fail—maybe a wish list for cisco. (Sure, different vendors offer different flavours, so when we buy a firewall, we need to consider all the facts and choose the right product that the business needs.)

FTD  you can use API and apply if you are familiar with that option.

Building a new FMC is not rocket science; it takes a maximum of 2-3 hours to restore the backup. Nowadays, most virtual environments are cheaper. If this is critical, I keep the FMC VM on standby for use when the Physical FMC dies.

Note: backup is key (does not matter what vendor you use - i use multi vendors (checkpoint, Palo, Fortinet along with Cisco, also Linux firewall, each one have a different flavour come with and cost)

[darrenj]

Thanks both for the responses. I will check out the feature in v7.7, sounds limited but could help. I am also thinking of testing the API to deploy a rule update, that would provide a good fallback option too.

The thing is, mistakes do happen even with strict change control - human error is a thing. If FMC was to push out a config update that blocked the firewall access to FMC (assume it is in the traffic flow) you are screwed. Its never happened to me but it has happened to a customer of mine, luckily the other firewall vendor allowed me to add a rule at the top allowing the firewall to talk to the management platform and the crisis was over. Doesn't sound like this happens with Cisco which is a big worry.......

Note: Agreed about backups, absolutely necessary.

DJ

[Rob Ingram]

@darrenj if the FTDs are managed by an FMC, I am not sure you could use API if the FMC is down.

You could also consider to migrate to the cloud delivered FMC (cdFMC), so long the FTD's have internet access they are manageable. You rely on the resilency of the cloud managed service.

[Marius Gunnerud]

If FMC was to completely fail, I'm guessing the firewalls keep working with the last pushed FMC config. If I need to make a change to the firewalls, is there a way to apply a firewall rule directly on the firewall via CLI or GUI (without FMC involvement)?

The firewalls will continue to work as normal.  With regard to configuration on the CLI, this is possible, though I would not recommend it unless it is absolutely necessary.  You can make configurations from expert mode and this is supported in all versions of FMC that I have done it on with the newest version I testing on being 7.2.x. 

This is not widely known as configuration via the FMC is the "true" way to do things.  The command I know of I have only tested with single line commands, but would be interesting to find out if it is possible to redirect or pipe a config file into the command. The following is an example for adding a static route.

When doing these types of changes you DO SO AT YOUR OWN RISK! Doing this without TAC support and something goes wrong might void TAC support for the device.

BE AWARE THAT THESE CHANGES WILL NOT BE LEARNED BY THE FMC WHEN IT IS BACK ONLINE, AND ALL CHANGES WILL NEED TO BE ADDED AGAIN INTO FMC BEFORE YOU DEPLOY FOR THE FIRST TIME OR IT WILL ALL BE LOST!!!

Enter expert mode: 

>expert 

# sudo su - 

root# cd /ngfw/var/sf/bin 

root# LinaConfigTool "route Localport-base 10.10.1.0 255.255.255.0 10.10.2.1"; 

Remember that cli command should be entered between "" and end with ; 

[darrenj]

Ooohhh thanks for this! Noted that this is done at own risk, etc but I will definitely try this in the lab (trying to add a firewall rule/policy). I'll report back how I go on.....

[darrenj]

In terms of an update, for testing I created a rule to block SSH between two hosts. I then "pretended" I lost FMC and needed to allow SSH between the two hosts. I did the below on FTD, basically allowing TCP between anyone.

admin@fmc:~$ sudo su -
Password:
root@fmc:~# cd /ngfw/var/sf/bin
root@fmc:bin# LinaConfigTool "access-list CSM_FW_ACL_ line 1 advanced permit tcp any any";
root@fmc:bin#

This entry was shown in my access-list output, so I thought it would work;

> show access-list
access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
alert-interval 300
access-list CSM_FW_ACL_; 10 elements; name hash: 0x4a69e3f3
access-list CSM_FW_ACL_ line 1 advanced permit tcp any any (hitcnt=3) 0xb2cceaf1

Although the traffic is allowed (a connection is created), the initial SYN never reaches the SSH server

One of the flags indicates that Snort is inspecting the flow, so it may be Snort dropping the traffic because the rule is not properly configured/deployed by FMC? 

> show conn detail
1 in use, 80 most used
Inspect Snort:
preserve-connection: 2 enabled, 0 in effect, 80 most enabled, 0 most in effect
Flags: A - awaiting responder ACK to SYN, a - awaiting initiator ACK to SYN,
B - TCP probe for server certificate,
b - TCP state-bypass or nailed,
C - CTIQBE media, c - cluster centralized,
D - DNS, d - dump, E - outside back connection, e - semi-distributed,
F - initiator FIN, f - responder FIN,
G - group, g - MGCP, H - H.323, h - H.225.0, I - initiator data,
i - incomplete, J - GTP, j - GTP data, K - GTP t3-response
k - Skinny media, L - decap tunnel, M - SMTP data, m - SIP media
N - inspected by Snort (1 - preserve-connection enabled, 2 - preserve-connection in effect,
3 - elephant-flow, 4 - elephant-flow bypassed, 5 - elephant-flow throttled, 6 - elephant-flow exempted )
n - GUP, O - responder data, o - offloaded,
P - inside back connection, p - passenger flow,
Q - QUIC, q - SQL*Net data, R - initiator acknowledged FIN,
R - UDP SUNRPC, r - responder acknowledged FIN,
T - SIP, t - SIP transient, U - up,
V - VPN orphan, v - M3UA W - WAAS,
w - secondary domain backup,
X - inspected by service module,
x - per session, Y - director stub flow, y - backup stub flow,
Z - Scansafe redirection, Z1 - zero-trust flow, z - forwarding stub flow

TCP outside: 192.168.6.10/22 inside: 192.168.5.10/40394,
flags aA N, idle 5s, uptime 5s, timeout 30s, bytes 0, Rx-RingNum invalid, Internal-Data invalid
Initiator: 192.168.5.10, Responder: 192.168.6.10
Connection lookup keyid: 108772834

Not sure if anyone has any other ideas...?

[Marius Gunnerud]

Well, first off are you able to ping both test devices from the firewall?  One thing I would suggest doing is setting up a packet capture on the interface that connects towards the device you are SSHing to to see if the packet is actually leaving the interface.  If you see the packet leaving the interface then there is an issue on the device that you are SSHing to.  If you do not see the packet leaving the device, then it is being dropped and I would suggest running "system support trace" and also enable firewall-engine-debug to try to see where it is being dropped.

[darrenj]

Hi there, yes before I did this test I can both ping and SSH between the two. Like I say, the rule is applied but the traffic not allowed - I assume because Snort or some other engine blocks it as it was not configured/deployed from FMC.....

[Marius Gunnerud]

So did you run a 'system support trace' and run a test with the config from CLI implemented? I would also suggest having a packet capture on both interfaces on the firewall where the traffic will be flowing.  if you did not have these running when you last tested, then I suggest you do it.  this will indicate if it is SNORT that is blocking.

Also, I was wondering if you were able to ping the two devices from the firewall when the configuration was implemented.

Is this a clean installed FTD with no previous access list implemented?  If yes, issue the command show run | in access-group is there an entry there that looks something like "access-group CSM_FW_ACL_ global" ? if not, then you will need to add this line to the configuration.

[darrenj]

Hi Marius, I'm genuine grateful for you input but have the read the initial post I made? This is not a "typical scenario" where I'm pushing config from FMC to FTD and traffic flows are not working (that is straightforward). I'm trying to see what options I have should FMC become unreachable to the FTD and I need to make an "emergency rule" change to FTD directly on the box itself (for example, within the Lina process). Example: Someone incorrectly creates a rule that blocks FTD from talking to FMC (mistakes do happen and I know change control should pick this up blah blah). I now want to add a rule directly to FTD to allow it to talk to FMC - just a temporary rule and once connected to FMC it will then take control again of all the config.....

Consider what I am trying to work out is a "break glass" scenario......

DJ

[Marius Gunnerud]

I have read your initial post and based off of your test results there is obviously something missing from the configuration.  I was in the exact same situation you which you are trying to reproduce, where there was a routing issue which caused connection between the FMC and FTD to be dropped.  That LinaConfigTool command I provided earlier is what saved me by placing a static route in the routing table.  but aside from the static route all other configuration was already in place.  So to figure out why your configuration is not working I need more context to how you have setup your lab.

Then please define your initial setup that you are testing with.  have you deployed an FMC and FDT in a lab provided initial configuration and then break that configuration or remove FMC from the picture?  Or have you just spun up an FTD and configure it from scratch from the CLI?  If you have not added an access policy before testing this you are probably missing the "access-group CSM_FW_ACL_ global" command, which is why I asked about that.

If that command is present then we need to figure out what is happening, which is where system support trace and packet-capture comes into play. 

You could also do a packet-tracer to see the path the packet takes inside the firewall.

[darrenj]

Hi there, so yes the traffic was working in a lab. I had an FMC pushed rule that allowed two devices to communicate via ping and SSH. All good. I then changed the rule to only allow ping. Now, the two devices can ping but SSH fails because it hits the default deny rule.

Then, I added a line via LinaConfig to allow the SSH traffic, this line showed in the "show access-list" command on the firewall. However, the SSH traffic does not work - I can see on the firewall an entry (via "show conn") but only the intial syn is shown and it never reaches the target SSH server. So, I have so far concluded that the traffic was allowed by the Lina process (old ASA) but is getting dropped by some other process because the rule was not pushed by FMC. Hope that makes sense.