Hello Community,
We have an HA pair of Cisco FMC 4500s (6.6.1) managing a pair of Firepower 9300 ACs (2.8.1.125) each housing one FP 9000 SM-24 FTD (6.6.1) in routed cluster mode. I know this mode is not optimal, but we are working with what we have. I am also relatively new to this environment (and NGFWs) and any historical institutional knowledge on this deployment is gone. At some point in the past, the FTDs were migrated from ASAs, I believe.
We often find that the post-deployment transcripts contain lines in the CLI apply for which we cannot identify a source. There are a few different CLI blocks that show up periodically but not necessarily predictably (that we have observed). Cisco TAC has previously been insistent that one of our techs is applying the configurations before the deploys, but these do not show up in GUI audit logs. All usernames in syslog are the built-in system names "Config" and "Enable_1" - syslog is attached.
I have checked for FlexConfigs, scheduled jobs, and any event response actions that I could find that may be modifying configuration, but haven't identified anything. I am sure that my inexperience would be a blocker for thorough discovery, though, so there could certainly be something that I missed.
I am wondering if there are any config changes that may trigger an additional deployment of config that is not obvious to the user, or if there may be events that cause these, some scheduled jobs that add these modifications when they run, possible back-end scripts running that may or may not be remnants of the ASA migration, or an API config push that could be causing these.
Any insight would be extremely welcome. Thank you very much.
Here are some of the blocks of commands that are curious or problematic, and the conditions observed when they were generated. The final one is the most critical as it was a global outage:
Transcript after a scheduled "Deploy Policies" job, notice SNMP user and OSPF config application:
=========SNORT APPLY=========
========= CLI APPLY =========
FMC >> clear configuration session OBJECT
XXXXX >> [info] : Session OBJECT does not exist.
FMC >> clear configuration session FMC_SESSION_1
XXXXX >> [info] : Session FMC_SESSION_1 does not exist.
FMC >> clear configuration session FMC_SESSION_2
XXXXX >> [info] : Session FMC_SESSION_2 does not exist.
FMC >> no strong-encryption-disable
FMC >> snmp-server user XXXXX Priv v3 auth sha ****** priv aes 128 ******
FMC >> router ospf 1
FMC >> area 0 range XXXXX 255.255.255.0 advertise
FMC >> crypto isakmp nat-traversal
========= INFRASTRUCTURE MESSAGES =========
Lina Config application was successful
Lina write mem operation successful
================================================================
Transaction ID: 408021967142
Device UUID: XXXXX
Selected policy group list: PG.FIREWALL.NGFWAccessControlPolicy, IntrusionPolicy, IntrusionPolicy, IntrusionPolicy, IntrusionPolicy
Out-of-date policy group list: PG.FIREWALL.NGFWAccessControlPolicy, IntrusionPolicy, IntrusionPolicy, IntrusionPolicy, IntrusionPolicy
Deployment Type: Full Deployment
================================================================
Transcript after deploying an ACP rule change to add subnets to a block rule, same SNMP and OSPF commands (never observed the other config session statement, but this happens without it):
=========SNORT APPLY=========
========= CLI APPLY =========
FMC >> clear configuration session OBJECT
FMC >> clear configuration session FMC_SESSION_1
XXXXX >> [info] : Session FMC_SESSION_1 does not exist.
FMC >> clear configuration session FMC_SESSION_2
XXXXX >> [info] : Session FMC_SESSION_2 does not exist.
FMC >> no strong-encryption-disable
FMC >> configure session OBJECT
...
FMC >> commit noconfirm revert-save
XXXXX >> [info] : WARNING: Other committed configuration sessions currently exist (use 'show configuration session' command to view them). Overlapping changes between these sessions might lead to inconsistent/unexpected changes to the running configuration when reverted in the wrong order.
FMC >> snmp-server user XXXXX Priv v3 auth sha ****** priv aes 128 ******
FMC >> router ospf 1
FMC >> area 0 range XXXXX 255.255.255.0 advertise
FMC >> crypto isakmp nat-traversal
FMC >> clear configuration session FMC_SESSION_1
FMC >> clear configuration session FMC_SESSION_2
XXXXX >> [info] : Session FMC_SESSION_2 does not exist.
FMC >> clear configuration session OBJECT
========= INFRASTRUCTURE MESSAGES =========
Lina Config application was successful
Lina write mem operation successful
================================================================
Transaction ID: 408021966169
Device UUID: XXXXX
Selected policy group list: PG.FIREWALL.NGFWAccessControlPolicy
Out-of-date policy group list: PG.FIREWALL.NGFWAccessControlPolicy
Deployment Type: Full Deployment
================================================================
And the most significant one that has caused several outages, with TAC noting the only way to source them is with audit logs that don't show it, this removal of ospf authentication. I do not know what change was associated with the first occurrence of this, but the second occurrence was associated with a NAT policy update to add a PAT pool for a subnet. This one is really odd. When the NAT change was rolled back, the OSPF authentication was ADDED BACK - with the correct key!! The user did nothing other than remove the NAT config and deploy. The first deploy issued the "no" command, the very next deploy added the config back. There must be some appending here, but I cannot find it in the config or in any documentation. I do not have a full transcript of either the removal or addition, but I do have a partial snippet of the removal:
FMC >> no strong-encryption-disable
FMC >> interface Port-channel1.3
FMC >> no ospf authentication message-digest
FMC >> no ospf message-digest-key 1 md5 ******
This config is on our outside interface, outside zone, and please note it is a subinterface. I think I have read that may be against best practices. It is not the same OSPF area as the previous OSPF commands.
Any help or guidance towards finding the source of the SNMP commands, the OSPF config, and the removal of the OSPF authentication would be really appreciated.
Thank you very much.
