cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1971
Views
5
Helpful
3
Replies

How to find sources of non-user configuration changes in FMC 6.6.1

StrikingViking
Level 1
Level 1

Hello Community,

 

We have an HA pair of Cisco FMC 4500s (6.6.1) managing a pair of Firepower 9300 ACs (2.8.1.125) each housing one FP 9000 SM-24 FTD (6.6.1) in routed cluster mode. I know this mode is not optimal, but we are working with what we have. I am also relatively new to this environment (and NGFWs) and any historical institutional knowledge on this deployment is gone. At some point in the past, the FTDs were migrated from ASAs, I believe.

 

We often find that the post-deployment transcripts contain lines in the CLI apply for which we cannot identify a source. There are a few different CLI blocks that show up periodically but not necessarily predictably (that we have observed). Cisco TAC has previously been insistent that one of our techs is applying the configurations before the deploys, but these do not show up in GUI audit logs. All usernames in syslog are the built-in system names "Config" and "Enable_1" - syslog is attached.

 

I have checked for FlexConfigs, scheduled jobs, and any event response actions that I could find that may be modifying configuration, but haven't identified anything. I am sure that my inexperience would be a blocker for thorough discovery, though, so there could certainly be something that I missed.

 

I am wondering if there are any config changes that may trigger an additional deployment of config that is not obvious to the user, or if there may be events that cause these, some scheduled jobs that add these modifications when they run, possible back-end scripts running that may or may not be remnants of the ASA migration, or an API config push that could be causing these. 

 

Any insight would be extremely welcome. Thank you very much.

 

Here are some of the blocks of commands that are curious or problematic, and the conditions observed when they were generated. The final one is the most critical as it was a global outage:

 


Transcript after a scheduled "Deploy Policies" job, notice SNMP user and OSPF config application:

 

=========SNORT APPLY=========

========= CLI APPLY =========

FMC >> clear configuration session OBJECT
XXXXX >> [info] : Session OBJECT does not exist.

FMC >> clear configuration session FMC_SESSION_1
XXXXX >> [info] : Session FMC_SESSION_1 does not exist.

FMC >> clear configuration session FMC_SESSION_2
XXXXX >> [info] : Session FMC_SESSION_2 does not exist.

FMC >> no strong-encryption-disable
FMC >> snmp-server user XXXXX Priv v3 auth sha ****** priv aes 128 ******
FMC >> router ospf 1
FMC >> area 0 range XXXXX 255.255.255.0 advertise
FMC >> crypto isakmp nat-traversal


========= INFRASTRUCTURE MESSAGES =========
Lina Config application was successful
Lina write mem operation successful

================================================================

Transaction ID: 408021967142
Device UUID: XXXXX

Selected policy group list: PG.FIREWALL.NGFWAccessControlPolicy, IntrusionPolicy, IntrusionPolicy, IntrusionPolicy, IntrusionPolicy

Out-of-date policy group list: PG.FIREWALL.NGFWAccessControlPolicy, IntrusionPolicy, IntrusionPolicy, IntrusionPolicy, IntrusionPolicy

Deployment Type: Full Deployment

================================================================

Transcript after deploying an ACP rule change to add subnets to a block rule, same SNMP and OSPF commands (never observed the other config session statement, but this happens without it):

 

=========SNORT APPLY=========

========= CLI APPLY =========

FMC >> clear configuration session OBJECT
FMC >> clear configuration session FMC_SESSION_1
XXXXX >> [info] : Session FMC_SESSION_1 does not exist.

FMC >> clear configuration session FMC_SESSION_2
XXXXX >> [info] : Session FMC_SESSION_2 does not exist.

FMC >> no strong-encryption-disable
FMC >> configure session OBJECT

...

FMC >> commit noconfirm revert-save
XXXXX >> [info] : WARNING: Other committed configuration sessions currently exist (use 'show configuration session' command to view them). Overlapping changes between these sessions might lead to inconsistent/unexpected changes to the running configuration when reverted in the wrong order.

FMC >> snmp-server user XXXXX Priv v3 auth sha ****** priv aes 128 ******
FMC >> router ospf 1
FMC >> area 0 range XXXXX 255.255.255.0 advertise
FMC >> crypto isakmp nat-traversal
FMC >> clear configuration session FMC_SESSION_1
FMC >> clear configuration session FMC_SESSION_2
XXXXX >> [info] : Session FMC_SESSION_2 does not exist.

FMC >> clear configuration session OBJECT


========= INFRASTRUCTURE MESSAGES =========
Lina Config application was successful
Lina write mem operation successful

================================================================

Transaction ID: 408021966169
Device UUID: XXXXX

Selected policy group list: PG.FIREWALL.NGFWAccessControlPolicy

Out-of-date policy group list: PG.FIREWALL.NGFWAccessControlPolicy

Deployment Type: Full Deployment

================================================================

And the most significant one that has caused several outages, with TAC noting the only way to source them is with audit logs that don't show it, this removal of ospf authentication. I do not know what change was associated with the first occurrence of this, but the second occurrence was associated with a NAT policy update to add a PAT pool for a subnet. This one is really odd. When the NAT change was rolled back, the OSPF authentication was ADDED BACK - with the correct key!! The user did nothing other than remove the NAT config and deploy. The first deploy issued the "no" command, the very next deploy added the config back. There must be some appending here, but I cannot find it in the config or in any documentation. I do not have a full transcript of either the removal or addition, but I do have a partial snippet of the removal:

 

FMC >> no strong-encryption-disable
FMC >> interface Port-channel1.3
FMC >> no ospf authentication message-digest
FMC >> no ospf message-digest-key 1 md5 ******

This config is on our outside interface, outside zone, and please note it is a subinterface. I think I have read that may be against best practices. It is not the same OSPF area as the previous OSPF commands.

 

Any help or guidance towards finding the source of the SNMP commands, the OSPF config, and the removal of the OSPF authentication would be really appreciated.

 

Thank you very much.

3 Replies 3

y00160609
Level 1
Level 1

it‘s helpful,thanks

Marvin Rhoads
Hall of Fame
Hall of Fame

I'd check the system updates schedule and scheduled tasks sections in FMC.

Are there perhaps some Flexconfigs that are set to apply during every deployment? That combined with automatic rule updates or scheduled deployment tasks could result in what you are seeing.

 

 

I really appreciate the suggestions, Marvin. Thank you.

 

I had previously checked the scheduled jobs - and this is my ignorance speaking - but I could not imagine that any of these jobs would cause these specific command sequences to be deployed so consistently, or the single OSPF MD5 removal command to be issued on the next manual deploy, or for the OSPF MD5 addition command to be reissued upon the subsequent manual deployed just 10 minutes later. 

 

FMC-scheduled-jobs.png

 

There were FlexConfigs as well, but they were commands to remove a non-existent applet. This applet was for multicast routing, but the FlexConfig was removing config that already does not exist, and I suppose that might have unintended consequences but I can't tie the specific events to this FlexConfig.  I had the FlexConfig removed as good practice cleanup, but I do not see how they would have caused a consistent SNMP command to be issued as well as pair of remove/add MD5 commands. Unless these are commands that are somehow the result of internal checks the system does when the schedules are run, but I can't fathom how they'd be related. I am certainly open to the fact I could be completely wrong, and I am just trying to understand the behavior so I can self-recover. There isn't anything I can find in Cisco literature, at least yet, to lead me to a plausible cause.

 

Is there a place I should look for any scripts that might be leftovers from the old ASA migration, maybe set in a GUI-inaccessible area? This issue smells like some automation check/change type thing running in the back end. I have searched the full running config for the strings of commands deployed, but they don't exist in the FTD config. Maybe as a script in CLISH? There are a ton of custom tools built in there.

Review Cisco Networking for a $25 gift card