topic Re: How to troubleshoot (or recover from) FTD/FMC Deployment failure in Network Security

How to troubleshoot (or recover from) FTD/FMC Deployment failure

tnakano03 — Fri, 21 Feb 2020 15:43:33 GMT

Model/Version:

Firepower 2110/Threat Defense (77) Version 6.2.2.3 (Build 66)

Firepower Management Center for VMWare/Software Version 6.2.3 (build 83)

===Issue

I modified "Floating Connection" timeouts parameter to 30 sec (default is 0) in Platform Settings and I deployed the new config from FMC to FTD. For some reason, the deployment failed. So, I set back the the "Floating Connection" timeouts parameter to default and push the config again. Now the deployment failed again. I rebooted both Active/Standby FTDs. But I still get the same error..."Deployment failed. Please modify the description of your Access Policy, save the policy, and attempt the deploy again. If problem persists after retrying, contact Cisco TAC." I can get only generic error messages and I don't know where to start troubleshooting. I wish FMC has a feature like Juniper or PAN such as commit check or validate the config and tell where is a wrong config.....

Question:

How can we troubleshoot a deployment issue? Or how can we cancel the bad deployment?

So far, I've checked the followings...

1. Deploy transcript in FMC => Too generic and I see any clue other that it was roll backed scucessfully.

=========SNORT APPLY=========
========= CLI APPLY =========

========= INFRASTRUCTURE MESSAGES =========
=========SNORT APPLY=========
========= CLI APPLY =========

========= INFRASTRUCTURE MESSAGES =========
null
Platform settings were successful.
Lina Files Rollback successful

Rollback APP was successful.ClusterAppConfRollbackStatus : 1
ClusterFileCopyFileName : null
MSG_ID : 35
NODE_ID : 1

2. tail -f /ngfw/var/log/action_queue.log in FTD => I see the roll back was run. But I don't see why FTD had to roll back....

May 6 17:15:33 NYP-EDGE-FW01-MDF policy_apply.pl[10441]: --Timer 'SF::NGFW::PolicyApply::signalDetection' 15.037 sec
May 6 17:15:33 NYP-EDGE-FW01-MDF policy_apply.pl[10441]: --Timer 'snort DAQ reload for de fcbdd69a-082d-11e8-8eb8-79ec0b048beb' 0.403 sec
May 6 17:15:33 NYP-EDGE-FW01-MDF policy_apply.pl[10441]: --Timer 'reload RNA on de fcbdd69a-082d-11e8-8eb8-79ec0b048beb' 1.056 sec
May 6 17:15:33 NYP-EDGE-FW01-MDF policy_apply.pl[10441]: --Timer 'hup sftunnel' 0.007 sec
May 6 17:15:33 NYP-EDGE-FW01-MDF policy_apply.pl[10441]: store deployment state to disk... at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/UMPD/Transaction.pm line 914.
May 6 17:15:33 NYP-EDGE-FW01-MDF policy_apply.pl[10441]: Released Deployment lock at /ngfw/usr/local/sf/bin/policy_apply.pl line 228.
May 6 17:15:33 NYP-EDGE-FW01-MDF policy_apply.pl[10441]: policy apply phase SIGNAL exiting with exit code: 0 at /ngfw/usr/local/sf/bin/policy_apply.pl line 51.
May 6 17:16:28 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: policy_apply.pl called with $VAR1 = [
May 6 17:16:28 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: '/var/cisco/deploy/db/21474854216/policy_deployment.db',
May 6 17:16:28 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: 'ROLLBACK',
May 6 17:16:28 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: undef
May 6 17:16:28 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: ];
May 6 17:16:28 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: including code from /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code at /ngfw/usr/local/sf/bin/policy_apply.pl line 115.
May 6 17:16:30 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: fmc_version is 6.2.3 and device_version is 6.2.2.3 . at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/NGFW/PolicyApply.pm line 2074.
May 6 17:16:30 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: Setting backward compatibility for devices with 6.2.2 version at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/NGFW/PolicyApply.pm line 2100.
May 6 17:16:30 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: Checking if any conflicting process is running at /ngfw/usr/local/sf/bin/policy_apply.pl line 168.
May 6 17:16:30 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: Checking for lock file /ngfw/var/sf/geodb/peers/.push_lck at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/NGFW/PolicyApply.pm line 300.
May 6 17:16:30 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: Checking for lock file /ngfw/var/sf/run/deployment.lock at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/NGFW/PolicyApply.pm line 300.
May 6 17:16:30 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: Checking for lock file /ngfw/var/sf/run/securityIntelligence.lock at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/NGFW/PolicyApply.pm line 300.
May 6 17:16:30 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: No conflicting process is running at /ngfw/usr/local/sf/bin/policy_apply.pl line 173.
May 6 17:16:30 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: Acquired Deployment lock at /ngfw/usr/local/sf/bin/policy_apply.pl line 178.
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: wrote /ngfw/var/tmp/OOiGTFploa at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/NGFW/PolicyApply.pm line 827.
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: SCRIPT CONTENT:
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: #!/usr/bin/perl
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]:
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: use Data::Dumper;
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: use FlyLoader;
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]:
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: unshift (@INC, '/ngfw/var/cisco/deploy/pkg/var/cisco/packages/exporter-6.2.3-83/code');
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: print "loading '/ngfw/var/cisco/deploy/pkg/var/cisco/packages/exporter-6.2.3-83/code'\n";
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]:
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: print "initiating rollback...\n";
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: SF::NGFW::PolicyApply::deploy( '/var/cisco/deploy/db/21474849832/policy_deployment.db', undef, 1 );
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: print "rollback complete!
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: ";
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]:
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: =========================== EXECUTE ROLLBACK SCRIPT =========================== at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/NGFW/PolicyApply.pm line 829.
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: Warning: something's wrong at /ngfw/var/cisco/deploy/sandbox/exporter-pkg/code/SF/NGFW/PolicyApply.pm line 832.
May 6 17:16:32 NYP-EDGE-FW01-MDF policy_apply.pl[13184]: ========================== DONE WITH ROLLBACK SCRIPT ==============

3. Lina preview => I don't see anything wrong... in here. Do you?

###Flex-config Prepended CLI ###

###CLI generated from managed features ###
logging debug-trace
timeout tcp-proxy-reassembly 0:01:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
monitor-interface diagnostic
no logging fmc MANAGER_VPN_EVENT_LIST

no logging list MANAGER_VPN_EVENT_LIST
logging list MANAGER_VPN_EVENT_LIST level errors class auth
logging list MANAGER_VPN_EVENT_LIST level errors class vpn
logging list MANAGER_VPN_EVENT_LIST level errors class vpnc
logging list MANAGER_VPN_EVENT_LIST level errors class vpnfo
logging list MANAGER_VPN_EVENT_LIST level errors class vpnlb
logging list MANAGER_VPN_EVENT_LIST level errors class webfo
logging list MANAGER_VPN_EVENT_LIST level errors class webvpn
logging list MANAGER_VPN_EVENT_LIST level errors class ca
logging list MANAGER_VPN_EVENT_LIST level errors class svc
logging list MANAGER_VPN_EVENT_LIST level errors class ssl
logging list MANAGER_VPN_EVENT_LIST level errors class dap
logging list MANAGER_VPN_EVENT_LIST level errors class ipaa
logging fmc MANAGER_VPN_EVENT_LIST

###Flex-config Appended CLI ###

Re: How to troubleshoot (or recover from) FTD/FMC Deployment failure

tnakano03 — Mon, 07 May 2018 01:00:04 GMT

Addtional Info:

I've checked the syslog in FTD but I can not find the reason why the deployment failed.

I don't want to be side tracked here but... In traditional ASA (Active/Standby), we will configure ASA active device and sync the config to ASA standby device. However, in FMC, it seems each commands such as 'timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02' are executed on both Active and Standby separately. Or I could be reading this syslog output wrong....

10.11.150.11 is Active FTD

10.11.150.12 is Standby FTD

===syslog output during the deployment failure:

May 6 20:17:17 10.11.150.11 :May 07 00:17:16 UTC: %ASA-sys-6-199018: May 7 00:17:17 NYP-EDGE-FW01-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][modification][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][111731][fabric/lan/A/pc-12][operSpeed(Old:1gbps, New:10gbps)][] Port Channel A/Port-channel12/12 modified
May 6 20:17:17 10.11.150.11 :May 07 00:17:16 UTC: %ASA-sys-6-199018: May 7 00:17:17 NYP-EDGE-FW01-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][modification][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][111732][fabric/lan/A/pc-11][operSpeed(Old:1gbps, New:10gbps)][] Port Channel A/Port-channel11/11 modified
May 6 20:17:17 10.11.150.11 :May 07 00:17:16 UTC: %ASA-sys-6-199018: May 7 00:17:17 NYP-EDGE-FW01-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][modification][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][111733][fabric/lan/A/pc-10][operSpeed(Old:1gbps, New:10gbps)][] Port Channel A/Port-channel10/10 modified
May 6 20:17:17 10.11.150.11 :May 07 00:17:16 UTC: %ASA-sys-6-199018: May 7 00:17:17 NYP-EDGE-FW01-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][modification][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][111734][sys/svc-ext/snmp-svc][descr(Old:SNMP Service, New:)][] SNMP service modified
May 6 20:17:17 10.11.150.11 :May 07 00:17:16 UTC: %ASA-sys-5-199017: May 7 00:17:17 NYP-EDGE-FW01-MDF syslog_utils: Set the system console level to: critical
May 6 20:17:17 10.11.150.11 :May 07 00:17:16 UTC: %ASA-sys-5-199017: May 7 00:17:17 NYP-EDGE-FW01-MDF syslog-ng[7025]: Configuration reload request received, reloading configuration;
May 6 20:17:17 10.11.150.11 :May 07 00:17:16 UTC: %ASA-sys-5-199017: May 7 00:17:17 NYP-EDGE-FW01-MDF syslog_utils: Set messages sent to lina level to: information
May 6 20:17:17 10.11.150.11 :May 07 00:17:16 UTC: %ASA-sys-5-199017: May 7 00:17:17 NYP-EDGE-FW01-MDF syslog-ng[7025]: Configuration reload request received, reloading configuration;
May 6 20:17:17 10.11.150.11 :May 07 00:17:17 UTC: %ASA-sys-5-199017: May 7 00:17:17 NYP-EDGE-FW01-MDF syslog_utils: Set the system log level to: critical
May 6 20:17:17 10.11.150.11 :May 07 00:17:17 UTC: %ASA-sys-5-199017: May 7 00:17:17 NYP-EDGE-FW01-MDF syslog-ng[7025]: Configuration reload request received, reloading configuration;
May 6 20:17:17 10.11.150.11 :May 07 00:17:17 UTC: %ASA-sys-5-199017: May 7 00:17:17 NYP-EDGE-FW01-MDF syslog-ng[7025]: Configuration reload request received, reloading configuration;
May 6 20:17:18 10.11.150.12 :May 07 00:17:18 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show checksum
May 6 20:17:20 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-6-199018: May 7 00:17:20 NYP-EDGE-FW02-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][modification][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][128266][fabric/lan/A/pc-12][operSpeed(Old:1gbps, New:10gbps)][] Port Channel A/Port-channel12/12 modified
May 6 20:17:20 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-6-199018: May 7 00:17:20 NYP-EDGE-FW02-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][modification][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][128267][fabric/lan/A/pc-11][operSpeed(Old:1gbps, New:10gbps)][] Port Channel A/Port-channel11/11 modified
May 6 20:17:20 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-6-199018: May 7 00:17:20 NYP-EDGE-FW02-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][modification][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][128268][fabric/lan/A/pc-10][operSpeed(Old:1gbps, New:10gbps)][] Port Channel A/Port-channel10/10 modified
May 6 20:17:20 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-6-199018: May 7 00:17:20 NYP-EDGE-FW02-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][modification][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][128269][sys/svc-ext/snmp-svc][descr(Old:SNMP Service, New:)][] SNMP service modified
May 6 20:17:20 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-5-199017: May 7 00:17:20 NYP-EDGE-FW02-MDF syslog_utils: Set the system console level to: critical
May 6 20:17:21 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-5-199017: May 7 00:17:21 NYP-EDGE-FW02-MDF syslog-ng[7028]: Configuration reload request received, reloading configuration;
May 6 20:17:21 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-5-199017: May 7 00:17:21 NYP-EDGE-FW02-MDF syslog_utils: Set messages sent to lina level to: information
May 6 20:17:21 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-5-199017: May 7 00:17:21 NYP-EDGE-FW02-MDF syslog-ng[7028]: Configuration reload request received, reloading configuration;
May 6 20:17:21 10.11.150.12 :May 07 00:17:20 UTC: %ASA-sys-5-199017: May 7 00:17:21 NYP-EDGE-FW02-MDF syslog_utils: Set the system log level to: critical
May 6 20:17:21 10.11.150.12 :May 07 00:17:21 UTC: %ASA-sys-5-199017: May 7 00:17:21 NYP-EDGE-FW02-MDF syslog-ng[7028]: Configuration reload request received, reloading configuration;
May 6 20:17:21 10.11.150.12 :May 07 00:17:21 UTC: %ASA-sys-5-199017: May 7 00:17:21 NYP-EDGE-FW02-MDF syslog-ng[7028]: Configuration reload request received, reloading configuration;
May 6 20:17:22 10.11.150.12 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config interface
May 6 20:17:23 10.11.150.12 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config interface
May 6 20:17:23 10.11.150.11 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config interface
May 6 20:17:23 10.11.150.12 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config interface
May 6 20:17:23 10.11.150.11 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config interface
May 6 20:17:23 10.11.150.12 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config interface
May 6 20:17:23 10.11.150.12 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config failover
May 6 20:17:23 10.11.150.12 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show failover
May 6 20:17:23 10.11.150.11 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config interface
May 6 20:17:23 10.11.150.11 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config interface
May 6 20:17:23 10.11.150.11 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config failover
May 6 20:17:23 10.11.150.11 :May 07 00:17:22 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show failover
May 6 20:17:24 10.11.150.12 :May 07 00:17:24 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show version
May 6 20:17:25 10.11.150.11 :May 07 00:17:25 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show version
May 6 20:17:53 10.11.150.11 :May 07 00:17:52 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show failover
May 6 20:17:54 10.11.150.11 :May 07 00:17:53 UTC: %ASA-config-5-111008: User 'enable_1' executed the 'pager 0' command.
May 6 20:17:54 10.11.150.11 :May 07 00:17:53 UTC: %ASA-config-5-111010: User 'enable_1', running 'N/A' from IP 0.0.0.0, executed 'pager 0'
May 6 20:17:54 10.11.150.12 :May 07 00:17:53 UTC: %ASA-config-5-111008: User 'failover' executed the 'pager 0' command.
May 6 20:17:54 10.11.150.12 :May 07 00:17:53 UTC: %ASA-config-5-111010: User 'failover', running 'N/A' from IP 0.0.0.0, executed 'pager 0'
May 6 20:17:54 10.11.150.11 :May 07 00:17:54 UTC: %ASA-config-5-111008: User 'enable_1' executed the 'more system:running-config' command.
May 6 20:17:54 10.11.150.11 :May 07 00:17:54 UTC: %ASA-config-5-111010: User 'enable_1', running 'N/A' from IP 0.0.0.0, executed 'more system:running-config'
May 6 20:17:56 10.11.150.11 :May 07 00:17:56 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show version
May 6 20:17:56 10.11.150.11 :May 07 00:17:56 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show crypto key mypubkey rsa
May 6 20:17:56 10.11.150.11 :May 07 00:17:56 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show asp inspect-dp snort
May 6 20:17:56 10.11.150.11 :May 07 00:17:56 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show interface
May 6 20:17:56 10.11.150.11 :May 07 00:17:56 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show running-config all monitor-interface
May 6 20:18:07 10.11.150.11 :May 07 00:18:07 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show failover
May 6 20:18:15 10.11.150.11 :May 07 00:18:14 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show checksum
May 6 20:18:35 10.11.150.12 :May 07 00:18:35 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show checksum
May 6 20:18:39 10.11.150.12 :May 07 00:18:39 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show interface
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show failover
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show cluster info
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-7-111009: User 'enable_1' executed cmd: show checksum
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111008: User 'enable_1' executed the 'strong-encryption-disable' command.
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111010: User 'enable_1', running 'N/A' from IP 0.0.0.0, executed 'strong-encryption-disable'
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111008: User 'enable_1' executed the 'logging debug-trace' command.
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111010: User 'enable_1', running 'N/A' from IP 0.0.0.0, executed 'logging debug-trace'
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111008: User 'failover' executed the 'strong-encryption-disable' command.
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111010: User 'failover', running 'N/A' from IP 0.0.0.0, executed 'strong-encryption-disable'
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111008: User 'enable_1' executed the 'timeout tcp-proxy-reassembly 0:01:00' command.
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111010: User 'enable_1', running 'N/A' from IP 0.0.0.0, executed 'timeout tcp-proxy-reassembly 0:01:00'
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111008: User 'enable_1' executed the 'timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02' command.
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111010: User 'enable_1', running 'N/A' from IP 0.0.0.0, executed 'timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02'
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111008: User 'enable_1' executed the 'no user-identity default-domain LOCAL' command.
May 6 20:19:58 10.11.150.11 :May 07 00:19:57 UTC: %ASA-config-5-111010: User 'enable_1', running 'N/A' from IP 0.0.0.0, executed 'no user-identity default-domain LOCAL'
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111008: User 'failover' executed the 'logging debug-trace' command.
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111010: User 'failover', running 'N/A' from IP 0.0.0.0, executed 'logging debug-trace'
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111008: User 'failover' executed the 'timeout tcp-proxy-reassembly 0:01:00' command.
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111010: User 'failover', running 'N/A' from IP 0.0.0.0, executed 'timeout tcp-proxy-reassembly 0:01:00'
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111008: User 'failover' executed the 'timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02' command.
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111010: User 'failover', running 'N/A' from IP 0.0.0.0, executed 'timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02'
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111008: User 'failover' executed the 'no user-identity default-domain LOCAL' command.
May 6 20:19:58 10.11.150.12 :May 07 00:19:58 UTC: %ASA-config-5-111010: User 'failover', running 'N/A' from IP 0.0.0.0, executed 'no user-identity default-domain LOCAL'
May 6 20:19:59 10.11.150.12 :May 07 00:19:59 UTC: %ASA-sys-6-199018: May 7 00:19:59 NYP-EDGE-FW02-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][modification][1525649179/2e4b3e3c-b7f2-46c2-a165-f08846a06852OzJjAoIUhh5GFNXZsTc5K][128474][sys/svc-ext/snmp-svc][descr(Old:, New:SNMP Service)][] SNMP service modified
May 6 20:19:59 10.11.150.12 :May 07 00:19:59 UTC: %ASA-sys-5-199017: May 7 00:19:59 NYP-EDGE-FW02-MDF syslog_utils: Set the system console level to: critical
May 6 20:19:59 10.11.150.12 :May 07 00:19:59 UTC: %ASA-sys-5-199017: May 7 00:19:59 NYP-EDGE-FW02-MDF syslog-ng[7028]: Configuration reload request received, reloading configuration;
May 6 20:20:00 10.11.150.12 :May 07 00:19:59 UTC: %ASA-sys-5-199017: May 7 00:20:00 NYP-EDGE-FW02-MDF syslog_utils: Set messages sent to lina level to: information
May 6 20:20:00 10.11.150.12 :May 07 00:19:59 UTC: %ASA-sys-5-199017: May 7 00:20:00 NYP-EDGE-FW02-MDF syslog-ng[7028]: Configuration reload request received, reloading configuration;
May 6 20:20:00 10.11.150.12 :May 07 00:19:59 UTC: %ASA-sys-5-199017: May 7 00:20:00 NYP-EDGE-FW02-MDF syslog_utils: Set the system log level to: critical
May 6 20:20:00 10.11.150.12 :May 07 00:19:59 UTC: %ASA-sys-5-199017: May 7 00:20:00 NYP-EDGE-FW02-MDF syslog-ng[7028]: Configuration reload request received, reloading configuration;
May 6 20:20:00 10.11.150.12 :May 07 00:20:00 UTC: %ASA-sys-5-199017: May 7 00:20:00 NYP-EDGE-FW02-MDF syslog-ng[7028]: Configuration reload request received, reloading configuration;
May 6 20:20:01 10.11.150.11 :May 07 00:20:01 UTC: %ASA-sys-6-199018: May 7 00:20:01 NYP-EDGE-FW01-MDF FPRM: <<%%FPRM-6-AUDIT>> [admin][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][modification][1525651181/9c22c8df-b2cc-4618-8523-5a192713c21coGdmsekp0CU5qE9hnxxjm][112034][sys/svc-ext/snmp-svc][descr(Old:, New:SNMP Service)][] SNMP service modified
May 6 20:20:01 10.11.150.11 :May 07 00:20:01 UTC: %ASA-sys-5-199017: May 7 00:20:01 NYP-EDGE-FW01-MDF syslog_utils: Set the system console level to: critical
May 6 20:20:01 10.11.150.11 :May 07 00:20:01 UTC: %ASA-sys-5-199017: May 7 00:20:01 NYP-EDGE-FW01-MDF syslog-ng[7025]: Configuration reload request received, reloading configuration;
May 6 20:20:02 10.11.150.11 :May 07 00:20:01 UTC: %ASA-sys-5-199017: May 7 00:20:02 NYP-EDGE-FW01-MDF syslog_utils: Set messages sent to lina level to: information
May 6 20:20:02 10.11.150.11 :May 07 00:20:01 UTC: %ASA-sys-5-199017: May 7 00:20:02 NYP-EDGE-FW01-MDF syslog-ng[7025]: Configuration reload request received, reloading configuration;
May 6 20:20:02 10.11.150.11 :May 07 00:20:01 UTC: %ASA-sys-5-199017: May 7 00:20:02 NYP-EDGE-FW01-MDF syslog_utils: Set the system log level to: critical
May 6 20:20:02 10.11.150.11 :May 07 00:20:01 UTC: %ASA-sys-5-199017: May 7 00:20:02 NYP-EDGE-FW01-MDF syslog-ng[7025]: Configuration reload request received, reloading configuration;
May 6 20:20:02 10.11.150.11 :May 07 00:20:01 UTC: %ASA-sys-5-199017: May 7 00:20:02 NYP-EDGE-FW01-MDF syslog-ng[7025]: Configuration reload request received, reloading configuration;
May 6 20:20:31 10.11.150.12 :May 07 00:20:30 UTC: %ASA-ha-1-105006: (Secondary) Link status 'Up' on interface INSIDE-VLAN14
May 6 20:20:31 10.11.150.12 :May 07 00:20:30 UTC: %ASA-ha-1-105006: (Secondary) Link status 'Up' on interface INSIDE-VLAN16
May 6 20:20:31 10.11.150.12 :May 07 00:20:30 UTC: %ASA-ha-1-105006: (Secondary) Link status 'Up' on interface INSIDE-VLAN18
May 6 20:20:31 10.11.150.12 :May 07 00:20:30 UTC: %ASA-ha-1-105006: (Secondary) Link status 'Up' on interface INSIDE-VLAN150
May 6 20:20:31 10.11.150.12 :May 07 00:20:30 UTC: %ASA-ha-1-105006: (Secondary) Link status 'Up' on interface OUTSIDE-VLAN801
May 6 20:20:31 10.11.150.12 :May 07 00:20:30 UTC: %ASA-ha-1-105006: (Secondary) Link status 'Up' on interface OUTSIDE-VLAN901
May 6 20:20:31 10.11.150.12 :May 07 00:20:30 UTC: %ASA-ha-1-105006: (Secondary) Link status 'Up' on interface DMZ-NYP
May 6 20:20:31 10.11.150.12 :May 07 00:20:30 UTC: %ASA-ha-1-105006: (Secondary) Link status 'Up' on interface diagnostic
May 6 20:20:33 10.11.150.12 :May 07 00:20:33 UTC: %ASA-ha-1-105003: (Secondary) Monitoring on interface INSIDE-VLAN14 waiting
May 6 20:20:33 10.11.150.12 :May 07 00:20:33 UTC: %ASA-ha-1-105003: (Secondary) Monitoring on interface INSIDE-VLAN16 waiting
May 6 20:20:33 10.11.150.12 :May 07 00:20:33 UTC: %ASA-ha-1-105003: (Secondary) Monitoring on interface INSIDE-VLAN18 waiting
May 6 20:20:33 10.11.150.12 :May 07 00:20:33 UTC: %ASA-ha-1-105003: (Secondary) Monitoring on interface INSIDE-VLAN150 waiting
May 6 20:20:33 10.11.150.12 :May 07 00:20:33 UTC: %ASA-ha-1-105003: (Secondary) Monitoring on interface OUTSIDE-VLAN801 waiting
May 6 20:20:33 10.11.150.12 :May 07 00:20:33 UTC: %ASA-ha-1-105003: (Secondary) Monitoring on interface OUTSIDE-VLAN901 waiting
May 6 20:20:33 10.11.150.12 :May 07 00:20:33 UTC: %ASA-ha-1-105003: (Secondary) Monitoring on interface DMZ-NYP waiting
May 6 20:20:35 10.11.150.11 :May 07 00:20:34 UTC: %ASA-ha-1-105004: (Primary) Monitoring on interface INSIDE-VLAN14 normal
May 6 20:20:35 10.11.150.11 :May 07 00:20:34 UTC: %ASA-ha-1-105004: (Primary) Monitoring on interface INSIDE-VLAN16 normal
May 6 20:20:35 10.11.150.11 :May 07 00:20:34 UTC: %ASA-ha-1-105004: (Primary) Monitoring on interface INSIDE-VLAN18 normal
May 6 20:20:35 10.11.150.11 :May 07 00:20:34 UTC: %ASA-ha-1-105004: (Primary) Monitoring on interface INSIDE-VLAN150 normal
May 6 20:20:35 10.11.150.11 :May 07 00:20:34 UTC: %ASA-ha-1-105004: (Primary) Monitoring on interface OUTSIDE-VLAN801 normal
May 6 20:20:35 10.11.150.11 :May 07 00:20:34 UTC: %ASA-ha-1-105004: (Primary) Monitoring on interface OUTSIDE-VLAN901 normal
May 6 20:20:35 10.11.150.11 :May 07 00:20:34 UTC: %ASA-ha-1-105004: (Primary) Monitoring on interface DMZ-NYP normal
May 6 20:20:39 10.11.150.12 :May 07 00:20:39 UTC: %ASA-ha-6-210022: LU missed 4 updates
May 6 20:20:43 10.11.150.12 :May 07 00:20:43 UTC: %ASA-ha-1-105004: (Secondary) Monitoring on interface INSIDE-VLAN14 normal
May 6 20:20:43 10.11.150.12 :May 07 00:20:43 UTC: %ASA-ha-1-105004: (Secondary) Monitoring on interface INSIDE-VLAN16 normal
May 6 20:20:43 10.11.150.12 :May 07 00:20:43 UTC: %ASA-ha-1-105004: (Secondary) Monitoring on interface INSIDE-VLAN18 normal
May 6 20:20:43 10.11.150.12 :May 07 00:20:43 UTC: %ASA-ha-1-105004: (Secondary) Monitoring on interface INSIDE-VLAN150 normal
May 6 20:20:43 10.11.150.12 :May 07 00:20:43 UTC: %ASA-ha-1-105004: (Secondary) Monitoring on interface OUTSIDE-VLAN801 normal
May 6 20:20:43 10.11.150.12 :May 07 00:20:43 UTC: %ASA-ha-1-105004: (Secondary) Monitoring on interface OUTSIDE-VLAN901 normal
May 6 20:20:43 10.11.150.12 :May 07 00:20:43 UTC: %ASA-ha-1-105004: (Secondary) Monitoring on interface DMZ-NYP normal

Re: How to troubleshoot (or recover from) FTD/FMC Deployment failure

tnakano03 — Tue, 08 May 2018 14:15:35 GMT

I opened a Cisco TAC case. Cisco support gave me the link (a bug) and we tried the workaround (rename Access Policy description/name and redeploy it) but it didn't fix my problem. I submitted Troubleshoot files of FTD and FMC to Cisco TAC. I really want to know where the deployment is failing....

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvi74560/?reffering_site=dumpcr

Re: How to troubleshoot (or recover from) FTD/FMC Deployment failure

tnakano03 — Fri, 11 May 2018 14:33:36 GMT

Resolved!

It took 4 days...but finally I got an experienced FTD/FMC TAC engineer and pointed me to the right direction. FTD/FMC has a troubleshooting tool called "pigtail deploy" (in linux mode) to show all deployment related debug logs in one session. I recommend to redirect a console output to a text file since they have a lot of outputs. Then, you need to find key word "ERROR:" to spot what FTD is complaining about.

[How to use "pigtail deploy"]

--FTD

> expert
admin@FTD:~$ sudo su -
Password:
root@FTD:~# pigtail deploy

--FMC

admin@firepower:~$ sudo su -
Password:
root@firepower:~# pigtail deploy

[Root case of my issue and how to fix]

Step1 - Root Cause:

I found the following error in FTD pigtail deploy output. I didn't ask to FMC but it seems FMC was trying to remove the following route-map. However, I was using it in Policy Based Routing....

NGFW: 05-10 22:56:00 <error-info id="36" type="error">ERROR: route-map RMAP-PBR-10.11.14.0-PL23 is attached to routing protocols
NGFW: 05-10 22:56:00 (EIGRP/RIP/OSPF/BGP/ISIS) or used in policy based routing.
NGFW: 05-10 22:56:00 Please remove the relevant configuration before removing the route_map

Step2 - How to fix.

I used the existing Flexconfig object called "Policy_Based_Routing_Clear" in Flexconfig and deployed. This time, the deployment was successful!

[Lesson Learned]

1. Flexconfig is very hard to use. Specially, the order of commands to put in Flexconfig

2. Need to check Preview carefully before you apply Flexconfig (You might see unexpected commands in there generated by FMC, not from your Flexconfig. And that might be conflicting with your intent)

3. When you have a deployment issue, "pigtail deploy" is your best friend

Re: How to troubleshoot (or recover from) FTD/FMC Deployment failure

Marvin Rhoads — Sun, 13 May 2018 14:05:23 GMT

Thanks for sharing the fix to your problem. +5

I've had some Cisco staff recommend to avoid Flexconfig if you don't really need those few features only available via it. It's a bit of a kludge to expose features they haven't quite gotten into the UI (or API) just yet. Unfortunately there are a number of things important to many customers that can only be configured that way. Catch 22.