SYMPTOMS
FTD's running v7.0.1 or v7.1.x , SNORT dies & Memory Blocks deplete, causing traffic flows to stop completely.
TAC Response
Known software bugs when run on 2100 series hardware. Upgrade to v7.1, same fault is present. This should now be fixed in release v7.0.2, but not tested by us, due to NO way to revert from v7.1 to v7.0.1. Downgrade is the ONLY option
*** NOTE - the v7.1 revert option is ONLY valid for 30days, after which the revert files are deleted ***
So I went with V7.0.1 when it became a Gold Star standard back in November 2021, that was a bad mistake. With two HA pairs randomly stopping processing traffic due to known Bugs, why Cisco made it Gold Star is beyond me. On the plus side v7.x has much better visibility of the FTD's functional state.
The Procedure
How to downgrade an FTD Image, for FTD-21xx HA Pair on FMC, whilst retaining Live traffic flows.
*** You MUST have console access to the FTD's before you start ***
*** Allow approximately 4hr's per FTD, 8hrs for an HA pair ***
*** This was tested from 7.1.x to 6.6.x on FTD 2140’s ***
*** Minus the FMC steps, this should work for any FTD ***
- 1. Backup the device config in FMC & Save backup offline (not included in time estimates above).
- 2. Make notes of all device interface/routing & NAT/VPN configuration (not in time estimates).
- 3. Reboot standby FTD (nominally the Secondary Unit).
- 4. Failover to Standby.
- 5. Isolate now standby unit (Primary Unit) from inside/outside networks, by disabling switch ports.
- 6. Break FTD HA on FMC.
- 7. Delete any VPN's associated with HA or a member FTD to be deleted (All other config is retained).
- 8. Delete target FTDs from the FMC (one will continue to function whilst you downgrade the other).
- Perform a Complete Reimage (Cisco Procedure in this link)
https://www.cisco.com/c/en/us/td/docs/security/firepower/2100/troubleshoot_fxos/b_2100_CLI_Troubleshoot/b_2100_CLI_Troubleshoot_chapter_011.html#task_uzp_kv1_hbb
*** A TFTP server MUST be used if no local image option, ensure Image file is in the correct ***
Connect via Console session to the FTD.
firepower # connect local-mgmt
firepower(local-mgmt) # format everything (Clears FTD images/config & Reboots FTD)
Hit ESC to break boot when message below seen.
Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.
(The system reboots and stops at the ROMMON prompt)
rommon 1 >
rommon 2 > ADDRESS=1.1.1.2 (set to your device Management IP)
rommon 3 > NETMASK=255.255.255.xxx (set to your subnet mask)
rommon 4 > GATEWAY=1.1.1.1 (set to your subnet Gateway)
rommon 5 > SERVER=2.2.2.2 (TFTP/FTP Server IP)
rommon 6 > IMAGE=cisco-ftd-fpXk.6.6.X-XX.SPA (change to your preferred Cisco Image)
rommon 7 > set
rommon 8 > sync
rommon 9 > ping 2.2.2.2 (Test ping the TFTP server, only works one way)
rommon 10 > tftp -b (TFTP boot new image)
*** This downloads approx. 268MB of 1.2GB image before rebooting, it takes approx. 50minutes ***
Once the system comes up, log in as admin PW= Admin123 and reconfigure the management IP address:
Set IP via Scope Fabric commands
firepower#/ scope fabric-interconnect a
firepower /fabric-interconnect # set out-of-band static ip 1.1.1.2 netmask 255.255.255.xxx gw 1.1.1.1
firepower /fabric-interconnect # commit-buffer (see cisco link if error on commit)
firepower /fabric-interconnect # exit
Download/Install Image via FTP with Scope Firmware commands
firepower # scope firmware
firepower # download image ftp://username:password@2.2.2.2/cisco-ftd-fpXk.6.6X-XX.SPA
(enter password if/when prompted)
firepower # show download-task (+detail for more info)
firepower # show package (To confirm download was successful)
firepower # scope auto-install
firepower # install security-pack version 6.6.4-59 force (yes, yes)
*** System will reboot (this takes a while) & need reconfigured as per a new FTD ***
- Login via Console & connect FTD, Set new Password & accept EULA.
- Re apply Management interface config, SSH access list & Managers/Keys, test SSH access
- Add back to FMC, select old Policy if still relevant, or new.
- Rebuild Device interface & routing plus NAT & VPN Configs manually if they no longer exist on the FMC.
*** Interface, Routing & VPN's will not exist (remember to tick enable interface boxes) ***
*** NAT may need interfaces updated, depending if it was created as individual FTD or HA originally ***
*** Copy NAT policy beforehand if it has interface groups names inside_ig etc, replace with inside interface ***
- Deploy updated config, plus Health Policy etc.
*** It is impossible to restore backup from later version software to earlier version software ***
- Confirm config matches live FTD from your records, or you can do a manual show run & diff compare.
- Isolate live FTD from inside/outside networks.
- Reconnect downgraded FTD to the inside/outside networks.
- Test customer/inter-site/VPN traffic flows. Ensure all monitoring returns to normal status. Get confirmation.
- If reintroduction @ #17 is successful start upgrade on isolated FTD. Return to #9 to start.
- Once 2nd rebuilt FTD is available in FMC, create a HA using the 1st rebuilt unit as the primary & it will inherit the config from the active FTD.
*** You will need to add Health Policy to the 2nd rebuilt unit ***
- Reconnect 2nd downgraded FTD to the inside/outside networks.
- Check HA status & if good do failover, failback testing.
Mission complete, phew
Created by Chris Walker, May 2022.
