CSCvn82378 - Traffic through ASA%2FFTD might stop passing upon upgrading FMC to 6.2.3.8-51

[davi.bittencourt]

Hello,

I'm facing this issue and a lot of downtime in the envinronment. I have downgraded the version of FTD device from 6.2.3.8 to 6.2.3.6, but the problem still happening. TAC support told me that is not needed to downgrade the FMC version, but I'm confusing and I guess that I will do it. Any suggestion?

[aaaaahhhhh]

My resolution was to spin up a new vFMC and point the sensors to that.  I exported the policies from the old one to the new one and now am happy.  I have also upgraded to 6.3.0.  Upgrading the old vFMC and sensors to 6.3.0 didn't work.

[ocsic25]

You cannot have your FMC or any managed device running v6.2.3.8 as that code contains the critical bug that somehow made it through Cisco QA.

If your managed devices have already been upgraded to v6.2.3.8 you'll need to downgrade them via the CLI first, then downgrade your FMC to v6.2.3.7 and redeploy policies.

[4T2]

To downgrade an SFR or FTD box you can do the following. (THIS WILL REBOOT THE DEVICE WHEN DONE)

1. SSH to the device
2. Escalate to BASH with command “expert”
3. Become super user “ sudo su “
4. Navigate to /var/sf/updates with “ cd /var/sf/updates “
5. Locate the uninstaller file for 6.2.3.8 with command “ ls –lah “
6. Execute following command   “install_update.pl /var/sf/updates/< name of the uninstall tar> “

Once that is done run the uninstaller in FMC.

If you run scheduled updates/upgrades be sure to delete the bad upgrades from the FMC under system > updates as well.

The only workaround we've seen is to possibly disable all file inspection policy in your access control policies.

[Barrett Cowan]

I can also confirm, Cisco TAC tells me to disable file inspection for a workaround, until 6.2.3.9 comes out tomorrow with the permanent fix. No downgrade needed.

[davi.bittencourt]

Yes, I did it and it worked for me. I'm waiting the new release with the permanent fix.

[foo.bk001]

Thanks for the tips on how to run the uninstaller. After I ran the uninstaller, the session terminated and the SFR status remains unresponsive. Rebooted the ASA, and ASA shows the SFR as the same 6.2.3.8-51 version.

[190109 23:30:58:548] BEGIN 200_pre/500_stop_system.sh
Remote card closed command session. Press any key to continue.
Command session with module sfr terminated.
MYKULM12FW1/sec/act# show module sfr

Mod Card Type Model Serial No.
---- -------------------------------------------- ------------------ -----------
sfr FirePOWER Services Software Module ASA5525 FCH19477BZZ

Mod MAC Address Range Hw Version Fw Version Sw Version
---- --------------------------------- ------------ ------------ ---------------
sfr cc46.d6f7.f8e4 to cc46.d6f7.f8e4 N/A N/A 6.2.3.8-51

Mod SSM Application Name Status SSM Application Version
---- ------------------------------ ---------------- --------------------------
sfr ASA FirePOWER Not Applicable 6.2.3.8-51

Mod Status Data Plane Status Compatibility
---- ------------------ --------------------- -------------
sfr Unresponsive Not Applicable

MYKULM12FW1/sec/act# show module sfr de
Getting details from the Service Module, please wait...
Unable to read details from module sfr

Card Type: FirePOWER Services Software Module
Model: ASA5525
Hardware version: N/A
Serial Number: FCH19477BZZ
Firmware version: N/A
Software version: 6.2.3.8-51
MAC Address Range: cc46.d6f7.f8e4 to cc46.d6f7.f8e4
App. name: ASA FirePOWER
App. Status: Not Applicable
App. Status Desc: Not Applicable
App. version: 6.2.3.8-51
Data Plane Status: Not Applicable
Console session: Ready
Status: Unresponsive

[davi.bittencourt]

The version 6.2.3.9 was released. I'll upgrade to this version all environment FMC and FTDs to test. If someone already have upgraded to this version, please, inform to me.

[Lukaszoo]

Did the upgrade to 6.2.3.9 solve the problem? did you previously have the File policy on with Http selected or no ?

Tx

[Maureen Smith]

I ran into the same thing on our system. The problem started after we'd only upgraded the FMC, while the sensor was still running 6.2.3.7. TAC knew right away what the issue was when I called them, and they had me change the system policy so the sensor would be in Monitor mode, which got things running (but of course opened a gaping security hole).

I reverted the FMC back to 6.2.3.7 and restored a backup I'd taken just before the upgrade, and for good measure I wiped and recreated the sensor on our ASA. That seems to have fixed the issue (26+ hours and no blackhole *knocks on wood*).

I'm going to give 6.3.2.9 some more time to mature before I make the leap.

[Synter]

We had to downgrade the FMC as well.  Beware:  The FMC may clear your config and cause interface, routing, and NAT problems.

If you need to restore your backup, you need to have all devices on the same software version. 

We have been going back and forth for 4 days with TAC.  We are still trying to return to a normal state at this time.

Steps that may save you trouble:

1) Downgrade everything to previous software release.

2) Restore from a backup before 6.2.3.8 was applied.  See this bug:CSCvb77246

Edit: 1/11 our system returned to normal.

Beware removing devices from FMC, still problem in 6.2.3.7.  See this bug:CSCvb77246