FMC 6.7.0 failed update

[HQuest]

Anyone else having trouble upgrading FMC to v6.7.0? I'm working on my test box to move from a v6.2.x to the latest v6.7.0 and no matter which upgrade path I choose, the v6.7.0 upgrade always stops with a fatal error on the LDAP External Auth fix script (looks wrong coding IMHO). I've got rid of AD integration and external user authentication settings in the hopes the upgrade process would skip this task, to not avail.

The error message is as follows:

Spoiler

**********************************************************
[201111 20:40:53:958] Starting script: 800_post/1027_ldap_external_auth_fix.pl
Entering script: 800_post/1027_ldap_external_auth_fix.pl
DBD::SQLAnywhere::db selectcol_arrayref failed: Syntax error near '"e;' on line 1  (DBD: prepare failed) at /usr/local/sf/lib/perl/5.10.1/SF/SFDBI.pm line 387.
        called from SF::Util::Stacktrace::ToString at /usr/local/sf/lib/perl/5.10.1/SF/SFDBI.pm, line 388
        called from SF::SFDBI::__ANON__ at /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm, line 110
        called from SF::Snapshots::IntrusionEmail::getRuleMessage at /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm, line 140
        called from SF::Snapshots::IntrusionEmail::Snapshot at /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm, line 165
        called from SF::Snapshots::IntrusionEmail::baseline at /usr/local/sf/lib/perl/5.10.1/SF/Snapshots.pm, line 615
        called from SF::Snapshots::baseline at 800_post/1027_ldap_external_auth_fix.pl, line 18
DBD::SQLAnywhere::db selectcol_arrayref failed: Syntax error near '"e;' on line 1  (DBD: prepare failed) at /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm line 110.
Can't use an undefined value as an ARRAY reference at /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm line 111.

Printing stack trace:
        called from /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm (111)
        called from /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm (140)
        called from /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm (165)
        called from /usr/local/sf/lib/perl/5.10.1/SF/Snapshots.pm (615)
        called from 800_post/1027_ldap_external_auth_fix.pl (18)
Exit return value = 1

Maybe I'll stick with v6.6.1 for a while and wait for a fix for v6.7.

[Marvin Rhoads]

I upgraded my lab FMC to 6.7 successfully. But I keep it up to date so I was going from 6.6.1 as the previous version. It does have AD integration and uses RADIUS external user authentication - both of those are still working OK.

6.6.1 is Gold Star now so that may be a better initial choice.

[HQuest]

Hi Marvin. Thank you, as always, for your comments.

v6.6.1 is working with a few minor caveats such as custom IPS policies that used to deploy on the ASA 5500-X platform that won't deploy on the FPR - deployment fails with "Device does not have required amount of memory resource" message, even while it only have 5 rules inside the policy and minimal, custom sensitive data detection flags turned on. But that's for another topic.

This test FMC and its database was migrated and updated with no errors at every new upgrade since the early code. Historically this box was born as FMC v5 and it just insists in keep going forward FMC/Device backups and VM snapshots are taken as well with each new code (a huge time saving to play around on the lab). During my first attempt to update it to the latest and greatest v6.7, I went the shortest path and it failed. The next attempt I went step by step using the available Cisco.com support files as of earlier last week, to not avail. Then tried almost anything in between I could think of (as per the available upgrade paths).

Maybe because I'm also doing a technology change (moving away from the ASA5500-X series, which some sensors aren't supported at all by past FMC v6.6), things are not as straightforward as it should be.

And I agree staying at v6.6.1 is indeed the sane thing to do at this point. Because it's on a lab environment, I wanted to push boundaries to find out what else we could extract off the product, but until Cisco catches up with the competition and adds TLSv1.3 decryption, I have no real reason to go past their recommended code and v6.6.1 will be for the time being.

Thank you again for the note. Should I have more time to play, I might dig up more on this.

[ryan14 · Accepted solution]

***Update. TAC responded to me and said its related to bug CSCvw38870.

From CLI do the following:

expert

sudo su

mv /new-root/etc/sf/sfmail /new-root/etc/sf/sfmail.old

upgrade_resume.sh

[HQuest]

As backwards as this sounds, it definitely makes 100% sense. I have email notifications on this FMC, and it is part of the error message ("called from /usr/local/sf/lib/perl/5.10.1/SF/Snapshots/IntrusionEmail.pm") displayed during the upgrade. I did removed the email notifications (Policies > Actions > Alerts) and the update was performed as expected. As an alternate test, I recovered the v6.6.1 VM snapshot and manually removed the email notification folder as per your TAC received info, and the upgrade to v6.7 also completed as expected.

While I was off in thinking about LDAP or AD integration, I was right this is indeed a wrong coding, but I do appreciate you to post such findings.

[Abheesh Kumar]

Hello Ryan,

I face the same issue when upgrade but when i tried to enter the commands getting an error that no such directory.

root@HQ-FMC:~# mv /new-root/etc/sf/sfmail /new-root/etc/sf/sfmail.old
mv: cannot stat '/new-root/etc/sf/sfmail': No such file or directory
root@HQ-FMC:~#

There is no directory named new-root.

Regards,

Abheesh

[chong00011]

This is what Cisco TAC instructed me to do and it worked for me.


Hello Chenh,

If you haven't cancelled the upgrade, you can proceed with the following commands:

SSH the FMC CLI:
>expert
$sudo su
#mv /new-root/etc/sf/sfmail /new-root/etc/sf/sfmail.old

And then resume the upgrade:
#upgrade_resume.sh

The upgrade should complete after this, you can recover the same configuration back once it is 6.7 with the following command:
#mv /etc/sf/sfmail.old /etc/sf/sfmail

Or another option could be to create a new Intrusion Email alert configuration. If the sfmail is not found, a new one will be written.


This only work if the upgrade failed "FMC upgrade to 6.7.0 failed at 800_post/1027_ldap_external_auth_fix.pl". See Bug search tool CSCvw38870.

I hope this help.

Chenh Hong
Network Administrator

Glantz
2501 Constant Comment Place
Louisville, KY 40299
Tel: 502.568.4429

[https://www.nglantz.com/ASSETS/IMAGES/CMS/STATIC_IMAGES/email_signature.png]
Shop at Glantz
[https://www.nglantz.com//ASSETS/IMAGES/CMS/STATIC_IMAGES/Facebook.jpg][https://www.nglantz.com//ASSETS/IMAGES/CMS/STATIC_IMAGES/Instagram.jpg][https://www.nglantz.com//ASSETS/IMAGES/CMS/STATIC_IMAGES/LinkedIn.jpg]
Disclaimer posted by 766HGC3dXXQ167

[chong00011]

I ran into the same issue above. TAC did gave me the same steps to fix it and completed the upgrade. but when FMC boot up, it take a long time. I'm still waiting on it to completely up. how long does it take? or can I do a hard shut down to reboot it. my FMC is virtual.

thank you, 

[ryan14]

I recommend looking at the console of the VM to see what it is doing. Depending on your system recourses, especially disk i/o will determine how long the upgrade takes.

[chong00011]

After TAC recommend workaround, the upgrade completed in 10 mins. reboot took like 5 mins, then I log into the console and accept the EULA. At that point I just got the blink _. It over an hour, I know something is wrong. lucky my FMC is a VM. I'm going to restore it back to 6.6.1 and then trying it again.

[ryan14]

IIRC, my upgrade took approx 2-3 hours for the GUI to become available again. I would wait 6 hours before assuming the upgrade did not work based on past experiences.

[chong00011]

Thanks ryan14. I will try it again. I was wondering if 6.7 support on Azure?

[ryan14]

It looks like it

https://www.cisco.com/c/en/us/td/docs/security/firepower/quick_start/fmcv/fpmc-virtual/fpmc-virtual-azure.html