ASA 5516-X Firepower Version Upgradation from 9.8(4)29 to 9.12.4.67

[King_1988]

Hi,

Can anyone please provide me the procedure the version upgradation from 9.8(4)29 to 9.12.4.67? What are pre-requisites ?

[HecOnPoint]

As a matter of fact we have been experiencing issues in our environment and the commonality between the Asa's is the  Software version 9.12.4.67 . Have an issue going on where the Primary Asa just gives out and causes outage have to make sure Failover is configured correctly. Upgraded around 6 Asa Pairs and have had these outages in 2 out of the 6 just in this last week. I have been in the environment 2 years and its the first time these Asa's give out. 

During outage yesterday I collected Show Tec and saw error:  CRYPTO: The ASA Crypto hardware accelerator Admin0 ring timed out.

At the moment I am not too sure its the software version but working with TAC to figure this out.

I will make sure to update this after my call with them shortly.

I wonder if anyone else has had issues with this Software version? 

[HecOnPoint]

As a matter of fact we have been experiencing issues in our environment and the commonality between the Asa's is the  Software version 9.12.4.67 . Have an issue going on where the Primary Asa just gives out and causes outage have to make sure Failover is configured correctly. Upgraded around 6 Asa Pairs and have had these outages in 2 out of the 6 just in this last week. I have been in the environment 2 years and its the first time these Asa's give out. 

During outage yesterday I collected Show Tec and saw error:  CRYPTO: The ASA Crypto hardware accelerator Admin0 ring timed out.

At the moment I am not too sure its the software version but working with TAC to figure this out.

I will make sure to update this after my call with them shortly.

I wonder if anyone else has had issues with this Software version? 

[Sheraz.Salim]

@HecOnPoint  please provide us the update from TAC on this issue. Thank you.

[HecOnPoint]

Wanted to provide a quick update after a TAC call turns out that version 9.12.4.67 does have a bug of CSCwi19654  basically depletes your ASA resources rendering services unavailable.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwi19654

Symptom: ASA 5508 firewall is reloading with thread name : CTM daemon.

Conditions: Seen only on 5506, 5508 or 5516 platforms running 9.12 In the last 30-40 minutes before the traceback, a few "%ASA-3-402148: CRYPTO: Random Number Generator error" can be observed.

Tac has now recommended to upgrade to version 9.16.4.57

This covers Arcane door vulnerability and CTM Daemon Bug

Remember to look at documentation for upgrade recommendations for example ASDM, Rommon, Anyconnect version requirements prior to upgrading. 

Alot of ciphers are depricated on the newer versions and might run into Anyconnect connection issues.  

Good Luck!

[Sheraz.Salim]

@HecOnPoint Thank you for the update. I really appreciate you taking the time to post here

[Izzat Shukri]

This is exactly what I am experiencing with 9.14.4.24. Upgraded to this version due to Arcane Door Vulnerability. Less than a week, my ASA rebooted 2 times. Planning to downgrade to 9.12.4.67 but seems you facing the same issue with mine. The reason I didn't upgrade to 9.16 yet is because of DH-Group 2,5 and 24 was removed. Need time to engage with clients to change the Tunnel configurations and i have more than 40 clients are using IPSec Tunnel with my ASA.

[Sheraz.Salim]

@Izzat ShukriI am in a very similar situation. We have more than 130 site-to-site VPN tunnels, and due to the deprecated DH group, we are unable to upgrade to newer versions. We are stuck with the 9.12 train of releases. Before the Arcane Door Vulnerability, we were running version 9.12.4.62. Since upgrading to 9.12.4.67, we have noticed packet ping loss. The rest of the applications are working fine, but we still experience ping loss. we have open a TAC case and see what they tell us.

Regarding our VPN tunnel upgrades, some of our third-party clients are responding very slowly, which has put us in a difficult situation and prevented us from upgrading to the 9.12+ releases.

[HecOnPoint]

Good Luck ! 

Please give us an update on how you go about this

[CSCO11737098]

Does this affect IPSec VPNs? The release notes just mention "Support has been removed for the DH groups 2, 5, and 24 in SSL DH group configuration. The ssl dh-group command has been updated to remove the command options group2, group5, and group24. ", which is a different configuration section than DH groups in VPNs.

[Sheraz.Salim]

as long as you on 9.12.x you are good to run with deprecated DH-Group 2,5,24. on wards 9.12.x version 2,5,24 no longer exists.

[CSCO11737098]

I just confirmed 9.16(4)61 still contains group5 in IKEv1, but group2 is removed. Seems to contradict the command reference guide. It does warn its deprecated and will be removed at some point, but its still present for now

(config-ikev1-policy)# group ?
ikev1-policy mode commands/options:
14 Diffie-Hellman group 14 (2048-bit MODP Group)
5 Diffie-Hellman group 5 (1536-bit MODP Group) (DEPRECATED)
(config-ikev1-policy)# group 5
WARNING: DH group 5 is considered insecure. This option is deprecated and will be removed in a later version

show run crypto | in group
group 5

[HecOnPoint]

I am in the process of upgrading to 9.16 sometime next week. I can follow up and update what to look out for and any issues that I run into. 

I would start with the client with the most users or if they are all equal then prioritize the client that is the most valuable.

I am Interested to know how you go about this.

[Izzat Shukri]

Currently i am in the middle planning and changing of the DH-Group. Need to identify which DH-Group will be used in our production environment. So I will upgrade to 9.16 after all tunnels configs changed to support 9.16 version.

What is your plan with the tunnel configurations with your 3rd party client? All has been done?

[Rob_M]

We have experienced similar issues as others mentioned here, ASA5516 running 9.12.4.67. We've experienced the issue on two ASA's. With this release, the ASA's seem to stop passing traffic after a few weeks of running. Physical status lights are all green, but no traffic is being passed and not responding to pings. Power reset brings them back.

Noticed on the disk there are several crypto_archive files all timestamped minutes after the ASA stopped responding and seemingly locked up, (crypto_archive/crypto_eng0_arch_1.bin through _5.bin). I did see bug ID CSCwh92345 which talks about random crypto archive files being generated after a software upgrade, it lists 9.12.4.67 as a known affected release. Not sure if its related, but I did notice these files all happen to be timestamped in the minutes AFTER the ASA stops passing traffic (which happened after a few weeks of uptime). And then they don't seem to be modified at all once the ASA is back up and running.

[CSCO11737098]

I would just update to 9.16(4)62 which came out a few weeks ago, all the ASAs we had crashing on 9.12.4.67 were resolved when I went to 9.16(4)61.