Cisco ASA 9.18.4.24: Don't install it !!! It's buggy !!!

[Bernd Nies]

Hi,

Last week we upgraded from ASA 9.18.4.22 to 9.18.4.24 running on Firepower2120 because the firewall reboots occasionaly since we upgraded from 9.18.3.56 due to VPN DDoS attacks.

ASA 9.18.4.24 introduces a new bug that kept me up all night to fix it: At boot time it delete all object-groups plus the associated access-lists and nat rules.

According to Cisco TAC it is this bug from 2023 that came back to latest suggested releas.

• https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwe64043
• https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-asaftd-aclconfig-wVK52f3z

However, in our case we have not enabled ACL optimization in any of our ASA contexts:

# show running-config object-group-search
no object-group-search access-control

Is it just me that gets the impression that ASA software qualitiy decreases from relase to release? Not even the suggested tag in download portal can be trusted anymore.

Regards,

Bernd

[balaji.bandi]

Since you are using the latest version of code, Not many people have tested on that version, personally not encountered that kind of situation where you lost the object groups ?

Do you see same scenario when you using ASDM ? or is this only CLI is the issue,

Cisco TAC is the right way to move forward to fix this issue, these are the people have interact with many deployments and have internal information compare to public community - personal views.

[Bernd Nies]

ASA 9.18.4.24 was released on 2 May 2024. I was fool to hope that 4 weeks was old enough.

It's an ASA in multi-context setup with four security contexts and 1 admin context.

Upgrade of Firepower 2120 node 2 went fine, but when upgrading node 1 in two security contexts Object-groups in two contexts were gone. All access-list lines containing object-groups and all nat rules using object-groups were also gone. On another context the access-lists with object-groups still existed, but the referenced object-groups did not exist anymore. 

I don't use ASDM.

[Bernd Nies]

Also a new bug that comes free with ASA 9.18.4.24:

# write mem all
Building configuration...
Do you wish to continue? [confirm]
Saving context :           system : (000/005 Contexts saved)
Cryptochecksum: b18d557c 81041d18 711a6e1b cc182cef

9022 bytes copied in 1.130 secs (9022 bytes/sec)
Saving context :            admin : (001/005 Contexts saved)
Cryptochecksum: c4622016 86d9c235 a837a214 77ad8c2b

26118 bytes copied in 1.270 secs (26118 bytes/sec)
Saving context :             web1 : (002/005 Contexts saved)
Cryptochecksum: 8e0ee469 2f5345b5 bb8bd888 376f736e

%Error reading system:/running-config (Configuration line too long)
Saving context :             web2 : (003/005 Contexts saved)
Cryptochecksum: 74ce0e68 b5f2e5d4 06864e7e 76ccdc9e

%Error reading system:/running-config (Configuration line too long)
Saving context :             wan1 : (004/005 Contexts saved)
Cryptochecksum: 1d29e697 fe79d3f6 50dbc5b7 c1e36cdc

%Error reading system:/running-config (Configuration line too long)
Saving context :             wan2 : (005/005 Contexts saved)
Cryptochecksum: 1c5f2191 d25a64b8 a7b57fd1 7531483a

%Error reading system:/running-config (Configuration line too long)
[OK]

Also changing to a context and saving running-config has the same effect.

firewall1/act# changeto context wan1
firewall1/wan1/act# write mem
Building configuration...
Cryptochecksum: 1d29e697 fe79d3f6 50dbc5b7 c1e36cdc

%Error reading system:/running-config (Configuration line too long)
[OK]
firewall1/wan1/act# copy run
firewall1/wan1/act# copy running-config start
firewall1/wan1/act# copy running-config startup-config

Source filename [running-config]?
Cryptochecksum: 1d29e697 fe79d3f6 50dbc5b7 c1e36cdc

%Error reading system:/running-config (Configuration line too long)

Temporary workaround is to do a "show running-config" and then running-config can be saved.

firewall1/wan1/act# show running-config
firewall1/wan1/act# write mem
Building configuration...
Cryptochecksum: a4b464cb 0bb22b65 6553f51d 24348f8e

66254 bytes copied in 1.140 secs (66254 bytes/sec)
[OK]

Cisco marketing should change their company logo to the facepalm emoji.

[Bernd Nies]

Possible related to this issue here:

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvu39353

But object-group-search is not active in any context.

firewall1/web1/act# show run all | i object-group-search
no object-group-search access-control
no object-group-search threshold
no object-group-search access-control interface

[tvotna]

CSCvu39353 is an ENH that went into 9.18.1. It is related of course, because the code was rewritten in 9.18, but this is not the root cause. The CSCwe64043 is also not the root cause, because "configuration line too long" error is something completely new, so TAC should open another bug ASAP if it doesn't exist yet.

[Bernd Nies]

Issue about "%Error reading system:/running-config (Configuration line too long)"

Found that now: https://quickview.cloudapps.cisco.com/quickview/bug/CSCwj93921

[Bernd Nies]

Entire code handling objects and object-groups is broken. It does not check if objects are used in groups when deleting them.

firewall1/web1/act(config)# show run object-group id dummy-group-1
object-group network dummy-group-1
 network-object object dummy-object-1
 network-object object dummy-object-2

firewall1/web1/act(config)# show run object id dummy-object-1
object network dummy-object-1
 host 1.0.0.0

firewall1/web1/act(config)# show run object id dummy-object-2
object network dummy-object-2
 host 2.0.0.0

firewall1/web1/act(config)# no object network dummy-object-1

firewall1/web1/act(config)# show run object-group id dummy-group-1
object-group network dummy-group-1
 network-object object dummy-object-1
 network-object object dummy-object-2
 network-object object dummy-object-1

Removing duplicate causes firewall to panic ...

firewall1/web1/act(config)# object-group network dummy-group-1
firewall1/web1/act(config-network-object-group)# no  network-object object dummy-object-1

Coredump starting....
Corehelper: /opt/cisco/csp/cores/core.lina.10.1266.1717407923
Waiting for Corehelper to finish....
Livecore: generating coredump of 1266
[New LWP 1377]
[New LWP 1378]
[New LWP 1379]
[New LWP 1380]
[New LWP 1382]
[New LWP 1383]
[New LWP 1384]
[New LWP 1385]
[New LWP 1386]
[New LWP 1387]
[New LWP 1388]
[New LWP 1389]
[New LWP 1390]
[New LWP 1391]
[New LWP 1392]
[New LWP 1393]
[New LWP 1394]
0x000000fff79a474c in pthread_cond_timedwait () from /lib64/libpthread.so.0
warning: target file /proc/1266/cmdline contained unexpected null characters

[Yordan Yordanov]

hi

for me this workaround doesnt work - https://bst.cisco.com/bugsearch/bug/CSCwj93921?rfs=qvlogin

do you have any idea to solve the issue? 

regards

[Bernd Nies]

For me this works on 9.18.4.24 in multi-context mode. Let the config display entirely

terminal pager 0
show running-config
write memory

However, I'm going to spend now another night downgrading back to 9.18.4.22. That one at least ran for four weeks with crashing only once ...

[Bernd Nies]

Hours later after downgrading to 9.18.4.22 and re-appling all lost object-groups and access-lists and nat rules again I noticed a new odd behaviour of 9.18.4.22. Object-group and object dependency checks no longer work. It used to complain when one edits an object-group and adds an object that already existed. Now nothing shown at all. Also it used to complain when deleting an object that it could not be deleted because it was still in use by an object-group or an access-list. 

Don't know if that was before on that version, but definitely not on a version prior to 9.18.x.

Did not try to delete an object that was used by an object-group. That caused the firewall to panic reboot before.

[tvotna]

There is a believe that the main issue with lost object-groups is a result of CSCwi84314 code fix which went into the following versions: 9.16.4.58, 9.18.4.23, 9.19.1.29, 9.20.2.13. The new bug id to track the fix is CSCwj68783 and it has already been fixed in 9.16.4.59/60, 9.18.4.25, 9.19.1.30, 9.20.2.20/21. If above is true, only the following official ASA releases are affected and should not be used: 9.18.4.24.

The CSCwj93921 reported above has the same root cause. Again, only official interim release 9.18.4.24 is affected.

Other issues reported here need be investigated.

FTD versions need be analyzed too.

[Marvin Rhoads]

FYI, CSCwj93921 is fixed in 9.18(4)29 which was published yesterday.
https://www.cisco.com/web/software/280775065/166425/ASA-9184-Interim-Release-Notes.html

[Bernd Nies]

Thanks. I've seen it last week. How long should one wait to actually download and install a new ASA release? Four weeks was too early last time ...

[Yordan Yordanov]

Don't do it for now! It is still buggy