Object group and access list disappear after ASA reload

[jjtech]

Hi,

After upgrading our FPR-2110 running as ASA version 9.12 (4) to 9.18.4.22 , an object group and access list disappeared upon reload.

We re-added them , however once again after issuing write memory and reloading the ASA again after a few days, the same object group and access list are disappearing again.

Has anyone faced this issue before, or any idea what may be causing this issue?

[MHM Cisco World]

Reload' did you wr the config?

It can thr startup config is different than running config or you use some backup config before you do change.

MHM

[jjtech]

Yes I already confirmed that write memory was already issued before the reloads.

[tvotna]

@jjtech You need to look at physical console during reload. There should be error messages there.

In ASA 9.18 entire ACL subsystem was redesigned which also changed the way how ASA boots up and also changed running-config command order:
CSCvu39353 ENH: Optimize ACL / object-groups

For example, it is expected that access-group is lost upon downgrade from 9.18 to 9.12, but not the other way around. Still, there might be issues with the new architecture. One such issue was fixed in 9.18.4.22:
CSCwh62731 FTD Upgrade from 6.6.5 to 7.2.5 removing OGS causing rule expansion on boot

There might be others.

Did you have "object-group-search access-control" enabled in 9.12? Is it still enabled in 9.18? ("show run all | i object-group-search" + "show run | i object-group-search" + "show asp rule-engine"). How big are your ACLs (total size of all configs on the box in MB if it runs in multiple mode)?

[jjtech]

Thank you @tvotna  for the detailed answer.

After checking, object group search was not enabled before upgrade, neither is it enabled now.

The ASA is running in single mode currently. Regarding the ACLs size, I the number of elements is acceptable (Total number of access-list elements is currently 7689) and I think the ACLs count as well, but how can I accurately see how big the ACLs are ? ( I couldn't find a command that would show such statistics)

[Bernd Nies]

Welcome to the club. I've upgraded from ASA 9.18.4.22 to 9.18.4.24 and had this issue. See my post of today.

https://community.cisco.com/t5/network-security/cisco-asa-9-18-4-24-don-t-install-it-it-s-buggy/td-p/5123108 

Object-group search is not enabled. I tried that once years ago and it was buggy.

firewall1/web1/act# show run all | i object-group-search
no object-group-search access-control
no object-group-search threshold
no object-group-search access-control interface


firewall1/web1/act# show asp rule-engine

Rule compilation Status:   Completed
Duration(ms):              288821

S.No  Start Time                - Last Complete Time        Run Time(sec)

1     08:20:44 UTC Jun 3 2024   - 08:20:45 UTC Jun 3 2024    1
2     08:20:45 UTC Jun 3 2024   - 08:20:49 UTC Jun 3 2024    4
3     08:20:49 UTC Jun 3 2024   - 08:20:58 UTC Jun 3 2024    9
4     08:20:58 UTC Jun 3 2024   - 08:21:08 UTC Jun 3 2024    10
5     08:21:56 UTC Jun 3 2024   - 08:22:06 UTC Jun 3 2024    10
6     08:24:03 UTC Jun 3 2024   - 08:24:13 UTC Jun 3 2024    10
7     08:24:33 UTC Jun 3 2024   - 08:24:39 UTC Jun 3 2024    6
8     08:24:39 UTC Jun 3 2024   - 08:24:49 UTC Jun 3 2024    10
9     08:19:00 UTC Jun 3 2024   - 08:19:05 UTC Jun 3 2024    5
10    08:19:05 UTC Jun 3 2024   - 08:19:10 UTC Jun 3 2024    5
11    08:19:10 UTC Jun 3 2024   - 08:19:20 UTC Jun 3 2024    10
12    08:19:22 UTC Jun 3 2024   - 08:19:32 UTC Jun 3 2024    10
13    08:19:53 UTC Jun 3 2024   - 08:20:03 UTC Jun 3 2024    10
14    08:20:30 UTC Jun 3 2024   - 08:20:34 UTC Jun 3 2024    4
15    08:20:34 UTC Jun 3 2024   - 08:20:44 UTC Jun 3 2024    10

Module      | Insert      | Remove      | Current     |

 NAT        | 3394        | 2792        | 602         |
 ROUTE      | 2315        | 781         | 1534        |
 IFC        | 1278        | 1030        | 248         |
 ACL        | 148570      | 123129      | 25441       |
 IDENTITY   | 589         | 404         | 185         |

 Total      |                           | 28010       |

[tvotna]

What a pain. @jjtech , did you upgrade to 9.18.4.22 or 9.18.4.24? I mean, if you upgraded to 9.18.4.22 you could have faced with a completely different issue than @Bernd Nies .

The number of ACL elements is shown by "show access-list | i element", but in your case it is small, so this is not related to ACL size.

[jjtech]

@tvotna We upgraded to 9.18.4.22, so yes could be unrelated to Bernd's issue

[tvotna]

Ok, so once again, you either need another box for testing with access to physical console or a TAC case and TAC engineer can try to repro this issue with your configuration. Usually, an error message should be produced if something goes wrong when the firewall interprets its configuration, although https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-asaftd-aclconfig-wVK52f3z tells us that firewall can keep silence sometimes.

[Yordan Yordanov]

hi,

I have the same problem. I have also tried Write Standby, but it does not help. I can't see many objects and rules in the standby node.

Any suggestions/workarounds? https://bst.cisco.com/bugsearch/bug/CSCwj93921?rfs=qvlogin - this one does not helps 

regards

[MHM Cisco World]

this way you can check if the run config is same as startup config or not 
show run | in check 
show run | in check 

check the checksum is it same or not 
MHM