Outside ACL stops working after ASA5510 version upgrade.

MILTON MENEGHINI · ‎10-26-2021

Dear members

I'm upgrading a pair of ASA5510 from 8.4.1 to 8.4.6 using the strategy of 0 downtime, upgrading the Stanby ASA first and switching it to active.

After switch the ASA5510 version 8.4.6, the outside acl seems to stop working.

I open the debug and I can see several Deny on outside ACL.

Of course, all the access to those server through NAT stop too.

I've tried other releases too, all of then the same issue: 8.4.5, 8.4.6, 8.4.7

I like some help from anyone who faced this issue before.

Thanks

Marvin Rhoads · ‎10-26-2021

There should not have been any syntax change for NAT or ACL between 8.4(1) and 8.4(6).

The packet-tracer cli utility will show you why your incoming traffic is being denied.

packet-tracer input <nameif of your outside interface> tcp 8.8.8.8 1234 <ip of your server> <allowed incoming tcp port number>

MILTON MENEGHINI · ‎10-26-2021

Thanks for your help Marvin

I agree that the configuration is exactly the same between the two versions and should work fine 8.4.1 and 8.4.6 ASAs.

I'll do this test next switch.

Marvin Rhoads · ‎10-26-2021

Try using the public IP address in your packet-tracer as that will be what the incoming packet should have in it (not the private IP - that's why the output indicates RPF drop reason).

MILTON MENEGHINI · ‎10-26-2021

I did this in the Standby ASA. I got the following output, but I can do the next switch afterhours (after 0:00):

vpn/sec/stby# packet-tracer input outside tcp 8.8.8.8 12345 200.196.235.16 http detailed

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
object network SRVTGPORTALWEB01
nat (inside,outside) static 200.196.235.16
Additional Information:
NAT divert to egress interface inside
Untranslate 200.196.235.16/80 to 10.234.1.60/80

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit tcp any object SRVTGPORTALWEB01 object-group DM_INLINE_TCP_8
object-group service DM_INLINE_TCP_8 tcp
port-object eq www
port-object eq https
Additional Information:

Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect http
service-policy global_policy global
Additional Information:

Phase: 5
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
object network SRVTGPORTALWEB01
nat (inside,outside) static 200.196.235.16
Additional Information:

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (fo-standby) Dropped by standby unit

I'll let you know tomorrow.

Thanks

MILTON MENEGHINI · ‎10-26-2021

I've just reload the Standby ASA with 8.4(6) version and the result is a DROP:

vpn/sec/stby# packet-tracer input outside tcp 8.8.8.8 12345 200.196.235.46 dom$

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 200.196.235.0 255.255.255.128 outside

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:
Forward Flow based lookup yields rule:
in id=0xaf630e08, priority=11, domain=permit, deny=true
hits=10, user_data=0x5, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

vpn/sec/stby# sh access-list | inc 10.234.1.213
access-list outside_access_in line 4 extended permit object-group DM_INLINE_SERVICE_3 any object 10.234.1.213 (hitcnt=0) 0x36732903
access-list outside_access_in line 4 extended permit tcp any host 10.234.1.213 eq domain (hitcnt=0) 0xd0d6c259
access-list outside_access_in line 4 extended permit udp any host 10.234.1.213 eq domain (hitcnt=0) 0x42dd3350

vpn/sec/stby#
************WARNING****WARNING****WARNING********************************
Mate version 8.4(1) is not identical with ours 8.4(6)
************WARNING****WARNING****WARNING********************************

MILTON MENEGHINI · ‎10-28-2021

Hi members

This morning I got success upgrading both ASA from 8.4(1) to 8.4(3) using the 0 downtime technique,

However, after that, I tried to upgrade to 8.4(5) with no success and to 8.4(6) with no sucess too.

While in 8.4(5) version, I tried several packet-tracer tests all of then with success.

But the access to all NATs from outside interface justto stop working: external DNS server, WEB server, etc.

even the ping from Standby unit to those IPs stop too. In 8.4(3) version the NATs can be pinged.

Any ideas?

Thanks in advance

************WARNING****WARNING****WARNING********************************
Mate version 8.4(3) is not identical with ours 8.4(5)
************WARNING****WARNING****WARNING********************************

vpn/pri/act# packet-tracer input outside udp 8.8.8.8 12345 200.196.235.46 domai$ udp 8.8.8.8 12345 200.196.235.46 domain detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 200.196.235.0 255.255.255.128 outside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit object-group DM_INLINE_SERVICE_7 any object 200.196.235.46
object-group service DM_INLINE_SERVICE_7
service-object tcp destination eq domain
service-object udp destination eq domain
Additional Information:
Forward Flow based lookup yields rule:
in id=0xac6d72e0, priority=13, domain=permit, deny=false
hits=573, user_data=0xa99bab80, cs_id=0x0, use_real_addr, flags=0x0, protocol=17
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=200.196.235.46, mask=255.255.255.255, port=53, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xac15f9e8, priority=0, domain=inspect-ip-options, deny=true
hits=6526, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 4
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xad572e18, priority=70, domain=inspect-dns-np, deny=false
hits=1246, user_data=0xad204fb8, cs_id=0x0, use_real_addr, flags=0x0, protocol=17
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=53, dscp=0x0
<--- More ---> input_ifc=outside, output_ifc=any

Phase: 5
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xac186528, priority=20, domain=lu, deny=false
hits=1252, user_data=0x0, cs_id=0x0, flags=0x0, protocol=17
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xabf3bb30, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=1909, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xac15f9e8, priority=0, domain=inspect-ip-options, deny=true
hits=6528, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 6405, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
<--- More ---> snp_fp_inspect_ip_options
snp_fp_inspect_dns
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_inspect_dns
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

vpn/pri/act# packet-tracer input outside tcp 8.8.8.8 12345 200.196.235.46 domai$ tcp 8.8.8.8 12345 200.196.235.46 domain detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in 200.196.235.0 255.255.255.128 outside

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside_access_in in interface outside
access-list outside_access_in extended permit object-group DM_INLINE_SERVICE_7 any object 200.196.235.46
object-group service DM_INLINE_SERVICE_7
service-object tcp destination eq domain
service-object udp destination eq domain
Additional Information:
Forward Flow based lookup yields rule:
in id=0xac6d70a0, priority=13, domain=permit, deny=false
hits=9, user_data=0xa99bab00, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=200.196.235.46, mask=255.255.255.255, port=53, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xac15f9e8, priority=0, domain=inspect-ip-options, deny=true
hits=7802, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 4
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xac088800, priority=20, domain=lu, deny=false
hits=379, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
<--- More ---> input_ifc=outside, output_ifc=any

Phase: 5
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xabf3bb30, priority=13, domain=ipsec-tunnel-flow, deny=true
hits=2381, user_data=0x0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xac15f9e8, priority=0, domain=inspect-ip-options, deny=true
hits=7804, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 7
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 7046, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
<--- More ---> snp_fp_fragment
snp_ifc_stat

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

johnlloyd_13 · ‎10-27-2021

hi,

it seems you're troubleshooting on the "standby" unit.

you should check/troubleshoot on the "active" unit instead (vpn/pri/act).

also make sure the ASA pair are on the same code/version.

MILTON MENEGHINI · ‎10-27-2021

Hi John, the active ASA is working fine and is in production. As I told, the problem occurs when I upgrade the version of the Standby ASA in order to follow the zero downtime technique proposed by Cisco. The new version 8.4(5), 8.4.(6) or 8.4(7) seems to disable the existing and working fine 8.4(1) access-lists. In order to perfome the zero downtime technique, the ASAs can have different version while you are switching active <-> standby to became active the ASA with new version. Thanks anyway.

johnlloyd_13 · ‎10-27-2021

hi,

i would assume this is an active-standby ASA setup.

all active traffic, NAT, ACL is being processed by the "active" FW: vpn/pri/act.

you're doing packet tracer and checking ACL in the "standby" FW: vpn/sec/stby which should NOT be done in a normal situation.

what you should do is perform a failover ("no failover active" on primary-active: vpn/pri/act) and test on the "secondary-active" FW: vpn/sec/act.

MILTON MENEGHINI · ‎10-27-2021

Hi John

Yes, Active-Standby ASA setup.

I did the failover to 8.4(6) ASA this morning (2:00AM) and all the access stop working.

When I did the packet-trace in the vpn/sec/stby with 8.4(1) version, I got an everything ALLOW but an DROP due to be in a Standby unit.

Action: drop
Drop-reason: (fo-standby) Dropped by standby unit

When I did the packet-trace in the vpn/sec/stby with 8.4(6) version, I got an DROP in Phase 2, showing the error:

Result: DROP
Config:
Implicit Rule

Considering the outside_access_in access-list exists, and is working fine in vpn/pri/act ASA, something is wrong with the migration.

So, I'm assuming this unit is not allowing traffic due to this error. If I failover to it, the network stops really (since I begin this upgrade process, the same error).

Thanks

johnlloyd_13 · ‎10-27-2021

hi,

when you perform a manual failover from "primary-active" to "secondary-standby", you should see the hostname prompt change to "sec-act" which i don't see in your output.

you're also running a different version 8.4(1) on the "secondary-standby" vs the "primary-active" 8.4(6). could you just upgrade both units on the same code first and re-do your failover test? again, ALWAYS PERFORM TEST ON THE "ACTIVE" ASA UNIT.

also make sure the switch ports and VLANs are the same on both ASA units/interfaces.