ASA unable to route between subnets (ping is OK)

[morleyc]

Dear all, I have a 5506 running ASA v9.8(1). I have tried getting my ASA to route traffic between subnets, i got it working for 10 minutes but after some changes i have broken something.

My interfaces are below, configured with security-levels.

!
interface GigabitEthernet1/1
description Trunk
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/1.10
description Management
vlan 10
nameif Vlan_Management
security-level 100
ip address 192.168.255.1 255.255.255.0
!
interface GigabitEthernet1/1.35
vlan 35
nameif vlan_Users
security-level 90
ip address 192.168.35.1 255.255.255.0
!
interface GigabitEthernet1/1.42
description Voice
vlan 42
nameif vlan_Voice
security-level 80
ip address 192.168.42.1 255.255.255.0
!
interface GigabitEthernet1/1.100
description Guest
vlan 100
nameif vlan_Guest
security-level 10
ip address 192.168.100.1 255.255.255.0
!
interface GigabitEthernet1/1.101
description Lab
vlan 101
nameif vlan_Lab
security-level 20
ip address 192.168.101.1 255.255.255.0
!
interface GigabitEthernet1/2
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/3
description Internet
nameif Outside
security-level 0
pppoe client vpdn group ISP
ip address pppoe setroute
!
object network Users
subnet 192.168.35.0 255.255.255.0
object network Remote_8
subnet 10.10.8.0 255.255.252.0
description Remote Site
object network Remote_200
subnet 10.10.200.0 255.255.255.0
description remote1
object network Corp
subnet 192.168.35.0 255.255.255.0
object network Voice
subnet 192.168.42.0 255.255.255.0
object network Lab
subnet 192.168.100.0 255.255.255.0
object network Guest
subnet 192.168.100.0 255.255.255.0
object-group network DM_INLINE_NETWORK_1
network-object object Remote_8
network-object object Remote_200

For debugging i have opened up the rules and applied rules to allow any traffic form user VLAN 35 to any other network, and vice versa to allow any traffic from VLAN 42 to any other... but i think ACL will be ignored due to the security-levels.

Remaining relevant config below:

pbkdf2
names
ip local pool VPN 192.168.35.2-192.168.35.240 mask 255.255.255.0
ip local pool webVPN 192.168.35.241-192.168.35.245 mask 255.255.255.0
ip local pool VPNN_addresses 192.168.35.246-192.168.35.248 mask 255.255.255.0
!
dns domain-lookup Vlan_Management
dns domain-lookup vlan_Users
dns domain-lookup vlan_Voice
dns domain-lookup vlan_Guest
dns domain-lookup vlan_Lab
dns domain-lookup Outside
same-security-traffic permit inter-interface
access-list Vlan_Corp_access_in extended permit ip any any
access-list Vlan_Corp_access_in extended permit ip 192.168.35.0 255.255.255.0 object-group DM_INLINE_NETWORK_1
access-list Vlan_Corp_access_in extended permit ip 192.168.35.0 255.255.255.0 192.168.42.0 255.255.255.0
access-list Vlan_Corp_access_in extended permit ip 192.168.35.0 255.255.255.0 interface Outside
access-list Outside_access_in extended permit ip any any
access-list Outside_cryptomap_1 extended permit ip object Users object Remote_8
access-list AnyConnect_Client_Local_Print extended deny ip any4 any4
access-list AnyConnect_Client_Local_Print extended permit tcp any4 any4 eq lpd
access-list AnyConnect_Client_Local_Print remark IPP: Internet Printing Protocol
access-list AnyConnect_Client_Local_Print extended permit tcp any4 any4 eq 631
access-list AnyConnect_Client_Local_Print remark Windows' printing port
access-list AnyConnect_Client_Local_Print extended permit tcp any4 any4 eq 9100
access-list AnyConnect_Client_Local_Print remark mDNS: multicast DNS protocol
access-list AnyConnect_Client_Local_Print extended permit udp any4 host 224.0.0.251 eq 5353
access-list AnyConnect_Client_Local_Print remark LLMNR: Link Local Multicast Name Resolution protocol
access-list AnyConnect_Client_Local_Print extended permit udp any4 host 224.0.0.252 eq 5355
access-list AnyConnect_Client_Local_Print remark TCP/NetBIOS protocol
access-list AnyConnect_Client_Local_Print extended permit tcp any4 any4 eq 137
access-list AnyConnect_Client_Local_Print extended permit udp any4 any4 eq netbios-ns
access-list vlan_Voice_access_in extended permit ip any any
access-list vlan_Lab_access_in_1 extended permit ip 192.168.101.0 255.255.255.0 interface Outside
access-list vlan_Guest_access_in extended permit ip 192.168.100.0 255.255.255.0 interface Outside
icmp permit any echo vlan_Users
icmp permit any echo-reply vlan_Users
no arp permit-nonconnected
nat (vlan_Users,Outside) source static Users Users destination static Remote_8 Remote_8 no-proxy-arp route-lookup
nat (vlan_Users,Outside) source static any any destination static Corp Corp no-proxy-arp route-lookup
nat (Outside,Outside) source static any any destination static Corp Corp no-proxy-arp route-lookup
!
object network Users
nat (vlan_Users,Outside) dynamic interface
object network Lab
nat (vlan_Lab,Outside) dynamic interface
object network Voice
nat (vlan_Voice,Outside) dynamic interface
object network Guest
nat (vlan_Guest,Outside) dynamic interface
access-group Vlan_Corp_access_in in interface vlan_Users
access-group vlan_Voice_access_in in interface vlan_Voice
access-group vlan_Guest_access_in in interface vlan_Guest
access-group vlan_Lab_access_in_1 in interface vlan_Lab
access-group Outside_access_in in interface Outside
route Outside 10.10.8.0 255.255.252.0 1.2.3.4 1
route Outside 10.10.200.0 255.255.255.255 1.2.3.4 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
timeout conn-holddown 0:00:15
timeout igp stale-route 0:01:10
user-identity default-domain LOCAL
aaa authentication enable console LOCAL
aaa authentication http console LOCAL
aaa authentication serial console LOCAL
aaa authentication ssh console LOCAL
aaa authentication telnet console LOCAL
aaa authorization command LOCAL
aaa authorization http console LOCAL
aaa authentication login-history
http server enable
no snmp-server location
no snmp-server contact
service sw-reset-button
!
console timeout 0
vpdn group ISP request dialout pppoe
vpdn group ISP localname ispuser
vpdn group ISP ppp authentication pap
vpdn username ispuser password ***** store-local
!
no threat-detection basic-threat
no threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
enable Outside
cache
disable
error-recovery disable
group-policy DefaultRAGroup internal
group-policy DefaultRAGroup attributes
banner none
vpn-tunnel-protocol l2tp-ipsec
group-policy DefaultRAGroup_1 internal
group-policy DefaultRAGroup_1 attributes
vpn-tunnel-protocol l2tp-ipsec
group-policy WVPN_policy internal
group-policy WVPN_policy attributes
vpn-tunnel-protocol ssl-clientless
group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol ikev1 ikev2 l2tp-ipsec
group-policy 1.2.3.4 internal
group-policy 1.2.3.4 attributes
vpn-tunnel-protocol ikev1 ikev2
dynamic-access-policy-record DfltAccessPolicy
tunnel-group DefaultRAGroup general-attributes
address-pool VPN
address-pool VPNN_addresses
authorization-server-group LOCAL
authorization-server-group (Outside) LOCAL
default-group-policy WVPN_policy
tunnel-group DefaultRAGroup ipsec-attributes
ikev1 pre-shared-key *****
tunnel-group DefaultRAGroup ppp-attributes
authentication ms-chap-v2
tunnel-group DefaultWEBVPNGroup general-attributes
address-pool webVPN
authentication-server-group (Outside) LOCAL
dhcp-server subnet-selection 192.168.35.3
tunnel-group 1.2.3.4 type ipsec-l2l
tunnel-group 1.2.3.4 general-attributes
default-group-policy 1.2.3.4
tunnel-group 1.2.3.4 ipsec-attributes
ikev1 pre-shared-key *****
ikev2 remote-authentication pre-shared-key *****
ikev2 local-authentication pre-shared-key *****
!
class-map inspection_default
match default-inspection-traffic
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
no tcp-inspection
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect esmtp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect ip-options
inspect icmp
!
service-policy global_policy global
prompt hostname context

Packet tracer reports OK from users to voice:

ASA# packet-tracer input vlan_Users tcp 192.168.35.3 15000 192.168.42.5 443
Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.42.5 using egress ifc vlan_Voice

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group Vlan_Corp_access_in in interface vlan_Users
access-list Vlan_Corp_access_in extended permit ip any any
Additional Information:

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 152, packet dispatched to next module

Result:
input-interface: vlan_Users
input-status: up
input-line-status: up
output-interface: vlan_Voice
output-status: up
output-line-status: up
Action: allow

And shows from voice to user drops due to implicit rule that it is on a lower security group number which i guess is OK as the connection to which it would normally be replying to (initiated by the user side) would be remembered (i don't necessarily want phones initiating connections, only users to phones):

ASA# packet-tracer input vlan_Voice tcp 192.168.42.5 15000 192.168.35.3 443

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.35.3 using egress ifc vlan_Users

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 4
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:

Result:
input-interface: vlan_Voice
input-status: up
input-line-status: up
output-interface: vlan_Users
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

 I am ok to start adding NAT rules if needed or would make things simpler (but i think this then starts to go against the way the ASA wants to work with its security-levels)?